“The Growth Is There–But It’s Not That Evident,Is It!”: A Study in Science Delivery Improvement

The purpose of this study was to monitor the development of teacher personal attribute and school environmental factors at an urban intermediate (years 7 and 8) school in New Zealand as a result of an extended science professional development intervention project. Over a 2-year period, a validated, comprehensive science curriculum delivery evaluation instrument, the Science Curriculum Implementation Questionnaire (SCIQ, Lewthwaite, 2001), was used as a tool for data collection to monitor the initial and perpetuating influences of a full-year professional development program on science delivery. Based on the data collected and staff responses emanating from the analysis and discussion of the data represented by the SCIQ, outcomes of and contributors to the development were identified. As well, the effectiveness of both the professional development program and the SCIQ as an instrument for monitoring personal attribute and environmental changes as a result of professional development interventions are evaluated.     

Managing the primary curriculum: policy into practice in England

This paper reports on research into how effective non‐statutory guidance has been in managing the primary science curriculum in England. A small‐scale qualitative study was used to elicit the views of teachers, local education authority advisors, an Ofsted inspector and a policy‐maker. The findings indicate that the guidance is influencing the management of primary science teaching at both national and local level. The use of this guidance has been supported nationally and individual schools have found it supportive. The effect of this policy on pupil learning has still to be investigated by further classroom observational studies. It is also hypothesised that the long‐term effect of this guidance could be stagnation and a lack of creativity and this needs further study. Has this guidance reached the end of its useful life? Does it need replacing or are primary science teachers now in a position of not needing such support?     

“Are you saying I'm to blame?” Exploring the Influence of a Principal on Elementary Science Delivery

This research exercise, employing an action research model for curriculum improvement, explored the factors influencing science program delivery at an elementary school in New Zealand. Using a validated science program delivery evaluation tool, the Science Curriculum Implementation Questionnaire (SCIQ), as a foundation for data collection, staff discussion and collaborative decision-making, a school embarked on a self-review process and, first of all, identified factors influencing science program delivery and, secondly, identified strategies for improvement of science delivery. After a school-wide cycle of focused science delivery improvement, the evaluation was repeated and further emerging trends and strategies for improvement were collaboratively discussed and implemented. Implications of this self-review process on science program delivery improvement are discussed, especially within the context of the role of the principal in influencing curriculum delivery.     

An analysis of the processes of change in two science teachers educators' thinking

This study examined the processes of change in thinking as a means of learning about the professional development of two science teacher educators. A qualitative methodology was used. The main research tool was a semistructured in‐depth interview. The primary data analyzed came from two science teacher educators, selected from a broader set of seven. Findings emphasize the importance of questions regarding the knowledge possessed by teacher educators and of the questions regarding the role of science teacher educators. That is, the knowledge of an expert science teacher educator was more than a list of givens; it was personal and context‐bound. This study contributes new insight into the processes of teacher educators' professional development and change in their thinking. © 2007 Wiley Periodicals, Inc. J Res Sci Teach 44: 1219–1245, 2007     

Evaluating Instrument Quality in Science Education: Rasch‐based analyses of a Nature of Science test

Given the central importance of the Nature of Science (NOS) and Scientific Inquiry (SI) in national and international science standards and science learning, empirical support for the theoretical delineation of these constructs is of considerable significance. Furthermore, tests of the effects of varying magnitudes of NOS knowledge on domain‐specific science understanding and belief require the application of instruments validated in accordance with AERA, APA, and NCME assessment standards. Our study explores three interrelated aspects of a recently developed NOS instrument: (1) validity and reliability; (2) instrument dimensionality; and (3) item scales, properties, and qualities within the context of Classical Test Theory and Item Response Theory (Rasch modeling). A construct analysis revealed that the instrument did not match published operationalizations of NOS concepts. Rasch analysis of the original instrument—as well as a reduced item set—indicated that a two‐dimensional Rasch model fit significantly better than a one‐dimensional model in both cases. Thus, our study revealed that NOS and SI are supported as two separate dimensions, corroborating theoretical distinctions in the literature. To identify items with unacceptable fit values, item quality analyses were used. A Wright Map revealed that few items sufficiently distinguished high performers in the sample and excessive numbers of items were present at the low end of the performance scale. Overall, our study outlines an approach for how Rasch modeling may be used to evaluate and improve Likert‐type instruments in science education.     

Developing and evaluating instructionally sensitive assessments in science

The purpose of this article is to address a major gap in the instructional sensitivity literature on how to develop instructionally sensitive assessments. We propose an approach to developing and evaluating instructionally sensitive assessments in science and test this approach with one elementary life‐science module. The assessment we developed was administered to 125 students in seven classrooms. The development approach considered three dimensions of instructional sensitivity; that is, assessment items should: represent the curriculum content, reflect the quality of instruction, and have formative value for teaching. Focusing solely on the first dimension, representation of the curriculum content, this study was guided by the following research questions: (1) What science module characteristics can be systematically manipulated to develop items that prove to be instructionally sensitive? and (2) Are the instructionally sensitive assessments developed sufficiently valid to make inferences about the impact of instruction on students' performance? In this article, we describe our item development approach and provide empirical evidence to support validity arguments about the developed instructionally sensitive items. Results indicated that: (1) manipulations of the items at different proximities to vary their sensitivity were aligned with the rules for item development and also corresponded with pre‐to‐post gains; and (2) the items developed at different distances from the science module showed a pattern of pre‐to‐post gain consistent with their instructional sensitivity, that is, the closer the items were to the science module, the larger the observed gains and effect sizes. © 2012 Wiley Periodicals, Inc. J Res Sci Teach 49: 691–712, 2012     

Primary teachers’ changing attitudes and cognition during a two‐year science in‐service programme and their effect on pupils

Changes in 70 teachers’ confidence, attitudes and science understanding were tested before and after a major in‐service programme. Attitudes were assessed using a 49‐item Likert‐scale test that probed attitudes to practical science teaching and in‐service training. Multi‐choice and open‐ended questions measured understanding of electricity; melting, dissolving and evaporation; forces; and investigations. Data on pupils’ attitudes and cognition were also collected. After in‐service, teachers’ confidence about science teaching had improved significantly. The majority, but not all, had developed satisfactory levels of understanding and more positive attitudes. Four broad types of response were identified: high attainers who improved attitudes and confidence; teachers with limited science knowledge who found the course difficult but made improvements; unaffected professionals who were already working well and for whom the course had little effect; and disaffected teachers who showed low levels of confidence and competence throughout. Pupil cognition and attitudinal differences related to these types were also found.     

The implications of content versus item validity on science tests

The use of content validity as the primary assurance of the measurement accuracy for science assessment examinations is questioned. An alternative accuracy measure, item validity, is proposed. Item validity is based on research using qualitative comparisons between (a) student answers to objective items on the examination, (b) clinical interviews with examinees designed to ascertain their knowledge and understanding of the objective examination items, and (c) student answers to essay examination items prepared as an equivalent to the objective examination items. Calculations of item validity are used to show that selected objective items from the science assessment examination overestimated the actual student understanding of science content. Overestimation occurs when a student correctly answers an examination item, but for a reason other than that needed for an understanding of the content in question. There was little evidence that students incorrectly answered the items studied for the wrong reason, resulting in underestimation of the students' knowledge. The equivalent essay items were found to limit the amount of mismeasurement of the students' knowledge. Specific examples are cited and general suggestions are made on how to improve the measurement accuracy of objective examinations.     

Attitude measurement: Judging the emotional intensity of likert-type science attitude statements

Emotional intensity, that readiness of a teacher to respond favorably or unfavorably toward such psychological objects as science or the teaching of science, is the quality that distinguishes the attitude concept from other related psychological concepts. It would seem, then, that valid attitude statements, if they are to reflect the definition of attitude, would evoke emotional intensity, responses in both a favorable and unfavorable direction by a group of teachers on each item on a science attitude scale. Science educators who design or modify science attitude scales should continue using item-total correlations and other quantitative techniques to test for emotional intensity, but qualitative judgments are necessary, too. In addition, the frequency distribution of data generated by each statement should be examined for skewness and high percentages of neutral responses, both of which can impair the emotional intensity of an item.     

Rasch analysis for psychometric improvement of science attitude rating scales

ABSTRACT

Students’ attitude towards science (SAS) is often a subject of investigation in science education research. Survey of rating scale is commonly used in the study of SAS. The present study illustrates how Rasch analysis can be used to provide psychometric information of SAS rating scales. The analyses were conducted on a 20-item SAS scale used in an existing dataset of The Trends in International Mathematics and Science Study (TIMSS) (2011). Data of all the eight-grade participants from Hong Kong and Singapore (N?=?9942) were retrieved for analyses. Additional insights from Rasch analysis that are not commonly available from conventional test and item analyses were discussed, such as invariance measurement of SAS, unidimensionality of SAS construct, optimum utilization of SAS rating categories, and item difficulty hierarchy in the SAS scale. Recommendations on how TIMSS items on the measurement of SAS can be better designed were discussed. The study also highlights the importance of using Rasch estimates for statistical parametric tests (e.g. ANOVA, t-test) that are common in science education research for group comparisons.     

Learning about Inheritance in an Out‐of‐School Setting

The purpose of this study was to investigate primary students’ learning through participation in an out‐of‐school enrichment programme, held in a science centre, which focused on DNA and genes and whether participation in the programme led to an increased understanding of inheritance as well as promoted interest in the topic. The sample consisted of two groups (245 students in the experimental group and 150 students in the control group) of upper primary students (Grade 5) from six schools in Singapore. Two instruments were developed—a 15‐item multiple‐choice test to measure learning gains and a 17‐item survey form to measure student feedback. Pre‐, post‐, and delayed post‐tests were administered. Results showed statistically significant gains in learning for the experimental group that appeared to be stable as well as high levels of interest stimulated by the programme.     

Simulation study of item bias detection with restricted factor analysis

Restricted factor analysis (RFA) can be used to detect item bias (also called differential item functioning). In the RFA method of item bias detection, the common factor model serves as an item response model, but group membership is also included in the model. Two simulation studies are reported, both showing that the RFA method detects bias in 7‐point scale items very well, especially when the sample size is large, the mean trait difference between groups is small, the group sizes are equal, and the amount of bias is large. The first study further shows that the RFA method detects bias in dichotomous items at least as well as an established method based on the one‐parameter logistic item response model. The second study concerns various procedures to evaluate the significance of two‐item bias indices provided by the RFA method. The results indicate that the RFA method performs best when it is used in an iterative procedure.     

The Benefits of Fixed Item Parameter Calibration for Parameter Accuracy in Small Sample Situations in Large‐Scale Assessments

Large‐scale assessments such as the Programme for International Student Assessment (PISA) have field trials where new survey features are tested for utility in the main survey. Because of resource constraints, there is a trade‐off between how much of the sample can be used to test new survey features and how much can be used for the initial item response theory (IRT) scaling. Utilizing real assessment data of the PISA 2015 Science assessment, this article demonstrates that using fixed item parameter calibration (FIPC) in the field trial yields stable item parameter estimates in the initial IRT scaling for samples as small as n = 250 per country. Moreover, the results indicate that for the recovery of the county‐specific latent trait distributions, the estimates of the trend items (i.e., the information introduced into the calibration) are crucial. Thus, concerning the country‐level sample size of n = 1,950 currently used in the PISA field trial, FIPC is useful for increasing the number of survey features that can be examined during the field trial without the need to increase the total sample size. This enables international large‐scale assessments such as PISA to keep up with state‐of‐the‐art developments regarding assessment frameworks, psychometric models, and delivery platform capabilities.     

Effects of a Collaborative Science Intervention on High Achieving Students’ Learning Anxiety and Attitudes toward Science

This study investigated the effects of a collaborative science intervention on high achieving students’ learning anxiety and attitudes toward science. Thirty‐seven eighth‐grade high achieving students (16 boys and 21 girls) were selected as an experimental group who joined a 20‐week collaborative science intervention, which integrated and utilized an innovative teaching strategy. Fifty‐eight eighth‐grade high achieving students were selected as the comparison group. The Secondary School Student Questionnaire was conducted to measure all participants’ learning anxiety and attitudes toward science. In addition, 12 target students from the experimental group (i.e., six active and six passive students) were recruited for weekly classroom observations and follow‐up interviews during the intervention. Both quantitative and qualitative findings revealed that experimental group students experienced significant impact as seen through increased attitudes and decreased anxiety of learning science. Implications for practice and research are provided.     

An Instructional Module on Mokken Scale Analysis

Mokken scale analysis (MSA) is a probabilistic‐nonparametric approach to item response theory (IRT) that can be used to evaluate fundamental measurement properties with less strict assumptions than parametric IRT models. This instructional module provides an introduction to MSA as a probabilistic‐nonparametric framework in which to explore measurement quality, with an emphasis on its application in the context of educational assessment. The module describes both dichotomous and polytomous formulations of the MSA model. Examples of the application of MSA to educational assessment are provided using data from a multiple‐choice physical science assessment and a rater‐mediated writing assessment.     

A Comparison of Student Teachers' Beliefs from Four Different Science Teaching Domains Using a Mixed Methods Design

The study presented in this paper integrates data from four combined research studies, which are both qualitative and quantitative in nature. The studies describe freshman science student teachers' beliefs about teaching and learning. These freshmen intend to become teachers in Germany in one of four science teaching domains (secondary biology, chemistry, and physics, respectively, as well as primary school science). The qualitative data from the first study are based on student teachers' drawings of themselves in teaching situations. It was formulated using Grounded Theory to test three scales: Beliefs about Classroom Organisation, Beliefs about Teaching Objectives, and Epistemological Beliefs. Three further quantitative studies give insight into student teachers' curricular beliefs, their beliefs about the nature of science itself, and about the student- and/or teacher-centredness of science teaching. This paper describes a design to integrate all these data within a mixed methods framework. The aim of the current study is to describe a broad, triangulated picture of freshman science student teachers' beliefs about teaching and learning within their respective science teaching domain. The study reveals clear tendencies between the sub-groups. The results suggest that freshman chemistry and—even more pronouncedly—freshman physics student teachers profess quite traditional beliefs about science teaching and learning. Biology and primary school student teachers express beliefs about their subjects which are more in line with modern educational theory. The mixed methods approach towards the student teachers' beliefs is reflected upon and implications for science education and science teacher education are discussed.     

Engaging Students in Science Across the Primary Secondary Interface: Listening to the Students’ Voice

Most studies indicate primary students’ attitudes towards and interest in science decline as they progress into the secondary years. Longitudinal qualitative research exploring this phenomenon is rare as is research which focuses on the students’ voice as they cross the interface. In this study multiple qualitative data sources, supported by a ‘science interest’ composite scale, followed 20 students over 2 years. In contrast to baseline data on their peers these students, in general, maintained their interest in science. Apart from identifying the teacher’s pedagogical approach and classroom environment as two key issues in understanding these students’ journeys, the importance of listening to and heeding the students’ voice may be an even more critical concern in addressing the decline in students’ attitudes and interest in science.