Scientific reasoning in elementary school children: Assessment and relations with cognitive abilities

The primary goal of this study was the broad assessment and modeling of scientific reasoning in elementary school age. One hundred fifty-five fourth graders were tested on 20 recently developed paper-and-pencil items tapping four different components of scientific reasoning (understanding the nature of science, understanding theories, designing experiments, and interpreting data). As confirmed by Rasch analyses, the scientific reasoning items formed a reliable scale. Model comparisons differentiated scientific reasoning as a separate construct from measures of intelligence and reading skills and revealed discriminant validity. Furthermore, we explored the relationship between scientific reasoning and the postulated prerequisites inhibitory control, spatial abilities and problem-solving skills. As shown by correlation and regression analyses, beside general cognitive abilities (intelligence, reading skills) problem-solving skills and spatial abilities predicted performance in scientific reasoning items and thus contributed to explaining individual differences in elementary school children's scientific reasoning competencies.     

Measuring ocean literacy of high school students: psychometric properties of a Chinese version of the ocean literacy scale

This study established a Chinese scale for measuring high school students’ ocean literacy. This included testing its reliability, validity, and differential item functioning (DIF) with the aim of compensating for the lack of DIF tests focusing on current scales. The construct validity and reliability were verified and tested by analyzing the established scale’s items using the Rasch model, and a gender DIF test was conducted to ensure the test results’ fairness when distinct groups were compared simultaneously. The results indicated that the scale established in this study is unidimensional and possesses favorable internal consistency and construct validity. The gender DIF test results indicated that several items were difficult for either female or male students to correctly answer; however, the experts and scholars discussed these items individually and suggested retaining them. The final Chinese version of the ocean literacy scale developed here comprises 48 items that can reflect high school students’ understanding of ocean literacy—which helps students understand the topics of marine science encountered in real life.     

Construction and validation of a classroom climate scale: a mixed methods approach

Students’ perceptions of their classroom climate have been found to relate significantly to students’ learning outcomes. The purpose of the present study was to construct an instrument for assessing elementary-school students’ perceptions of classroom climate, based on a previous instrument that was being used in Chile by a public national school mental health program as a tool for aiding teachers in improving classroom management, but which showed poor psychometric properties. We used a six-staged mixed-methods approach to construct relevant items and dimensions based on this measure and by adapting previously-existing scales. Item development included participatory construction of items involving program officials, focus groups with students, and a pilot study. The final version was administered to a sample of 6813 elementary-school students. Results showed adequate reliability and construct validity, convergent validity with school climate, and divergent validity with peer victimisation. When consequential validity was explored through semi-structured interviews with program officials and school administrators, we found that the instrument was being used as a tool for helping teachers to improve their school climate and management skills. We discuss the importance of constructing instruments using a mixed-methods approach.     

Construct validity of the Nepalese school leaving english reading test

There has been a steady interest in investigating the validity of language tests in the last decades. Despite numerous studies on construct validity in language testing, there are not many studies examining the construct validity of a reading test. This paper reports on a study that explored the construct validity of the English reading test in the Nepalese school leaving examination. Eight students were asked to take the test and think-aloud, followed by retrospective interviews. Additionally, seven experts were asked to make judgments regarding the skills tested by the test. The findings provide grounded insights into students’ response behaviors prompted by the reading tasks, and indicate some threats to the construct validity of the test. Additionally, the study reports a low level of agreement among the experts, and a big gap between the skills used by the students and the skills that the experts thought were being examined by the test.     

Pedagogy of Science Teaching Tests: Formative assessments of science teaching orientations

A critical aspect of teacher education is gaining pedagogical content knowledge of how to teach science for conceptual understanding. Given the time limitations of college methods courses, it is difficult to touch on more than a fraction of the science topics potentially taught across grades K-8, particularly in the context of relevant pedagogies. This research and development work centers on constructing a formative assessment resource to help expose pre-service teachers to a greater number of science topics within teaching episodes using various modes of instruction. To this end, 100 problem-based, science pedagogy assessment items were developed via expert group discussions and pilot testing. Each item contains a classroom vignette followed by response choices carefully crafted to include four basic pedagogies (didactic direct, active direct, guided inquiry, and open inquiry). The brief but numerous items allow a substantial increase in the number of science topics that pre-service students may consider. The intention is that students and teachers will be able to share and discuss particular responses to individual items, or else record their responses to collections of items and thereby create a snapshot profile of their teaching orientations. Subsets of items were piloted with students in pre-service science methods courses, and the quantitative results of student responses were spread sufficiently to suggest that the items can be effective for their intended purpose.     

The Case for a Middle School College Readiness Index: Rejoinder to Lazowski,Barron, Kosovich,and Hulleman and Mattern,Allen, and Camara

This rejoinder provides a reply to comments on a middle school college readiness index, which was devised to generate earlier and more nuanced readiness diagnoses to K‐12 students. Issues of reliability and validity (including construct underrepresentation and construct‐irrelevant variance) are discussed in detail. In addition, comments from Lazowski et al. and Mattern et al. provide a useful summary of areas where more college readiness research is needed. This rejoinder elaborates on those essential next steps, including deploying readiness indicators in school settings. Advances in the measurement of noncognitive skills will require more validity research using more contemporary data from authentic school settings. Still, the middle school college readiness index is a promising proof of concept, demonstrating the utility of diverse, early measures to support targeted, timely interventions.     

Development of an item bank for assessing generic competences in a higher-education institute: a Rasch modelling approach

This paper describes the development and validation of an item bank designed for students to assess their own achievements across an undergraduate-degree programme in seven generic competences (i.e., problem-solving skills, critical-thinking skills, creative-thinking skills, ethical decision-making skills, effective communication skills, social interaction skills and global perspective). The Rasch modelling approach was adopted for instrument development and validation. A total of 425 items were developed. The content validity of these items was examined via six focus group interviews with target students, and the construct validity was verified against data collected from a large student sample (N?=?1151). A matrix design was adopted to assemble the items in 26 test forms, which were distributed at random in each administration session. The results demonstrated that the item bank had high reliability and good construct validity. Cross-sectional comparisons of Years 1–4 students revealed patterns of changes over the years. Correlation analyses shed light on the relationships between the constructs. Implications are drawn to inform future efforts to develop the instrument, and suggestions are made regarding ways to use the instrument to enhance the teaching and learning of generic skills.     

Comparing Two Inquiry Professional Development Interventions in Science on Primary Students’ Questioning and Other Inquiry Behaviours

Developing students’ skills to pose and respond to questions and actively engage in inquiry behaviours enables students to problem solve and critically engage with learning and society. The aim of this study was to analyse the impact of providing teachers with an intervention in inquiry pedagogy alongside inquiry science curriculum in comparison to an intervention in non-inquiry pedagogy alongside inquiry science curriculum on student questioning and other inquiry behaviours. Teacher participants in the comparison condition received training in four inquiry-based science units and in collaborative strategic reading. The experimental group, the community of inquiry (COI) condition, received training in facilitating a COI in addition to training in the same four inquiry-based science units. This study involved 227 students and 18 teachers in 9 primary schools across Brisbane, Australia. The teachers were randomly allocated by school to one of the two conditions. The study followed the students across years 6 and 7 and students’ discourse during small group activities was recorded, transcribed and coded for verbal inquiry behaviours. In the second year of the study, students in the COI condition demonstrated a significantly higher frequency of procedural and substantive higher-order thinking questions and other inquiry behaviours than those in the comparison condition. Implementing a COI within an inquiry science curriculum develops students’ questioning and science inquiry behaviours and allows teachers to foster inquiry skills predicated by the Australian Science Curriculum. Provision of inquiry science curriculum resources alone is not sufficient to promote the questioning and other verbal inquiry behaviours predicated by the Australian Science Curriculum.     

Identifying Promising Items: The Use of Crowdsourcing in the Development of Assessment Instruments

The psychometrically sound development of assessment instruments requires pilot testing of candidate items as a first step in gauging their quality, typically a time-consuming and costly effort. Crowdsourcing offers the opportunity for gathering data much more quickly and inexpensively than from most targeted populations. In a simulation of a pilot testing protocol, item parameters for 110 life science questions are estimated from 4,043 crowdsourced adult subjects and then compared with those from 20,937 middle school science students. In terms of item discrimination classification (high vs. low), classical test theory yields an acceptable level of agreement (C-statistic = 0.755); item response theory produces excellent results (C-statistic = 0.848). Item response theory also identifies potential anchor items without including any false positives (items with low discrimination in the targeted population). We conclude that the use of crowdsourcing subjects is a reasonable, efficient method for the identification of high-quality items for field testing and for the selection of anchor items to be used for test equating.     

The development of a science process assessment for fourth-grade students

In this study, a multiple-choice test entitled the Science Process Assessment was developed to measure the science process skills of students in grade four. Based on the Recommended Science Competency Continuum for Grades K to 6 for Pennsylvania Schools, this instrument measured the skills of (1) observing, (2) classifying, (3) inferring, (4) predicting, (5) measuring, (6) communicating, (7) using space/time relations, (8) defining operationally, (9) formulating hypotheses, (10) experimenting, (11) recognizing variables, (12) interpreting data, and (13) formulating models. To prepare the instrument, classroom teachers and science educators were invited to participate in two science education workshops designed to develop an item bank of test questions applicable to measuring process skill learning. Participants formed “writing teams” and generated 65 test items representing the 13 process skills. After a comprehensive group critique of each item, 61 items were identified for inclusion into the Science Process Assessment item bank. To establish content validity, the item bank was submitted to a select panel of science educators for the purpose of judging item acceptability. This analysis yielded 55 acceptable test items and produced the Science Process Assessment, Pilot 1. Pilot 1 was administered to 184 fourth-grade students. Students were given a copy of the test booklet; teachers read each test aloud to the students. Upon completion of this first administration, data from the item analysis yielded a reliability coefficient of 0.73. Subsequently, 40 test items were identified for the Science Process Assessment, Pilot 2. Using the test-retest method, the Science Process Assessment, Pilot 2 (Test 1 and Test 2) was administered to 113 fourth-grade students. Reliability coefficients of 0.80 and 0.82, respectively, were ascertained. The correlation between Test 1 and Test 2 was 0.77. The results of this study indicate that (1) the Science Process Assessment, Pilot 2, is a valid and reliable instrument applicable to measuring the science process skills of students in grade four, (2) using educational workshops as a means of developing item banks of test questions is viable and productive in the test development process, and (3) involving classroom teachers and science educators in the test development process is educationally efficient and effective.     

Trace Lines for Testlets: A Use of Multiple-Categorical-Response Models

It is not always convenient or appropriate to construct tests in which individual items are fungible. There are situations in which small clusters of items (testlets) are the units that are assembled to create a test. Using data from a test of reading comprehension constructed of four passages with several questions following each passage, we show that local independence fails at the level of the individual questions. The questions following each passage, however, constitute a testlet. We discuss the application to testlet scoring of some multiple-category models originally developed for individual items, In the example examined, the concurrent validity of the testlet scoring equaled or exceeded that of individual-item-level scoring     

Assessing Teachers' Beliefs about Their Science Teaching Context

The primary purpose of this study was to develop and apply a method for assessing teachers' context beliefs about their science teaching environment. Interviews with 130 purposefully selected teachers resulted in 28 categories of environmental factors and/or people who were perceived to influence science teaching. These categories were used to develop items for the Context Beliefs about Teaching Science instrument and provided evidence for content validity. Construct validity was partially confirmed through factor analysis that resulted in 26 items and two subscales on the final instrument. Using Ford's Motivation Systems Theory and Bandura's Theory of Collective Efficacy, additional evidence for construct validity was found in the modest correlation of context beliefs with outcome expectancy beliefs and the low correlation with science teaching self‐efficacy beliefs. The instrument was tested using 262 teachers participating in long‐term science professional development programs. These teachers possessed fairly positive context beliefs and, according to Ford's theory, should be capable of effective functioning in the classroom. It was concluded that the assessment of context beliefs would complement current science teacher self‐efficacy measures, thereby allowing researchers to develop profiles of science teachers' personal agency belief patterns. It could also be used to determine the factors which predict particular personal agency belief patterns, and assess teachers' perceptions of the strengths and weaknesses of school science programs, and could be used in planning and monitoring professional development experiences for science teachers. © 2000 John Wiley & Sons, Inc. J Res Sci Teach 37: 275–292, 2000.     

Students' perceptions of clinical attachments across rural and metropolitan settings

For many years, medical students at the University of Sydney undertook their clinical clerkships in traditional metropolitan teaching hospitals, which were regarded as the ‘gold standard’ for clinical training. In 2001 the university established a rural clinical school at which increasing numbers of students now complete a significant proportion of their medical education. The aim of the study reported here is to examine students’ perceptions of what facilitates their learning in clinical settings and to compare their perceptions across rural and metropolitan settings. Focus groups were conducted to collect students’ views on their experiences of learning in clinical settings. The findings were used to generate a questionnaire with items designed directly from focus‐group data ensuring content validity. The questionnaire was sent to all students in the 2004 cohort. Exploratory factor analysis was used to provide evidence of construct validity. The internal consistency reliability of the questionnaire was assessed using Cronbach’s alpha. Factor scores were computed to compare students’ perceptions across the two settings. Four factors were extracted: (1) clinical teachers’ orientation to teaching; (2) opportunities to develop clinical skills; (3) supportiveness of the clinical setting; and (4) student confidence and sense of self‐efficacy. Students rated the rural experience more highly and positively than the metropolitan hospital experience with regard to all four factors. This study highlights the positive role that rural attachments can play in providing an educationally sound learning experience. The findings are important in the context of both the drive, among medical programs worldwide, to seek out additional and alternative settings for clinical education and the national agenda to foster student interest in rural careers through positive rural training experiences.     

Challenging student teachers' conceptions of science and technology education

To address expected negative attitudes to studying science and technology held by primary school student teachers, we devised a learning model which combined cooperative group strategies with a learners' questions approach in a context which allowed for pluralism in methodology and epistemology. The model was used in a teacher education elective subject studied by final year Diploma of Teaching students at the University of Technology, Sydney. We found that some students were inexperienced in participating in the planning and design of their learning and that for many students, being responsible for their learning in a science and technology context aroused reactions of alarm and determined avoidance so that alternative pathways for achievement in the subject had to be offered. Some students reported feelings of satisfaction in their successful learning despite initial anxiety, low confidence or indifference. Specializations: children's learning in science and technology; inclusion; contexts, teaching models.     

My Attitudes Toward Science (MATS): the development of a multidimensional instrument measuring students’ science attitudes

The number of students in the United States choosing science, technology, engineering or mathematics careers is declining at a time when demand for these occupations is rapidly increasing. Numerous efforts have been undertaken to reverse this trend, yet results are uncertain. One’s attitude is key to many choices one makes, and this includes, for many, what career is pursued. Hence, teachers, informal science educators and researchers often wish to measure children’s attitudes towards science using a pretest and a posttest to determine the effects of a curriculum, an activity or an intervention. However, measuring children’s attitudes toward science has been problematic because of both the limited use of basic psychometrics in checking reliability and validity of instruments and the lack of a single construct of students’ attitudes towards science being surveyed. This article reports the development and testing of an instrument for measuring students’ science attitudes across several dimensions. Thirty-two scientists and teachers from the northeastern and south central United States participated in content validity trials. The instrument was field tested with 549 children (92 elementary-school students, 327 middle-school students and 130 high-school students) from 6 rural and suburban school systems located in the northeastern United States to determine inter-item reliability for each dimension. The resulting instrument, entitled My Attitudes Toward Science (MATS), has 40 items that measure four dimensions: (1) Attitude towards the subject of science; (2) Desire to become a scientist; (3) Value of science to society; and (4) Perception of scientists. The MATS, as a multidimensional instrument, can measure several facets of students’ attitude toward science and is designed to be used across grades levels and to be scored easily.     

Threats to the validity of science performance assessments for English language learners

In recent years, at the same time that performance assessments in science have become more popular, the number of English language learners (ELLs) (i.e., students whose native language is other than English) served by the U.S. educational system has also increased rapidly. While the research base is growing in each of these areas independently, little attention has been paid to their intersection. This case study of the use of a science performance assessment with 96 ELLs in five high school science classes investigated the face, construct, and consequential validity of this intersection. Qualitative and quantitative data analyses showed that both teachers and students had an overall favorable response to the assessment, although students' English comprehension and expression skills were determining factors for certain items. While most responses were reliably scored, ELL spelling and syntax on certain responses were significant sources of error. The degree of specificity of teachers' guidance also significantly affected students' scores. Recommendations from this study include increasing the clarity of an assessment's design, allowing ELLs more time to complete assessments, and scoring by raters who are knowledgeable about typical patterns in written English for this student population. Furthermore, it is recommended that the use of performance assessments with ELLs be exploratory until such time as their validity and reliability with this population can be more adequately established. J Res Sci Teach 34: 721–743, 1997.     

Teacher-student classroom interactions among science students of different sex and cultural background

The purpose of this study was to determine associations between science students' perceptions of their interactions with their teachers, the cultural background of students and their attitudinal and cognitive achievement outcomes. A sample of 3,215 students from 158 secondary school science classes in 43 schools completed a survey including the Questionnaire on Teacher Interaction (QTI), an attitude to calss scale, an enquiry skills scale and questions relating to cultural background. The sample was chosen in order tobe representative, though only coeducational classes were used in order to permit an unconfounded test of sex differences. Statistical analyses confirmed the reliability and validity of the QTI for secondary school science students. Generally, the dimensions of the QTI were found to besignificantly associated with student attitude scores. In particular, students' attitude scores were higher in classrooms in which students perceived greater leadership, helping/friendly, and understanding behaviours in their teachers. Correlations between cognitive achievement and interpersonal behaviour were not as strong, but there were positive associations with cooperative behaviours and negative associations with oppositional behaviours. Females perceived their teachers in a more positive way than did males and students from an Asian background tended to perceive their teachers more positively than those from the other cultural groups identified in the study.     

The implementation and development of science process skills in the natural sciences: A case study of teachers' perceptions

Abstract

The notion of scientific literacy, with its concomitant emphasis on science process skills gave rise to this study. This case study research sought to explore the perceptions of four primary school Natural Sciences teachers with respect to the implementation and development of science process skills at classroom level in a disadvantaged urban primary school in South Africa. Qualitative data, generated via focus group interviews and classroom observations were analysed. These data revealed that the implementation and development of science process skills were strongly influenced by the teacher's confidence and understandings of these skills, and that these in turn could be coupled to their present and past experience of science process skills. A key recommendation of this study, that might go some way to addressing these issues, is the implementation of high quality and sustained (over a long period of time) appropriate in-service training of primary school Natural Sciences teachers.