Evaluation of Procedure-Based Scoring for Hands-On Science Assessment

This article evaluates a procedure-based scoring system for a performance assessment (an observed paper towels investigation) and a notebook surrogate completed by fifth-grade students varying in hands-on science experience. Results suggested interrater reliability of scores for observed performance and notebooks was adequate (>.80) with the reliability of the former higher. In contrast, interrater agreement on procedures was higher for observed hands-on performance (.92) than for notebooks (.66). Moreover, for the notebooks, the reliability of scores and agreement on procedures varied by student experience, but this was not so for observed performance. Both the observed-performance and notebook measures correlated less with traditional ability than did a multiple-choice science achievement test. The correlation between the two performance assessments and the multiple-choice test was only moderate (mean = .46), suggesting that different aspects of science achievement have been measured. Finally, the correlation between the observed-performance scores and the notebook scores was .83, suggesting that notebooks may provide a reasonable, albeit less reliable, surrogate for the observed hands-on performance of students.     

Questioning Questions: Elementary Teachers’ Adaptations of Investigation Questions Across the Inquiry Continuum

Questioning is a central practice in science classrooms. However, not every question translates into a “good” science investigation. Questions that drive science investigations can be provided by many sources including the teacher, the curriculum, or the student. The variations in the source of investigation questions were explored in this study. A dataset of 120 elementary science classroom videos and associated lesson plans from 40 elementary teachers (K-5) across 21 elementary school campuses were scored on an instrument measuring the amount of teacher-direction or student-direction of the lessons’ investigation questions. Results indicated that the investigation questions were overwhelmingly teacher directed in nature, with no opportunities for students to develop their own questions for investigation. This study has implications for researchers and practitioners alike, calling attention to the teacher-directed nature of investigation questions in existing science curriculum materials, and the need for teacher training in instructional strategies to adapt their existing curriculum materials across the continuum of teacher-directed and student-directed investigation questions. Teachers need strategies for adapting the teacher-directed questions provided in their existing curriculum materials in order to allow students the opportunity to engage in this essential scientific practice.     

Cognitive Research and Elementary Science Instruction: From the Laboratory,to the Classroom,and Back

Can cognitive research generate usable knowledge for elementary science instruction? Can issues raised by classroom practice drive the agenda of laboratory cognitive research? Answering yes to both questions, we advocate building a reciprocal interface between basic and applied research. We discuss five studies of the teaching, learning, and transfer of the “Control of Variables Strategy” in elementary school science. Beginning with investigations motivated by basic theoretical questions, we situate subsequent inquiries within authentic educational debates—contrasting hands-on manipulation of physical and virtual materials, evaluating direct instruction and discovery learning, replicating training methods in classroom, and narrowing science achievement gaps. We urge research programs to integrate basic research in “pure” laboratories with field work in “messy” classrooms. Finally, we suggest that those engaged in discussions about implications and applications of educational research focus on clearly defined instructional methods and procedures, rather than vague labels and outmoded “-isms.”     

Diagnosing and altering three aristotelian alternative conceptions in dynamics: Microcomputer simulations of scientific models

This is a report on the investigation of a microcomputer-based system for the diagnosis and remediation of three Aristotelian alternative conceptions of force and motion held by eighth-grade physical science students. Diagnosis and posttesting were done with computer-displayed, graphics-based, multiple-choice questions. The two remediation simulations were designed to present scientific idealizations and to be perceived by the student as anomalous to the three alternative conceptions. Structured interviews were employed at several points during the study to obtain indications of the conceptions of force and motion of students with different achievement rankings, as well as to determine the students' reactions to the computer pretest questions or the simulations. A student's possession of alternative conceptions was unrelated to whether the student was a strong or weak learner of science. Students who were currently studying dynamics in their classes exhibited a very different pattern of nonscientific answers on the computer diagnostic test than did students who had completed that topic. The completed students who were selected for possession of alternative conceptions were facilitated by the computer simulations in altering their naive conceptions to a significant degree.     

On the Stability of Performance Assessments

This study examined the stability of scores on two types of performance assessments, an observed hands-on investigation and a notebook surrogate. Twenty-nine sixth-grade students in a hands-on inquiry-based science curriculum completed three investigations on two occasions separated by 5 months. Results indicated that: (a) the generalizability across occasions for relative decisions was, on average, moderate for the observed investigations (.52) and the notebooks (.50); (b) the generalizability for absolute decisions was only slightly lower; (c) the major source of measurement error was the person by occasion (residual) interaction; and (d) the procedures students used to carry out the investigations tended to change from one occasion to the other.     

Individual Inquiry Investigations in an Elementary Science Methods Course

This study explored how preservice teachers grew in their understanding of inquiry science through conducting an in-depth individual inquiry investigation in their science methods course. Elementary preservice teachers’ understanding about inquiry investigations, their attitudes towards the assignment, and their views towards using individual inquiry investigations in their own classrooms were studied through written reflections, science notebook writings, and researcher field notes. After conducting an out-of-class inquiry investigation based on their own questions and design, the preservice teachers demonstrated a variety of strengths in their understandings of inquiry-based teaching, their attitudes towards doing science, and their projected use of inquiry in their own classrooms. Weak areas included forming researchable questions and superficial experimental design.     

Teacher Practices and Middle‐school Science Achievements

We report on the construction and application on an instrument entitled the “Science Achievement Influences Survey” to assess combined effects of student attitudes about science, peer interaction, and home support, and the frequency of student‐centred and teacher‐centred instructional practices on student achievement. Controlling for pre‐test content knowledge, results indicated that student‐centred teaching practices have a positive association with student achievement (p < .01; i.e., group experiments) and a negative association with teacher‐centred teaching practices (p < .01; i.e., copying notes). Additionally, student attitudes about science were positively associated with student‐centred teaching practices (p < .01) and negatively associated with teacher‐centred teaching practices (p < .01). Most significantly, this study documents the predicted gains in science achievement associated with frequency of specific instructional practices used by middle‐school science teachers. Especially noteworthy and significant is the finding that near‐daily implementation of group experiments and reduction of extensive note‐copying during class yield the greatest positive impact on student achievement. Outside of school, peer interaction and home support were not significantly associated (p > .05) with student achievement. The student sample included 611 middle‐school science students with a wide range of socioeconomic and cultural backgrounds.     

School Size Effects on Achievement in Secondary Education Evidence from the Netherlands,Sweden and the USA 1

ABSTRACT

In this article the results of an investigation into the relation between school size and achievement are reported. The findings relate to mathematics achievement in Dutch, Swedish and American secondary education and to science achievement in the Netherlands. The analyses sought to provide an answer to the following questions:

(1) Is school size related to achievement independently of student background characteristics such as sex, achievement motivation, socio‐economic status and cognitive aptitude? (2) Is the effect of school size related to any of the aforementioned background characteristics? (3) Does the effect of school size on achievement differ between the educational systems of the Netherlands, Sweden and the USA? (4) Is the effect of school size the same for different measures of student achievement (mathematics versus science)?

It was hypothesized that school size would be most strongly related to achievement in the USA. The analyses, however, revealed little empirical evidence for the existence of school size effects on achievement in any of the three countries, possibly because school size and curriculum comprehensiveness are not strongly related in these countries.

Because the investigations involved the analysis of five separate datasets, the research outcomes revealed some useful additional information with respect to the robustness of the detected relations between the five covariates and student achievement.

     

Science for All: Engaging Students with Special Needs in and About Science

The notion of “science for all” suggests that all students—irrespective of achievement and ability—should engage in opportunities to understand the practice and discourse of science. Improving scientific literacy is an intrinsic goal of science education, yet current instructional practices may not effectively support all students, in particular, students with special needs. Argument‐based inquiry approaches, such as the Science Writing Heuristic (SWH), require all students to construct their scientific understandings by engaging in investigations and negotiating their ideas in multiple contexts, such as discussions and writing. Various SWH studies demonstrated that students engaged in appropriating the language, culture, practice, and dispositions of science generally improved their critical thinking and standardized test scores. The implementation of such an approach has several implications for science and special education research and practice, including how learning environments should be established to encourage the inclusion of all students’ ideas, as well as how scaffolded supports can and should be used to support science learning.     

“I'm just not that great at science”: Science self-efficacy in arts and communication students

Research in science education confirms the importance of self-efficacy in students' persistence and success in the sciences. The current study examined the role of science self-efficacy in nonspecialist, arts and communication-oriented students encountering science in a general education context. Participants (N = 275) completed a beginning- and end-of-semester survey including a Science Self-Efficacy Scale, a “connection to science” measure—the Inclusion of Science in Self Scale—and a Science Anxiety Scale. Participants also responded to two open-ended “sources of science efficacy” questions, and provided background/demographic information and access to their academic records. Results showed a significant increase in science self-efficacy and connection to science—although no change in science anxiety—over the course of the semester. The observed shift in self-efficacy for minority and international students was of particular note. These students started the course with lower confidence but, by the end of the semester, reported comparable science self-efficacy, and achieved similar grades to their White/Non-Hispanic and US resident classmates. Contrary to expectations, science self-efficacy did not predict performance in the class. However, students' self-reported sources of efficacy indicated increased confidence in using science in daily life, and confirmed the value of mastery experiences and of personally meaningful, student-centered course design in scaffolding student confidence. Results are discussed in terms of the individual and instructional factors that support science self-efficacy and student success in this unique, general education science environment.     

Note on Sources of Sampling Variability in Science Performance Assessments

In 1993, we reported in Journal of Educational Measurement that task-sampling variability was the Achilles' heel of science performance assessment. To reduce measurement error, tasks needed to be stratified before sampling, sampled in large number, or possibly both. However, Cronbach, Linn, Brennan, & Haertel (1997) pointed out that a task-sampling interpretation of a large person x task variance component might be incorrect. Task and occasion sampling are confounded because tasks are typically given on only a single occasion. The person x task source of measurement error is then confounded with the pt x occasion source. If pto variability accounts for a substantial part of the commonly observed pt interaction, stratifying tasks into homogenous subsets—a cost-effective way of addressing task sampling variability—might not increase accuracy. Stratification would not address the pro source of error. Another conclusion reported in JEM was that only direct observation (DO) and notebook (NB) methods of collecting performance assessment data were exchangeable; computer simulation, short-answer, and multiple-choice methods were not. However, if Cronbach et al. were right, our exchangeability conclusion might be incorrect. After re-examining and re-analyzing data, we found support for Conbach et al. We concluded that large task-sampling variability was due to both the person x task interaction and person x task x occasion interaction. Moreover, we found that direct observation, notebook and computer simulation methods were equally exchangeable, but their exchangeability was limited by the volatility of student performances across tasks and occasions.     

Relationship Between Language and Concept Science Notebook Scores of English Language Learners and/or Economically Disadvantaged Students

Despite research interest in testing the effects of literacy-infused science interventions in different contexts, research exploring the relationship, if any, between academic language and conceptual understanding is scant. What little research exists does not include English language learners (ELLs) and/or economically disadvantaged (ED) student samples—students most at risk academically. This study quantitatively determined if there exists a relationship, and if so, how strong of a relationship, between ELL and ED students’ academic language and conceptual understanding based on science notebook scores used in a larger science and literacy-infused intervention with a sample of culturally diverse students. The study also considered strengths of relationships between language and concept science notebook scores within student language status groups (ELL, former ELL, and English speaking). Correlational analyses noted positive, large, and significant correlations between students’ language and concept scores overall, with the largest correlations for science notebook entries using more academic language. Large correlations also existed for ELL student entries at the end of the school year. Implications of the findings for future research and practice in science classrooms including literacy interventions, such as science notebooks, with populations of culturally diverse students are discussed.

     

Relationship Between Language and Concept Science Notebook Scores of English Language Learners and/or Economically Disadvantaged Students

Despite research interest in testing the effects of literacy-infused science interventions in different contexts, research exploring the relationship, if any, between academic language and conceptual understanding is scant. What little research exists does not include English language learners (ELLs) and/or economically disadvantaged (ED) student samples—students most at risk academically. This study quantitatively determined if there exists a relationship, and if so, how strong of a relationship, between ELL and ED students’ academic language and conceptual understanding based on science notebook scores used in a larger science and literacy-infused intervention with a sample of culturally diverse students. The study also considered strengths of relationships between language and concept science notebook scores within student language status groups (ELL, former ELL, and English speaking). Correlational analyses noted positive, large, and significant correlations between students’ language and concept scores overall, with the largest correlations for science notebook entries using more academic language. Large correlations also existed for ELL student entries at the end of the school year. Implications of the findings for future research and practice in science classrooms including literacy interventions, such as science notebooks, with populations of culturally diverse students are discussed.     

The Effect of REACT and Computer-Assisted Instruction Model in 5E on Student Achievement of the Subject of Acids,Bases and Salts

This research was carried out to investigate the effect of the 5E learning method and 5E method enriched with REACT strategy on student achievement in science education. The research was carried out with 64 first-year students (30 in the experimental groups, 34 in the control groups) in the Department of Primary Education at a state university in the spring semester of 2017–2018 in the context of “Acids, Bases and Salts”. The pre-test/post-test quasi-experimental design with a control group was used in this study. In the experimental group, the 5E method enriched with the REACT strategy was used while the computer-assisted 5E instruction method was used in the control group. The data were collected by means of a “Subject Achievement Test”, consisting of open-ended questions. The dependent t test results of the experimental and control group revealed a significant difference in favour of the post-test in both groups. According to the independent t test results of the groups, the 5E method enriched with REACT strategy was found to be more effective in order to improve student achievement compared to computer-assisted 5E learning method.

     

Student-Generated Content: Enhancing learning through sharing multiple-choice questions

The relationship between students' use of PeerWise, an online tool that facilitates peer learning through student-generated content in the form of multiple-choice questions (MCQs), and achievement, as measured by their performance in the end-of-module examinations, was investigated in 5 large early-years science modules (in physics, chemistry and biology) across 3 research-intensive UK universities. A complex pattern was observed in terms of which type of activity (writing, answering or commenting on questions) was most beneficial for students; however, there was some evidence that students of lower intermediate ability may have gained particular benefit. In all modules, a modest but statistically significant positive correlation was found between students' PeerWise activity and their examination performance, after taking prior ability into account. This suggests that engaging with the production and discussion of student-generated content in the form of MCQs can support student learning in a way that is not critically dependent on course, institution, instructor or student.     

Let the Questions Be Your Guide: MBE as Interdisciplinary Science

From its inception, the field of Mind, Brain, and Education (MBE) has been conceived as an interdisciplinary science, and with good reason: The phenomena the field aims to understand often arise from interactions among multiple factors, span levels of analysis, and are context dependent. In this article, we argue that to reach its potential as an interdisciplinary science—and in order to explain such complex phenomena—MBE must be fundamentally organized around meaningful, discipline‐spanning questions, and the questions must determine tools and research methods (not the other way around). Using examples from three central questions in MBE—“who,”“when,” and “how”—we highlight the limits of single disciplines, and the value of a question‐driven interdisciplinary approach in MBE, with respect to questions that can be asked, the perspectives that can be considered, and the array of methods, tools, and models that can be made available. We believe that the future is bright for MBE, and that the field has a unique opportunity to provide meaningful answers to some of the most difficult questions in education today. However, realizing this potential depends on, as a first step, allowing the questions themselves to drive the field's work moving forward.     

The effect of A teacher questioning strategy training program on teaching behavior,student achievement,and retention

The use of questions in the classroom has been employed throughout the recorded history of teaching. One still hears the term “Socratic method” during discussions of questioning procedures. The use of teacher questions is presently viewed as a viable procedure for effective instruction. This study was conducted to investigate the feasibility of training teachers in the use of a questioning technique and the resultant effect upon student learning. The Post-Test Only Control Group Design was used in randomly assigning teachers and students to experimental and control groups. A group of teachers was trained in the use of a specific questioning technique. Follow-up periodic observations were made of questioning technique behavior while teaching science units to groups of students. Post-unit achievement tests were administered to the student groups to obtain evidence of a relationship between the implementation of specific types of teacher questions and student achievement and retention. Analysis of observation data indicated a higher use of managerial and rhetorical questions by the control group than the experimental group. The experimental group employed a greater number of recall and data gathering questions as well as higher order data processing and data verification type questions. The student posttest achievement scores for both units of instruction were greater for the experimental groups than for the control groups. The retention scores for both units were Beater for the experimental groups than for the control groups.     

Using data for decision-making: perspectives from 16 principals in Michigan, USA

In response to the vast amounts of data associated with the accountability movement and the rhetoric of data-informed decision-making, we interviewed 16 principals to find out what streams of data they used and what decisions they made by using the data. We found that: (a) student achievement data are predominantly used to the extent of neglecting other streams of data such as student and community background data and school process data; (b) student achievement data are used more for accountability purposes—for assessing “of” rather than “for” the learning; (c) different streams of data are rarely used together to derive rich meaning for decision-making; and (d) school districts differ in the extent to which their principals use data to improve curriculum and instruction. The study pointed both to the challenges and to the opportunities of making data-informed decisions to improve our schools.     

Educational Assessment as a Promising Area for Psychometric Research

Tests of educational achievement typically present items in the multiple-choice format. Some achievement test items may be so "saturated with aptitude" (Willingham, 1980) as to be insensitive to skills acquired through education. Multiple-choice tests are ill-suited for assessing productive thinking and problem-solving skills, skills that often constitute important objectives of education. Viewed as incentives for learning, multiple-choice tests may impede student progress toward these objectives. There is need for accelerated research to develop alternatives to multiple-choice achievement tests, with content selected to match the specified educational objectives.