Evaluation of Procedure-Based Scoring for Hands-On Science Assessment

This article evaluates a procedure-based scoring system for a performance assessment (an observed paper towels investigation) and a notebook surrogate completed by fifth-grade students varying in hands-on science experience. Results suggested interrater reliability of scores for observed performance and notebooks was adequate (>.80) with the reliability of the former higher. In contrast, interrater agreement on procedures was higher for observed hands-on performance (.92) than for notebooks (.66). Moreover, for the notebooks, the reliability of scores and agreement on procedures varied by student experience, but this was not so for observed performance. Both the observed-performance and notebook measures correlated less with traditional ability than did a multiple-choice science achievement test. The correlation between the two performance assessments and the multiple-choice test was only moderate (mean = .46), suggesting that different aspects of science achievement have been measured. Finally, the correlation between the observed-performance scores and the notebook scores was .83, suggesting that notebooks may provide a reasonable, albeit less reliable, surrogate for the observed hands-on performance of students.     

An examination of alternate assessment durations when assessing multiple‐skill computational fluency: The generalizability and dependability of curriculum‐based outcomes within the context of educational decisions

The current study extended previous research on curriculum‐based measurement in mathematics (M‐CBM) assessments. The purpose was to examine the generalizability and dependability of multiple‐skill M‐CBM computation assessments across various assessment durations (1, 2, 3, 4, 5, and 6 minutes). Results of generalizability and dependability studies (N = 104 students) suggest that relative interindividual decisions can rely on the results from 1‐minute administrations for low‐stakes decisions and the results of 4‐minute administrations for high‐stakes decisions. Moreover, absolute intraindividual decisions can rely on the results from 4‐minute administrations for low‐stakes decisions and 13‐minute administrations for high‐stakes decisions. The implications and limitations of these results are discussed. © 2005 Wiley Periodicals, Inc. Psychol Schs 42: 615–622, 2005.     

The effect of practice on Raven's Advanced Progressive Matrices

Sixty-seven participants (39 men and 28 women), ranging in age from 26 to 79 years, were administered Raven's Advanced Progressive Matrices (APM) on three occasions. Although total APM scores were found to be highly reliable across the three occasions, the reliabilities of most individual items were extremely low. A single-factor model remained a borderline adequate fit (explaining approximately 20% of the variance) for the interitem correlation matrix on all three occasions. Total APM scores increased significantly across the three occasions (approximately two items per occasion). Improvements in total score across the occasions happened within a context of subjects changing both correct and incorrect responses from the previous occasion. The number of items left unanswered was found to be unrelated to both APM score on any given occasion and the amount of gain in score made across occasions. These findings suggest that the improvements in performance were not based on the acquisition of a strategy design to respond to more items or on the retention of item-specific information, but rather, the improvement reflected learning, something common to the types of items found in the APM.     

The developmental trajectory of intrinsic reading motivation: Measurement invariance,group variations,and implications for reading proficiency

The goal of the present study is to extend previous research on the developmental trajectory of intrinsic reading motivation during early adolescence. Using large-scale panel data on secondary school students in Germany, we examined: (1) the longitudinal measurement invariance of intrinsic reading motivation, (2) the generalizability of the developmental trajectory of intrinsic reading motivation across students’ gender, parental socioeconomic status (SES), and school tracks (academic vs. vocational), and (3) the associations between the developmental trajectory of intrinsic reading motivation and the developmental trajectory of reading proficiency. The scale we used to measure intrinsic reading motivation showed the (strict) measurement invariance across six occasions of measurement from Grades 5 to 10, indicating the high structural similarity (e.g., factor loadings, intercepts) of intrinsic reading motivation during early adolescence. Our analyses of latent growth curve models also confirm previous findings that students tend to experience a steady and significant linear decline in intrinsic reading motivation from Grades 5 to 10. This developmental decline also seems to be more pronounced in size (Δ =  − 0.772, p < .001) than previously reported. The developmental decline in intrinsic reading motivation was observed irrespective of students’ gender, parental SES, and school tracks. Male students expressed lower mean-levels of intrinsic reading motivation across the waves and exhibited a steeper motivational decline compared to female students. Despite mean-level differences across the waves, students showed similar degrees of a motivational decline across parental SES and school tracks. Finally, the larger decline in students’ intrinsic reading motivation was associated with the smaller growth of their reading proficiency from Grades 5 to 10. Our study provides further support for the high prevalence of the developmental decline in intrinsic reading motivation during early adolescence, its generalizability across students’ demographic characteristics, and its implications for the development of reading proficiency.     

Learning the Cell Structures with Three-Dimensional Models: Students’ Achievement by Methods, Type of School and Questions’ Cognitive Level

The cell topic was taught to 9th-grade students in three modes of instruction: (a) students “hands-on,” who constructed three-dimensional cell organelles and macromolecules during the learning process; (b) teacher demonstration of the three-dimensional model of the cell structures; and (c) teaching the cell topic with the regular learning material in an expository mode (which use one- or two-dimensional cell structures as are presented in charts, textbooks and microscopic slides). The sample included 669, 9th-grade students from 25 classes who were taught by 22 Biology teachers. Students were randomly assigned to the three modes of instruction, and two tests in content knowledge in Biology were used. Data were treated with multiple analyses of variance. The results indicate that entry behavior in Biology was equal for all the study groups and types of schools. The “hands-on” learning group who build three-dimensional models through the learning process achieved significantly higher on academic achievements and on the high and low cognitive questions’ levels than the other two groups. The study indicates the advantages students may have being actively engaged in the learning process through the “hands-on” mode of instruction/learning.     

Estimating reliability of school-level scores using multilevel and generalizability theory models

The purpose of this study was to investigate the methods of estimating the reliability of school-level scores using generalizability theory and multilevel models. Two approaches, ‘student within schools’ and ‘students within schools and subject areas,’ were conceptualized and implemented in this study. Four methods resulting from the combination of these two approaches with generalizability theory and multilevel models were compared for both balanced and unbalanced data. The generalizability theory and multilevel models for the ‘students within schools’ approach produced the same variance components and reliability estimates for the balanced data, while failing to do so for the unbalanced data. The different results from the two models can be explained by the fact that they administer different procedures in estimating the variance components used, in turn, to estimate reliability. Among the estimation methods investigated in this study, the generalizability theory model with the ‘students nested within schools crossed with subject areas’ design produced the lowest reliability estimates. Fully nested designs such as (students:schools) or (subject areas:students:schools) would not have any significant impact on reliability estimates of school-level scores. Both methods provide very similar reliability estimates of school-level scores.     

Withholding answers during hands-on scientific investigations? Comparing effects on developing students’ scientific knowledge,reasoning, and application

As more concerns have been raised about withholding answers during science teaching, this article argues for a need to detach ‘withholding answers’ from ‘hands-on’ investigation tasks. The present study examined students’ learning of light-related content through three conditions: ‘hands-on’ + no ‘withholding’ (hands-on only: HO), ‘hands-on’ + ‘withholding’ (hands-on investigation with answers withheld: HOW), and no ‘hands-on’ + no ‘withholding’ (direction instruction: DI). Students were assessed in terms of how well they (1) knew the content taught in class; (2) reasoned with the learned content; and (3) applied the learned content to real-life situations. Nine classes of students at 4th and 5th grades, N?=?136 in total, were randomly assigned to one of the three conditions. ANCOVA results showed that students in the hands-on only condition reasoned significantly better than those in the other two conditions. Students in this condition also seemed to know the content fairly better although the advance was not significant. Students in all three conditions did not show a statistically significant difference in their ability to apply the learned content to real-life situations. The findings from this study provide important contributions regarding issues relating to withholding answers during guided scientific inquiry.     

Hands-on Activities and Their Influence on Students’ Interest

This study investigates the influence of hands-on activities on students’ interest. We researched whether students with experience in specific hands-on activities show higher interest in these activities than students without experience. Furthermore, the relationship between the quality of the hands-on experience and interest in the respective activity was examined. In total, 28 typical hands-on activities of biology education were considered. The activities were divided into the categories experimentation, dissection, work with microscopes, and classification. A total of 141 students from the 11th grade completed questionnaires on interest in the hands-on activities, their experience with each activity, and the quality of the respective experience. Students’ interest in experimenting, working with microscopes, dissecting and classifying tends to benefit from performing hands-on activities. However, findings indicated that the performance of various hands-on activities can influence students’ interest differently. For seven hands-on activities, we identified a positive effect of hands-on experience on interest, while in one case, practical work appeared to have influenced students’ interest negatively. However, for most hands-on activities, no effect of experience on interest was found. The quality of hands-on experiences showed positive correlations with interest in the respective hands-on activities. Therefore, this paper argues in favour of designing biology lessons that allow for experiences with hands-on activities that also interest students. Our findings underline the necessity of investigating the effects of various hands-on activities in a differentiated manner.     

An analysis of frequency of hands-on experience and science achievement

A variance analysis of the relation between the amount of time students spent experiencing hands-on science and science achievement was performed. Data collected by the National Education Longitudinal Study of 1988 on a nationally representative sample of eighth-grade students were analyzed. Student achievement in science was measured by a cognitive test battery developed by the Educational Testing Service. Information regarding the frequency of hands-on experience was collected through a self-administered teacher questionnaire, which included a series of questions specific to the science curriculum. From the analysis it was concluded that significant differences existed across the hands-on frequency variable with respect to science achievement. Specifically, students who engaged in hands-on activities every day or once a week scored significantly higher on a standardized test of science achievement than students who engaged in hands-on activities once a month, less than once a month, or never. © 1996 John Wiley & Sons, Inc.     

Using generalizability analysis to estimate parameters for anatomy assessments: A multi‐institutional study

With integrated curricula and multidisciplinary assessments becoming more prevalent in medical education, there is a continued need for educational research to explore the advantages, consequences, and challenges of integration practices. This retrospective analysis investigated the number of items needed to reliably assess anatomical knowledge in the context of gross anatomy and histology. A generalizability analysis was conducted on gross anatomy and histology written and practical examination items that were administered in a discipline‐based format at Indiana University School of Medicine and in an integrated fashion at the University of Alabama School of Medicine and Rush University Medical College. Examination items were analyzed using a partially nested design in which items were nested within occasions (i:o) and crossed with students (s). A reliability standard of 0.80 was used to determine the minimum number of items needed across examinations (occasions) to make reliable and informed decisions about students' competence in anatomical knowledge. Decision study plots are presented to demonstrate how the number of items per examination influences the reliability of each administered assessment. Using the example of a curriculum that assesses gross anatomy knowledge over five summative written and practical examinations, the results of the decision study estimated that 30 and 25 items would be needed on each written and practical examination to reach a reliability of 0.80, respectively. This study is particularly relevant to educators who may question whether the amount of anatomy content assessed in multidisciplinary evaluations is sufficient for making judgments about the anatomical aptitude of students. Anat Sci Educ 10: 109–119. © 2016 American Association of Anatomists.     

Academic discipline and generalizability of student evaluations of instruction

Previous research on the generalizability of student ratings of instruction has raised questions about the effects of academic discipline and item types on the generalizability of these data for making relative decisions about instructors and about courses. In particular, although student evaluation data appear to provide a reasonable basis for making decisions about instructors when generalizing across courses and students, when course is the object of measurement, the data appear to be less generalizable. It was suggested in the literature that this may be due to the type of evaluation items used or it may be due to academic discipline differences in the type of courses selected for study. This study used Biglan's (1973a) model for classifying disciplines along the dimensions of paradigmatic/preparadigmatic (hard/soft) and pure/applied. A nested sampling procedure yielded two sample types: courses within teachers, in which individual instructors taught more than one course; and teachers within courses, in which individual courses were taught by more than one instructor. For each sample type, evaluation forms for twenty courses within each discipline classification were sought. The evaluation items for this study were classified as measuring six dimensions of instruction: organization, breadth of coverage, group interaction, enthusiasm, grading, and individual rapport. Generalizability and decision studies were conducted in which, for one sample, teacher was the object of measurement, and for the second sample, course was the object of measurement. Results indicated that reliable decisions about instructors could reasonably be made from all six of the evaluation dimensions; however, reliability for course decisions varied greatly with the evaluation dimension, being highest for breadth of coverage and lowest for grading. The same general pattern was noted for the paradigmatic disciplines and the preparadigmatic-applied disciplines but not for the preparadigmatic-pure disciplines. It is suggested that a single evaluation instrument may not be uniformly applicable to all discipline areas.     

Examining the Sustained Effects of Reading Recovery

Though the immediate effect of Reading Recovery (RR) is both strong and well established, the longer term or sustained effect has been less studied and the evidence regarding it has been less conclusive. Michigan Reading Recovery students (n = 328) were compared to control students (n = 264) while in first (2009–2010), third (2011), and fourth grades (2012), using propensity score matching to generate 3 levels of eligibility. Although the immediate effect measured at mid-year of first grade on the Observation Survey was large (1.17), the effect by the end of first grade on the same measure was .51, and by third grade, the effect was .16 on the state reading test. The overall effect completely diminished by fourth grade, but it was significant (.35) for the most eligible students in reading, and for moderately eligible (.34) and most eligible students (.35) in writing. The sustained effect overall was present but diminished by third grade, and was sustained into fourth grade for those students at greater risk. The findings suggest that RR instruction should be better tailored to the initial literacy profiles of individual students to maximize the longevity of the effect for all participants.     

The effects of hands-on and teacher demonstration laboratory methods on science achievement in relation to reasoning ability and prior knowledge

The present study compared the relative effects of hands-on and teacher demonstration laboratory methods on declarative knowledge (factual and conceptual) and procedural knowledge (problem-solving) achievement. Of particular interest were (a) whether these relationships vary as a function of reasoning ability and (b) whether prior knowledge and reasoning ability predict student achievement. Ninth-grade physical science students were randomly assigned to classes taught by either a hands-on or a teacher demonstration laboratory method. Students' reasoning ability and prior knowledge of science were assessed prior to the instruction. The two instructional methods resulted in equal declarative knowledge achievement. However, students in the hands-on laboratory class performed significantly better on the procedural knowledge test than did students in the teacher demonstration class. These results were unrelated to reasoning ability. Prior knowledge significantly predicted performance on the declarative knowledge test. Both reasoning ability and prior knowledge significantly predicted performance on the procedural knowledge test, with reasoning ability being the stronger predictor.     

Blended inquiry with hands-on and virtual laboratories: the role of perceptual features during knowledge construction

This study examined the characteristics of virtual and hands-on inquiry environments for the development of blended learning in a popular domain of bio-nanotechnology: the separation of different-sized DNA fragments using gel-electrophoresis, also known as DNA-fingerprinting. Since the latest scientific developments in nano- and micro-scale tools are based on molecular movement in electric fields, gel electrophoresis is an excellent model for learning-related concepts and processes. This study employed two environments (a 2D virtual laboratory (VRL) and a hands-on laboratory (HOL)) and documented the benefit of using VRLs to ground students' knowledge construction, before more complex, hands-on investigation. A comparative analysis explored how the perceptual features of the two learning environments supported students in designing experiments, evaluating data from experimental trials and reasoning for the mechanisms by which these data came about. The findings provide evidence for the design of blended inquiry-learning environments that integrate virtual and hands-on laboratories.     

Students Who Demonstrate Strong Talent and Interest in STEM Are Initially Attracted to STEM through Extracurricular Experiences

What early experiences attract students to pursue an education and career in science, technology, engineering, and mathematics (STEM)? Does hands-on research influence them to persevere and complete a major course of academic study in STEM? We evaluated survey responses from 149 high school and undergraduate students who gained hands-on research experience in the 2007–2013 Aspiring Scientists Summer Internship Programs (ASSIP) at George Mason University. Participants demonstrated their strong interest in STEM by volunteering to participate in ASSIP and completing 300 h of summer research. The survey queried extracurricular experiences, classroom factors, and hands-on projects that first cultivated students’ interest in the STEM fields, and separately evaluated experiences that sustained their interest in pursuing a STEM degree. The majority of students (65.5%, p < 0.0001) reported extracurricular encounters, such as the influence of a relative or family member and childhood experiences, as the most significant factors that initially ignited their interest in STEM, while hands-on lab work was stated as sustaining their interest in STEM (92.6%). Based on these findings collected from a cohort of students who demonstrated a strong talent and interest in STEM, community-based programs that create awareness about STEM for both children and their family members may be key components for igniting long-term academic interest in STEM.     

How accurate are ESL students’ holistic writing scores on large-scale assessments?—A generalizability theory approach

Using generalizability theory, this study examined both the rating variability and reliability of ESL students’ writing in the provincial English examinations in Canada. Three years’ data were used in order to complete the analyses and examine the stability of the results. The major research question that guided this study was: Are there any differences between the rating variability and reliability of the writing scores assigned to ESL students and to Native English (NE) students in the writing components of the provincial examinations across three years? A series of generalizability studies and decision studies was conducted. Results showed that differences in score variation did exist between ESL and NE students when adjudicated scores were used. First, there was a large effect for both language group and person within language-by-task interaction. Second, the unwanted residual variance component was significantly larger for ESL students than for NE students in all three years. Finally, the desired variance associated with the object of measurement was significantly smaller for ESL students than for NE students in one year. Consequently, the observed generalizability coefficient for ESL students was significantly lower than that for NE students in that year. These findings raise a potential question about the fairness of the writing scores assigned to ESL students.     

Two-year follow-up of a code-oriented intervention for lower-skilled first-graders: the influence of language status and word reading skills on third-grade literacy outcomes

For 2 years we followed lower-performing English learner (EL) and native English speaking (non-EL) students who participated in an efficacy trial of a supplemental first-grade code-oriented intervention implemented by paraeducators. At the end of grade three, across all students (n = 180 of the original 187 students), treatment effects were maintained on word reading (approximate d = .45), spelling (.36) and reading comprehension (.24). However, treatment effects tended to be smaller for EL students, and were significantly smaller for spelling in particular. While pretest grade one word reading did not moderate treatment response for either ELs or non-ELs, it was found to strongly predict all three end-of- grade-three outcomes, although to a lesser extent for ELs on reading comprehension. Findings add support to previous research on the benefits of early code-oriented tutoring.     

Academic Self‐Perceptions,Effort, and Strategy Use in Students with Learning Disabilities: Changes Over Time

This study investigated changes in teachers' and students' perceptions of students' effort, strategy use, and academic difficulties when strategy instruction was infused into the classroom curriculum. The sample consisted of 201 students with learning disabilities, 210 average achievers, and 57 teachers from Grades 4–9 in two urban and suburban communities. After six months of classroom‐based strategy instruction, students with learning disabilities reported more consistent use of strategies with their schoolwork and perceived themselves as struggling less in reading, writing, and spelling. Teachers perceived the students with learning disabilities as more strategic and as applying more effort to their schoolwork. Teachers also perceived their students as showing significant improvements in spelling, regardless of whether they had learning disabilities. These findings extended the results of previous investigations and indicated the small, positive impact of classroom‐based strategy instruction. Further investigations are critical to evaluate the generalizability of these findings.