Empirical versus subjective procedures for identifying gender differences in science test items

When judgmental and statistical procedures are both used to identify potentially gender-biased items in a test, to what extent do the results agree? In this study, both procedures were used to evaluate the items in a statewide, 78-item, multiple-choice test of science knowledge. Only one item was flagged by the sensitivity reviewers as being potentially biased, but this item was not flagged by the statistical procedure. None of the nine items flagged by the Mantel-Haenszel procedure were flagged by the sensitivity reviewers. Eight of the nine statistically flagged items were differentially easier for males. Four of these eight measured the same category of objectives. The authors conclude that both judgmental and statistical procedures provide useful information and that both should be used in test construction. They caution readers that content-validity issues need to be addressed when making decisions based on the results of either procedure.     

Elementary teachers’ science subject matter knowledge across the teacher career cycle

Although widely touted as important, there is little evidence regarding the influence of teaching experience on elementary teachers' science subject matter knowledge (SMK). To better understand this phenomenon, we administered an assessment of science topics taught in the fifth and sixth grades to 169 preservice teachers, 231 fifth-grade teachers, and 208 sixth-grade teachers. We then compared the mean scores of teachers at different stages in the career cycle using one-way and two-way ANOVA and explored the relationship between SMK scores and years of teaching experience using regression analysis. Findings indicate that (i) being assigned to a specific grade level had an impact on teachers' SMK for topics included in the grade level, (ii) teachers' SMK scores were lower later in their careers for both science topics they had never taught and for science topics they were responsible for teaching, and (iii) results differed for fifth and sixth grades. This study adds to the existing literature through the examination of a large sample of elementary teachers, with teachers of varying years of experience, while focusing on the science topics these teachers are responsible for teaching. The results of this study provide strong evidence that (i) years of experience teaching specific science topics is associated with the development of teachers' knowledge of these topics, and (ii) teaching experience, what teachers do in the context of their everyday practice, can be an effective means of self-directed learning for teachers. There is also evidence that this influence is not uniform across years of teaching experience or science topics taught. These findings raise important implications about future research into the mechanism of SMK development through teaching experience and teacher grade-level assignments.     

A comparison of generalizability theory and many-facet Rasch measurement in an analysis of college sophomore writing

A pilot study was conducted to evaluate and improve the rating procedure proposed for use in a research effort designed to assess the essay writing ability of college sophomores.Generalizability theory and the Many-Facet Rasch Model were each used to (a) estimate potential sources of error in the rating, (b) to obtain reliability estimates, and (c) to make recommendations for improving the rating process. Variance due to Task (writing prompt) and the Person-by-Task interaction were high while the variance attributable to Raters and Occasion was low. Twenty-two percent of the variability in the ratings was unexplained. The common and unique features of generalizability theory and the Many-Facet Rasch Model are described, and the advantages and disadvantages of each are discussed.     

Rethinking exams and letter grades: how much can teachers delegate to students?

In this article we report a 3-yr study of a large-enrollment Cell Biology course focused on developing student skill in scientific reasoning and data interpretation. Specifically, the study tested the hypothesis that converting the role of exams from summative grading devices to formative tools would increase student success in acquiring those skills. Traditional midterm examinations were replaced by weekly assessments administered under test-like conditions and followed immediately by extensive self, peer, and instructor feedback. Course grades were criterion based and derived using data from the final exam. To alleviate anxiety associated with a single grading instrument, students were given the option of informing the grading process with evidence from weekly assessments. A comparative analysis was conducted to determine the impact of these design changes on both performance and measures of student affect. Results at the end of each year were used to inform modifications to the course in subsequent years. Significant improvements in student performance and attitudes were observed as refinements were implemented. The findings from this study emphasized the importance of prolonging student opportunity and motivation to improve by delaying grade decisions, providing frequent and immediate performance feedback, and designing that feedback to be maximally formative and minimally punitive.     

Academic performance,course completion rates,and student perception of the quality and frequency of interaction in a virtual high school

This study examined the relationship between students’ perceptions of teacher–student interaction and academic performance at an asynchronous, self-paced, statewide virtual high school. Academic performance was measured by grade awarded and course completion. There were 2269 students who responded to an 18-item survey designed to measure student perceptions on the quality and frequency of teacher–student interaction. Quality of interaction was subdivided into three constructs representing feedback, procedural, and social interaction. A confirmatory factor analysis helped to establish the fit of the statistical model for teacher–student interaction. Hierarchical logistical regression indicates that an increase in the quality and frequency of interaction resulted in an increased likelihood of course completion but had minimal influence on grade awarded. The estimated effect for quality and frequency composite items on completion was .83 and .56 respectively. Low practical significance of student–teacher interaction on grade awarded may be the result of mastery-based teaching approaches that skew grades for the completers toward the high end.     

The development and application of affective assessment in an upper‐level cell biology course

This study exemplifies how faculty members can develop instruments to assess affective responses of students to the specific features of the courses they teach. Means for assessing three types of affective responses are demonstrated: (a) student attitudes towards courses with differing instructional objectives and methodologies, (b) student self‐efficacy (confidence) in completing tasks common to practicing experts, and (c) interests in subject‐specific topics and associated intellectual skills. The iterative processes used in refining the instruments and performing the statistical analyses of their effectiveness are detailed. An analysis of data obtained using these instruments is also included. Positive attitudes towards courses emphasizing analytical thinking increased significantly over the course of a 14‐week semester, as did the measures of self‐efficacy. Despite the rigorous analytical nature of the course experience, the initial strong interest in cell biology topics and higher order thinking skills remained unchanged. There were no significant differences based on gender in any of the affective measures. We discuss the apparent dichotomy, revealed in these assessments, between students' idealistic, academic attitudes towards the course's effectiveness and their opinions of its effects on them personally. We conclude by reporting how insights garnered both from these formal assessments and from anecdotal communications have prompted experimental modifications in the design and conduct of the course. © 2007 Wiley Periodicals, Inc. J Res Sci Teach 44: 1057–1087, 2007     

USING ITEM RESPONSE THEORY TO CONDUCT A DISTRACTER ANALYSIS ON CONCEPTUAL INVENTORY OF NATURAL SELECTION

Concept inventories are often used to assess current student understanding although conceptual change models are problematic. Due to controversies with conceptual change models and the realities of student assessment, it is important that concept inventories are evaluated using a variety of theoretical models to improve quality. This study used a modified item response theory model to determine university nonmajor biology students’ levels of understanding of natural selection (n = 1,192). Using Conceptual Inventory of Natural Selection, we have reported how we applied Bock’s modified nominal item response theory model and the distracter test item analysis. We found that the use of this model can define student levels of understanding and identify problematic distracters.