Factors Affecting the Implementation of Argument in the Elementary Science Classroom. A Longitudinal Case Study

This longitudinal case study describes the factors that affect an experienced teacher’s attempt to shift her pedagogical practices in order to implement embedded elements of argument into her science classroom. Research data was accumulated over 2 years through video recordings of science classes. The Reformed Teacher Observation Protocol (RTOP) is an instrument designed to quantify changes in classroom environments as related to reform as defined by the National Research Council (National science education standards. Washington, DC: National Academy Press, 1996b) and the National Research Council (Fulfilling the promise: Biology education in the nation’s schools, Washington, DC: National Academy Press, 1990) and was used to analyze videotaped science lessons. Analysis of the data shows that there was a significant shift in the areas of teacher questioning, and student voice. Several levels of subsequent analysis were completed related to teacher questioning and student voice. The data suggests a relationship between these areas and the implementation of scientific argument. Results indicate that the teacher moved from a traditional, teacher-centered, didactic teaching style to instructional practices that allowed the focus and direction of the lesson to be affected by student voice. This was accomplished by a change in teacher questioning that included a shift from factual recall to more divergent questioning patterns allowing for increased student voice. As student voice increased, students began to investigate ideas, make statements or claims and to support these claims with strong evidence. Finally, students were observed refuting claims in the form of rebuttals. This study informs professional development related to experienced teachers in that it highlights pedagogical issues involved in implementing embedded elements of argument in the elementary classroom.     

Assessment matters: Early intervention: Using assessment to reduce student attrition

Brian A. Vander Schee describes how incorporating the College Student Inventory into an existing freshman seminar measurably improved student retention.     

Effect of Content Knowledge on Angoff‐Style Standard Setting Judgments

Evidence to support the credibility of standard setting procedures is a critical part of the validity argument for decisions made based on tests that are used for classification. One area in which there has been limited empirical study is the impact of standard setting judge selection on the resulting cut score. One important issue related to judge selection is whether the extent of judges’ content knowledge impacts their perceptions of the probability that a minimally proficient examinee will answer the item correctly. The present article reports on two studies conducted in the context of Angoff‐style standard setting for medical licensing examinations. In the first study, content experts answered and subsequently provided Angoff judgments for a set of test items. After accounting for perceived item difficulty and judge stringency, answering the item correctly accounted for a significant (and potentially important) impact on expert judgment. The second study examined whether providing the correct answer to the judges would result in a similar effect to that associated with knowing the correct answer. The results suggested that providing the correct answer did not impact judgments. These results have important implications for the validity of standard setting outcomes in general and on judge recruitment specifically.     

An early career academic network: what worked and what didn’t

This article documents the experiences of three early career academics trying to establish a network of early career academics (ECAs) in a middle-ranked university in Australia. The changing context of academia means that ECAs face considerable challenges in understanding and negotiating effective career paths. Some of the issues encountered include insecure employment arrangements; unclear and shifting expectations; heavy workloads and competing demands; and conflicting experiences around the collegiate culture of academia. As research and teaching institutions, universities must ensure the ongoing development of new academics. While there is a growing interest in exploring the issues confronted by new academics, much remains to be done to better understand, and improve, the pathways of academic development. To this end we reflect on our efforts to establish an ECA network that aimed to enhance professional development, facilitate an improved research culture and establish an informal peer support network. We did so through establishing an online presence for sharing information, hosting a series of professional development seminars and hosting a 2.5 day writing retreat. Our experiences suggest that, while efforts to enhance the capacity of ECAs are worthwhile, the very same pressures that our network was attempting to address were simultaneously creating barriers to ECA involvement in the network and its activities.     

Digital Module 13: Monte Carlo Simulation Studies in Item Response Theory

In this digital ITEMS module, Dr. Brian Leventhal and Dr. Allison Ames provide an overview of Monte Carlo simulation studies (MCSS) in item response theory (IRT). MCSS are utilized for a variety of reasons, one of the most compelling being that they can be used when analytic solutions are impractical or nonexistent because they allow researchers to specify and manipulate an array of parameter values and experimental conditions (e.g., sample size, test length, and test characteristics). Dr. Leventhal and Dr. Ames review the conceptual foundation of MCSS in IRT and walk through the processes of simulating total scores as well as item responses using the two-parameter logistic, graded response, and bifactor models. They provide guidance for how to implement MCSS using other item response models and best practices for efficient syntax and executing an MCSS. The digital module contains sample SAS code, diagnostic quiz questions, activities, curated resources, and a glossary.     

The Impact of Examinee Performance Information on Judges’ Cut Scores in Modified Angoff Standard‐Setting Exercises

This research evaluated the impact of a common modification to Angoff standard‐setting exercises: the provision of examinee performance data. Data from 18 independent standard‐setting panels across three different medical licensing examinations were examined to investigate whether and how the provision of performance information impacted judgments and the resulting cut scores. Results varied by panel but in general indicated that both the variability among the panelists and the resulting cut scores were affected by the data. After the review of performance data, panelist variability generally decreased. In addition, for all panels and examinations pre‐ and post‐data cut scores were significantly different. Investigation of the practical significance of the findings indicated that nontrivial fail rate changes were associated with the cut score changes for a majority of standard‐setting exercises. This study is the first to provide a large‐scale, systematic evaluation of the impact of a common standard setting practice, and the results can provide practitioners with insight into how the practice influences panelist variability and resulting cut scores.