Conceptualizing the Classroom of Target Students: A Qualitative Investigation of Panelists' Experiences during Standard Setting

Increasingly, research has focused on the cognitive processes associated with various standard-setting activities. This qualitative study involved an examination of 16 third-grade reading teachers' experiences with the cognitive task of conceptualizing an entire classroom of hypothetical target students when the single-passage bookmark method or the yes/no method was used during a one-day mock panel meeting. Data were collected using in-depth focus group interviews with eight participants from each of the panel meetings, and a whole-text analysis revealed three categories. Most participants experienced difficulty in attempting to conceive of an entire classroom of target students. Most of them were ultimately unable to do so and made use of alternative cognitive strategies. More fundamental issues also contributed to the difficulties experienced in attempting to complete the required cognitive task. Implications of the findings for standard setting and for further research are also discussed.     

Making the Cut in School Districts: Alternative Methods for Setting Cutscores

School districts are under increasing pressure to demonstrate that students are competent in various skills, such as reading and mathematics. Often, demonstrating competence involves comparing performance on assessments to a standard of performance, as embodied in a test score. These scores, called cutscores, separate competent and noncompetent examinees. Because school districts have varied sources of data to inform cutscore decisions, various methods are available for suggesting cutscores. In 2 studies, we examine a selection of methods for arriving at rational and defensible cutscores in school districts. Methods examined are the Angoff (1971) method; the borderline and contrasting groups methods; and 2 new methods, 1 based on course enrollment and 1 based on expert expectations. In Study 1, the Angoff, borderline group, and course enrollment results were consistent, whereas in Study 2, the Angoff and professional judgment methods yielded suggested cutscores that were lower than the borderline group method. Suggestions for further study include the reaction of teachers to the cutscore-setting methods, the effect of different teacher attributes on the results of cutscore-setting methods, and the efficiency of and most effective order for employing the various methods.     

The Choice of Response Probability in Bookmark Standard Setting: An Experimental Study

Evidence of the internal consistency of standard-setting judgments is a critical part of the validity argument for tests used to make classification decisions. The bookmark standard-setting procedure is a popular approach to establishing performance standards, but there is relatively little research that reflects on the internal consistency of the resulting judgments. This article presents the results of an experiment in which content experts were randomly assigned to one of two response probability conditions: .67 and .80. If the standard-setting judgments collected with the bookmark procedure are internally consistent, both conditions should produce highly similar cut scores. The results showed substantially different cut scores for the two conditions; this calls into question whether content experts can produce the type of internally consistent judgments that are required using the bookmark procedure.     

An Experimental Study of the Internal Consistency of Judgments Made in Bookmark Standard Setting

Validating performance standards is challenging and complex. Because of the difficulties associated with collecting evidence related to external criteria, validity arguments rely heavily on evidence related to internal criteria—especially evidence that expert judgments are internally consistent. Given its importance, it is somewhat surprising that evidence of this kind has rarely been published in the context of the widely used bookmark standard‐setting procedure. In this article we examined the effect of ordered item booklet difficulty on content experts’ bookmark judgments. If panelists make internally consistent judgments, their resultant cut scores should be unaffected by the difficulty of their respective booklets. This internal consistency was not observed: the results suggest that substantial systematic differences in the resultant cut scores can arise when the difficulty of the ordered item booklets varies. These findings raise questions about the ability of content experts to make the judgments required by the bookmark procedure.     

A Standard-Setting Method Designed for Complex Performance Assessments: Categorical Assignments of Student Work

Results of variations on a categorical standard-setting procedure for use with performance assessments with multiple performance standards are reported. Panelists were asked to make independent ordinal categorical assignments of student work. Student work came from the administration of 1 of the 1996 Grade 8 National Assessment of Educational Progress Science Booklets. This study focused on the comparability of cutscores using 2 data analysis strategies, the effects of 2 strategies for making the categorical classifications (sorting vs. direct classification), the efficacy of long and short versions of the classification scale, and the effects of discussion on the basic, proficient, and advanced cutscores. The results indicate that there were minimal differences in cutscores across the 2 data analysis strategies. The direct classification approach was more feasible than the sorting approach, at least with the design as it was implemented in this study. The short version of the classification scale yielded comparable results to the longer version and took less time (about 20%) for the panelists to complete. Discussion produced the expected effect of more agreement among panelists. These results should be interpreted with caution due to the small sample sizes.     

Setting Standards for English Foreign Language Assessment: Methodology,Validation, and a Degree of Arbitrariness

Cut‐scores were set by expert judges on assessments of reading and listening comprehension of English as a foreign language (EFL), using the bookmark standard‐setting method to differentiate proficiency levels defined by the Common European Framework of Reference (CEFR). Assessments contained stratified item samples drawn from extensive item pools, calibrated using Rasch models on the basis of examinee responses of a German nationwide assessment of secondary school language performance. The results suggest significant effects of item sampling strategies for the bookmark method on cut‐score recommendations, as well as significant cut‐score judgment revision over cut‐score placement rounds. Results are discussed within a framework of establishing validity evidence supporting cut‐score recommendations using the widely employed bookmark method.     

Adventure education and resilience: The double-edged sword

Abstract

Adventure education philosophers have argued that controlled exposure to challenge can enhance participants' psychological resilience. This study supports this claim, demonstrating significantly greater gains in psychological resilience for 41 young adults participating in 22-day Outward Bound programs than in a control group. All Outward Bound participants reported positive changes in their resilience and their overall change effect size was large. Perceived levels of social support predicted 24% of the variance in resilience gain scores, with participants' ratings of the least supportive group member being the best predictor. The findings for enhanced resilience and the important role of social support warrant wider investigation. In promoting psychological resilience adventure educators are alerted to the importance of facilitating positive interpersonal relations and minimising the detrimental impact of the diverse needs of group members.     

Commentary: A Response to Reckase's Conceptual Framework and Examples for Evaluating Standard Setting Methods

A look at real data shows that Reckase's psychometric theory for standard setting is not applicable to bookmark and that his simulations cannot explain actual differences between methods. It is suggested that exclusively test-centered, criterion-referenced approaches are too idealized and that a psychophysics paradigm and a theory of group behavior could be more useful in thinking about the standard setting process. In this view, item mapping methods such as bookmark are reasonable adaptations to fundamental limitations in human judgments of item difficulty. They make item ratings unnecessary and have unique potential for integrating external validity data and student performance data more fully into the standard setting process.     

Time,Third Space,and Counternarratives of Achievement for Young Mothers in High School

This article shares alternative conceptions of time in a qualitative study about the schooling experiences of teenage mothers. The counternarratives of three women of color are presented as alternatives for understanding temporally restricted discourses of achievement in high school. Using third space theory as a lens for understanding the participants' experiences, this article illustrates how each participant's school story creates room for thinking about school success differently. This understanding of participants' experiences provokes a deliberate reconsideration of time as it relates to student success and achievement in school. This article concludes with an analysis of the participants' counternarratives and implications for future work. This work is useful for practitioners and researchers in reconceptualizing difference among students and in creating room for school structures that support diversity in classrooms and schools.     

北京市中小学青年教师专业素养现状调查与分析——以“京教杯”参赛青年教师为研究对象

以网上问卷调查的方式,对849名参加"京教杯"比赛的青年教师进行专业素养现状调查。结果发现:"京教杯"参赛青年教师对学科知识的理解尚不够深入,反思能力低于其他能力,教育观念还需深化,情感素养亟待提升。制约参赛教师专业发展的主要因素是"工作繁忙、缺乏自主时间""缺乏专业指导""缺乏教育资源"等。基于此,我们需要整体、全面地认识中小学青年教师的专业素养,提升其情感素养,营造良好的成长氛围,提供专业支持与指导,激发其自我实现的内驱力,促进其自主成长。     

“We are sorry to inform you…”—The effects of early elimination on science competition participants’ career aspirations

In the past, students' participation in science competitions has been positively associated with their aspirations to pursue a career in science. Previous studies, however, were predominantly focused around successful competitors, overlooking the largest group of participants: those who are faced with early elimination. We therefore aimed to investigate the effects of elimination on the development of biology-related study and career task values and expectancy of success in first-round participants of the German Biology Olympiad (N = 381, mean age 16.5 years, 72% female). This study was the first of its kind to use a latent change score model approach to examine the effects of early elimination, with a particular focus on participants who placed great emphasis on succeeding in the competition. We found that, regardless of success or failure, participants' biology-related study and career task value remained stable from the first to the second round of the competition, while their expectancy of success in biology-related studies and career developed positively. Yet, for those participants who placed great importance on advancing in the competition, early elimination interfered with the development of study and career expectations, resulting in a weaker development. The outcomes of this study suggest that (1) science competitions should re-envision themselves to more directly address participants' values about studies and careers, especially in earlier competition rounds, and (2) science competitions should find innovative ways to provide detailed feedback to students and teachers to improve post-elimination performance. Our findings complement existing expectancy-value research and can serve as a starting point for future studies exploring mechanisms behind early elimination in different science domains and cultural contexts, providing empirical insight into creating an inclusive and supportive environment for all science competition competitors.     

Using communities of practice as a tool to analyse developing identity in online discussion

In this article, we address the methodological implications of analysing online discussion boards with a focus on participants' changing identities. More specifically, we propose the use of a Communities of Practice framework as a heuristic method for considering how participants' contributions to online discussion play a role in changing who they become, as opposed to what they learn. We argue that many analyses of online communication fail to take account of learning as a change in identity – what/who the learner is (becoming) – and focus mainly on the opportunity for cognitive development – what they know. In doing so, we use a single case study from a discussion board for secondary-aged students of mathematics which acts as a ‘telling case’ before demonstrating what this might say about other contexts for online learning. We argue that this has important implications for both research into, and the design of, online education.     

Setting Reliable National Curriculum Standards: a guide to the Angoff procedure

The Northern Ireland Curriculum, like the English National Curriculum, records pupil achievement on a 10‐level scale. The level to which a pupil is ‘assigned’ at the end of a Key Stage is based upon two sources of assessment information: classroom‐based measures provided by the teacher and summative information from Common Assessment Instruments (CAIs), which are pen‐and‐paper tests taken at the end of the Key Stage. CAIs play a central role in confirming the accuracy with which teachers judge the level at which a pupil is working. While the teacher might judge a pupil to have mastered level 7 in Algebra, for example, based upon observation in class, test data and homeworks, the CAI will only confirm this level if the pupil scores above the level 7 cutscore on the CAI. If this cutscore does not accord with a reliable measure of what constitutes level 7 performance in Algebra in the classroom, there is likely to be misclassification of pupils with attendant difficulties for the efficient planning of teaching and learning. Misclassifications can be minimised when examiners and teachers interpret level 7 achievement in Algebra similarly. The Angoff standard‐setting procedure was used to establish level 5 cutscores in the Number and Handling Data tests of the mathematics CAI so that comparisons might be made between the published level 5 cutscores and those which result from a judgemental standard‐setting procedure. The 21 teachers involved in the procedure were offered the opportunity to recommend a level 5 ‘standard’ using the Angoff methodology, and to review their recommendations in the light of test data from the February 1993 CAI administration. A further opportunity was offered following a discussion during which individual teachers articulated their reasons for the standards they recommended. The results confirm that the reliability of recommended standards increases both as a consequence of receiving normative data and of discussion. All statistical measures reported in this article indicate that the procedure could command the confidence of examiners, teachers and the public. While the recommended cutscore for Number is in close accord with that published by the examiners, the extent of the mismatch in the Handling Data test is such as might give rise to some misclassification of pupils. It is important to stress that this mismatch had no real consequences since 1993 was a pilot year and no test outcomes were reported. The article concludes with an outline of the contribution which the Angoff methodology can make to the resolution of some of the difficulties faced by English national assessment, as identified in Sir Ron Dealing's interim report “The National Curriculum and its Assessment”.     

The use of narrative to promote primary school children's understanding of evolution

This study explores learning about evolution when information is presented in either a narrative or an expository text (ET). Narratives engage the imagination, and consequently may allow children to overcome conceptual constraints that make evolution difficult to comprehend. Participants were 16 Year One, 21 Year Two and 25 Year Three pupils from a British primary school. Expository and narrative texts about evolution were read with children before assessing participants' understanding. Those reading the narrative showed a greater understanding of the book's contents about evolution than those reading the ET. Implications for using the imagination and narrative in the science classroom are discussed.     

The interplay of positivity and self-verification strivings: Feedback preference under increased desire for self-enhancement

Feedback preference is motivated. In the educational context, two main drivers of feedback preference are the desire for positive evaluation (self-enhancement motive) and the desire for subjectively accurate information (self-verification motive). Whereas the former produces positivity bias, the latter is seen in the preference for feedback consistent with one's established self-perception. Therefore, before the study, we measured participants' self-perception in the social domain. In the laboratory, we presented them with two evaluations of their social abilities: one positive, one negative. We tested participants' preferences for these feedback options under increased self-enhancement motivation. Study 1 showed that participants high (vs. low) on the desire for positive evaluation exhibited an enhanced preference for positive feedback. This was mainly true for participants with positive self-perception. Study 2 showed that under situationally increased self-enhancement motivation, people prefer positive feedback, regardless of its perceived accuracy. The results shed light on how motivation affects feedback processing.     

Investigating Students' Perceived Discipline Relevance Subsequent to Playing Educational Computer Games: A Personal Interest and Self-Determination Theory Approach

The purpose of this study is to explain students' perceived relevance of playing an educational game as a means for development of discipline competence. Based on self-determination theory and the concept of personal interest, we propose that: Satisfying students' basic needs for competence, autonomy, and relatedness when playing educational games is critical for arousal of intrinsic motivation in learning-through-gaming; Students' interest in the core subject and their intrinsic motivation in learning-through-gaming are important for their perceptions of how relevant they believe the gaming session is for the discipline they are studying. The results from an empirical study indicate that arousal of intrinsic motivation in the ongoing gaming session is more important than pre-gaming interest in the core subject for participants' post-gaming perception of how relevant such gaming is as a means for developing their discipline competence. The results also indicate that satisfying basic needs for competence, autonomy, and relatedness is critical for arousal of intrinsic gaming motivation. Our findings have implications for the choice of pedagogical means when utilizing educational games as a complementary alternative to traditional learning methods. To that end, we offer directions for future research.     

African-American women's decisions to become teachers: Sociocultural perspectives

In this study the authors describe sociocultural and school-related factors that influenced African-American women's perceptions of teaching and decisions to teach. Surveys were conducted of the participants' demographic and background characteristics, and life histories were developed that focused on early experiences with the family, in the community, and at school and how these experiences influenced their perceptions of teaching. These data revealed that the mother, women in the community, and female teachers were significant influences on the participants' decisions to teach. Moreover, their experiences and interpretations of these experiences have shaped their images of good teaching as shared, culturally responsive mothering. The participants' life histories further revealed that as their conceptions of good mothering and good teaching crystallized, their reasons for teaching transcended altruism and a love for children, resulting in perspectives on caring, and a sense of responsibility for the school success of all students. The authors suggest that improved understanding about African-American prospective teachers' lived experiences that give rise to their perceptions of and purposes for teaching hold implications for teacher education and African-American teacher recruitment and retention efforts.