An NCME Instructional Module on: Setting Passing Scores

This module describes standard setting for achievement measures used i n education, licensure, and certification. On completing the module, readers will be able to: describe what standard setting is, why it is necessary, what some of the purposes of standard setting are, and what professional guidelines apply to the design and conduct of a standard-setting procedure; differentiate among different models of standard setting; calculate a cutting score using various methods; identify appropriate sources of validity evidence and threats to the validity of a standard-setting procedure; and list some elements to be considered when evaluating the success of a standard-setting procedure. A self-test and annotated bibliography are provided at the end of the module. Teaching aids to accompany the module are available through NCME.     

Setting Performance Standards: Contemporary Methods

This module describes some common standard-setting procedures used to derive performance levels for achievement tests in education, licensure, and certification. Upon completing the module, readers will be able to: describe what standard setting is; understand why standard setting is necessary; recognize some of the purposes of standard setting; calculate cut scores using various methods; and identify elements to be considered when evaluating standard-setting procedures. A self-test and annotated bibliography are provided at the end of the module. Teaching aids to accompany the module are available through NCME.     

Issues in Setting Motor Performance Standards

Setting motor performance standards has long been a process of interest to physical educators. Theoretical advances in the measurement technology appropriate for standard-setting, however, have occurred only in the last decade. The first portion of this paper is devoted to a discussion of issues in setting standards and a brief review of procedures for standard-setting. In the latter section, gender differences in motor performance are examined and the impact of these differences on standard-setting is considered.     

Installing a System of Performance Standards for National Assessments in the Republic of Trinidad and Tobago: Issues and Challenges

This article explores the challenge of setting performance standards in a non-Western context. The study is centered on standard-setting practice in the national learning assessments of Trinidad and Tobago. Quantitative and qualitative data from annual evaluations between 2005 and 2009 were compiled, analyzed, and deconstructed. In the mixed methods research design, data were integrated under an evaluation framework for validating performance standards. The quantitative data included panelists’ judgments across standard-setting rounds and methods. The qualitative data included both retrospective comments from open-ended surveys and real-time data from reflective diaries. Findings for procedural and internal validity were mixed, but the evidence for external validity suggested that the final outcomes were reasonable and defensible. Nevertheless, the real-time qualitative data from the reflective diaries highlighted several cognitive challenges experienced by panelists that may have impinged on procedural and internal validity. Additional unique hindrances were lack of resources and wide variation in achievement scores. Ensuring a sustainable system of performance standards requires attention to these deficits.     

Adopting Cut Scores: Post-Standard-Setting Panel Considerations for Decision Makers

Standard-setting studies utilizing procedures such as the Bookmark or Angoff methods are just one component of the complete standard-setting process. Decision makers ultimately must determine what they believe to be the most appropriate standard or cut score to use, employing the input of the standard-setting panelists as one piece of information among multiple sources. However, guidance for weighing the various components is limited. The current article describes considerations about data that are used to make standard-setting decisions, as previously outlined by Geisinger (1991) . The ten points provided by Geisinger have been expanded as they relate to shifts in educational policy and practice in educational measurement. They have been amended with six new components as well. The new considerations addressed are smoothing across grades, raising standards in progression (over grades or over time), opportunity to learn or instructional validity, input from other groups, equating or linking to previous standards, and organizational vision and goals .     

A study of the use of pairwise comparison in the context of social online moderation

Traditional moderation of student assessments is often carried out with groups of teachers working face-to-face in a specified location making judgements concerning the quality of representations of achievement. This traditional model has relied little on modern information communications technologies and has been logistically challenging. We argue that social online moderation, coupled with the use of analytical and pairwise scoring methods and technologies, can provide better moderation outcomes and highly valuable professional learning experiences improving teachers’ understandings of assessment standards. This paper reports on a component of a study involving Visual Arts teachers from rural schools making comparative judgements of digitised student artworks. We report the teachers’ observations of the social online moderation processes, including the quality and standard of the digitised artworks, the effectiveness of the pairwise comparison process, the functionality of the online tools, and the concept of using online scoring for moderation and standard setting purposes.     

Some Problems, Pitfalls, and Paradoxes in Educational Measurement

This article provides brief discussions of an eclectic set of problems, pitfalls, and paradoxes that challenge measurement theory and practice, especially for K–12 achievement testing. The technical issues considered are: true scores and error scores, fixed versus random facets, lower limits to reliability, the reliability-validity paradox, reliability of difference scores, equating and linking, and scaling. The practical issues considered are: validation, standard setting, instructional improvement and high-stakes accountability, multiple measures, adequate yearly progress, the National Assessment of Educational Progress as a confirmation model, and capacity.     

Response to Clemans

Clemans (1993) argued that the use of item response theory (IRT) to vertically scale Form E of the California Achievement Tests produces inappropriate results. In this response we show that (a) Cleman's analysis of school district data was incomplete, inconsistent, and did not follow good measurement practice; (b) the simulation he conducted was unfairly stacked against IRT, was unrealistic, and ignored other realistic published simulations that demonstrated the accuracy of IRT scaling procedures; (c) his "common sense" evaluations of student performance ignored basic facts about the measurement of student achievement; and (d) the concerns expressed in his article were irrelevant to the vast majority of uses of norm-referenced tests.     

Cross-cultural comparability of noncognitive constructs in TIMSS and PISA

Abstract

Noncognitive assessments in Programme for International Student Assessment (PISA) and Trends in International Mathematics and Science Study share certain similarities and provide complementary information, yet their comparability is seldom checked and convergence not sought. We made use of student self-report data of Instrumental Motivation, Enjoyment of Science and Sense of Belonging to School targeted in both surveys in 29 overlapping countries to (1) demonstrate levels of measurement comparability, (2) check convergence of different scaling methods within survey and (3) check convergence of these constructs with student achievement across surveys. We found that the three scales in either survey (except Sense of Belonging to School in PISA) reached at least metric invariance. The scale scores from the multigroup confirmatory factor analysis and the item response theory analysis were highly correlated, pointing to robustness of scaling methods. The correlations between each construct and achievement was generally positive within each culture in each survey, and the correlational pattern was similar across surveys (except for Sense of Belonging), indicating certain convergence in the cross-survey validation. We stress the importance of checking measurement invariance before making comparative inferences, and we discuss implications on the quality and relevance of these constructs in understating learning.     

The association between homework and primary school children's academic achievement. International evidence from PIRLS and TIMSS

The association between time devoted to homework and children's academic achievement has long been an issue of great debate. A small number of mainly correlational studies have been conducted into this issue in a primary school setting, but have produced somewhat mixed results. In this paper we contribute to this literature by investigating the relationship between time spent upon homework and children's outcomes across 24 countries. By using a student fixed-effects approach, capturing differences in homework time amongst the same student across different school subjects, we argue that our results are likely to be subject to less confounding than much of the existing literature. We find little evidence that the amount of homework time primary school children are assigned is related to their academic achievement. This holds true across a large number of countries, survives various robustness tests and does not vary by gender or socio-economic status. We interpret this finding as suggesting that the homework assigned to primary school pupils may not be adequate to produce a positive association, and needs be improved if this time-consuming activity is ever going to bring benefits for children's academic achievement.     

Commentary: A Response to Reckase's Conceptual Framework and Examples for Evaluating Standard Setting Methods

A look at real data shows that Reckase's psychometric theory for standard setting is not applicable to bookmark and that his simulations cannot explain actual differences between methods. It is suggested that exclusively test-centered, criterion-referenced approaches are too idealized and that a psychophysics paradigm and a theory of group behavior could be more useful in thinking about the standard setting process. In this view, item mapping methods such as bookmark are reasonable adaptations to fundamental limitations in human judgments of item difficulty. They make item ratings unnecessary and have unique potential for integrating external validity data and student performance data more fully into the standard setting process.     

Leadership as Accountability for Learning: The Effects of School Poverty, Teacher Experience, Previous Achievement, and Principal Preparation Programs on Student Achievement

In the current era of accountability for achievement, school principals play the pivotal role of instructional leader. In a high-stakes testing environment, leadership preparation programs in universities and school districts need to be positively related to academic outcomes. The purpose of this study was to examine the relationship between school leadership preparation programs and student achievement in urban settings. Because leadership is contingent on the setting, school contextual factors and their impact on student achievement framed this study. Regression techniques were employed to construct a conceptual model with predictors of criterion and norm-referenced student achievement scores. Confirming previous research findings, student poverty, teacher experience, and previous achievement were the strongest predictors and accounted for a significant amount of variance in student achievement; however, university and district preparation programs were not significant predictors. Implications for policy, practice, and future research are discussed.     

Some Considerations for Recreation in a Changing Society

Achievement goal theory, as an attempt to explain the factors that influence motivated behavior, has received signiticant attention from youth sport researchers, sport psychologists, and educators since its proposal in 1980. This papr reviews the basic tenets of achievement goal theory and outlines potential problems through its usage in attempting lo understand student behavior and attitudes in physical education. Specifically, I argue this reductionist and decontextualized research treats the physical education student as a motivational problem, the teacher as the solution, and the social context of physical education and sport as nonproblematic. This positivist approach to knowledge construction may indirectly act to perpetuate inequitable power relations and dominant idenlogies found in physical education. Ironically, this could dissuade certain youth from participating in sport or physical education, the antithesis of the activity promotion objectives of many achievement goal theorists. I conclude by suggesting to enhance physical education practice, research should employ critical and reflective methods of knowing and continue to seek out student and teacher voices to help create responsive leaming environments for diverse student needs.     

标准设定与等级划分

标准设定在教育测量领域是一个相当重要的议题,它涉及面十分广泛、备受人们争议,解决起来非常棘手。为了解决此难题,国外涌现出大量标准设定方面的理论研究和实践应用。我们在标准设定方面的研究还比较欠缺,文章归纳各种方法进行标准设定的步骤、介绍一些经典的标准设定方法以及剑桥评价在进行等级划分时的运用,以期对我们的标准设定实践有所助益,增加考试的信度。     

Solomon Islands school leaders readiness for inclusive education

In this paper, I critically examine the discourse surrounding response to intervention (RTI), a US-based education reform that has garnered a considerable amount of attention (as well as controversy) in a very short amount of time. A multi-pronged reform effort, RTI is a tiered approach to delivering instructional intervention to students at risk, an on-going and systematic model of monitoring student performance, as well as an alternative to the ability/achievement discrepancy model for identifying learning disabilities. In this paper, I argue, however, that RTI is not so much a reform but a tactic, aimed at returning to the status quo of segregated special education and reinvigorating many of the foundational assumptions of traditional special education practice.     

RESEARCH SECTION: Why do students think they fail? The relationship between attributions and academic self‐perceptions

Attributions can have a significant effect on academic achievement and students with learning difficulties are more likely to display negative attributions than their peers. In this article, the attributions of students identified as having learning difficulties are compared with those of other non-labelled low achievers, and non-labelled average achievers. Margaret Banks, MSc research methods student, and Dr Lisa Woolfson, Reader in Educational Psychology, both at the University of Strathclyde, involved 26 low-achieving students (15 identified as having learning difficulties and 11 without any such label) and 27 averagely achieving students in their exploratory study. The students attempted to solve two sets of puzzles that, unknown to the participants, were actually impossible. They then gave a reason for their failure and rated the stability and controllability of their failure. Participants were also asked to give a self-rating of academic achievement. It was found that teacher and student perceptions of academic achievement were not synonymous. In terms of attributions, Margaret Banks and Lisa Woolfson argue that teachers' perceptions of student achievement may not be as important as students' self concept. They suggest that low self-perception of achievement and negative attributions are both associates of low self-esteem.     

The Bookmark Standard-Setting Method: A Literature Review

The Bookmark method for setting standards on educational tests is currently one of the most popular standard-setting methods. However, research to support the method is scarce. In this report, we review the published and unpublished literature on this method as well as some seminal work in the area of evaluating standard-setting studies. Our review highlights both strengths and limitations of the method. Strengths include its wide acceptance and panelist confidence in the method. Limitations include a potential bias to produce lower-than-intended standards and problems in selecting the most appropriate response probability value for ordering the items presented to panelists. It is clear that more research on this method is needed to support its wide use. Several areas for future research to better understand the validity of the Bookmark method for setting standards on educational tests are presented.