Predictive validity evidence for an assessment program based on the Work Sampling System in mathematics and language and literacy

This study examined the predictive nature of teacher ratings of student performance on the mathematical thinking and language and literacy domains of a state mandated curriculum-embedded performance assessment for children in first grade to student achievement on a criterion-referenced test in third grade in mathematics and English language arts, using a multilevel modeling approach. The sample included 1,281 elementary school children nested within 132 classrooms in an urban school district in South Carolina. Results showed positive associations between first grade teacher ratings and students’ third grade achievement. However, first grade teacher ratings were not found to uniquely contribute to the proportion of explained variance in third grade performance on a high-stakes statewide assessment, over and above student demographic variables.     

Linking rights,needs, and fairness in high-stakes assessments and examinations

This paper presents a novel explanation for the continued absence of a children’s rights strategy within high-stakes educational assessment with reference to the competing purposes of high-stakes assessments and group-based constructions of fairness in assessment. We provide an original critique of group-based perspectives on the validity of assessment accommodations which supports an individual perspective on fair educational assessment. From this, the (almost forgotten) concept of ‘student assessment needs’ is (re-)introduced as a central axiom, to be constructed through feedback from, and dialogue with, students about their experience of high-stakes assessments, giving primacy to their purpose as an attainment demonstration opportunity for the student. To promote a new movement towards student participation in educational assessment processes and reforms, we propose ‘rights respecting assessment’, complementary to UNICEF’s ‘Rights Respecting Schools’ initiative, in which regular system-wide student representation would contribute to the development of high-stakes examination systems. In recognition of this aspiration, this paper is co-authored with a recent school-leaving age examination candidate.     

Assessment Validation in the Context of High-Stakes Assessment

Including the perspectives of stakeholder groups (e.g., teachers, parents) can improve the validity of high-stakes assessment interpretations and uses. How stakeholder groups view high-stakes assessments and their uses may differ significantly from state-level policy officials. The views of these stakeholders can contribute to identifying the strengths and weaknesses of the intended assessment interpretations and uses. This article proposes a process approach to validity that addresses assessment validation in the context of high-stakes assessment. The process approach includes a test evaluator or validator who considers the perspectives of five stakeholder groups at four different stages of assessment maturity in relationship to six aspects of construct validity. The tasks of the test evaluator and how stakeholders' views might be incorporated are illustrated at each stage of assessment maturity. How the test evaluator might make judgments about the merit of high-stakes assessment interpretations and uses is discussed.     

Cultivating the myopic learner: the shared project of high-stakes and low-stakes assessment

This paper argues that, despite the obvious and important differences between high-stakes and low-stakes assessment, there remain important points in common. These manifest themselves at a sociological level, where each tradition of assessment shares a similar disposition towards power. It is argued that both high-stakes and low-stakes assessment, as they are practised in England today, act together in support of a wider regime of power. This regime relies upon the construction of specialised subjectivities, defined by a myopic and self-perpetuating concern with individual progression.     

Unintended Consequences of Test-Based Accountability When Testing in "Milepost" Grades

Educational accountability systems that use test scores as the primary measure of performance are in use in many states. Research has shown that such high-stakes testing can have negative consequences, including narrowing of the curriculum and overemphasizing decontextualized skills. This study examined the impact of a particular form of high-stakes testing, dubbed milepost testing, in which tests are only given at selected grade levels. Data were gathered in Kentucky at a time when state-mandated tests were given in some subjects in grades 4, 7, and 11, and in other subjects in grades 5, 8, and 11. This approach was used to reduce the cost and the burden of assessment and accountability. The study found that many of the effects identified in previous research were present when tests were only given in selected grade levels. This study documents substantial differences in practice between tested and nontested grades and draws implications for national testing proposals that are currently being debated.     

GI Forum v. Texas Education Agency: Observations for States

Seven conclusions for professionals who administer state assessment programs are drawn from the GI Forum v. Texas Education Agency ruling: (a) the American Educational Research Association, American Psychological Association, and National Council on Measurement in Education (1999) standards are appropriate to use; (b) items showing different p values for subgroups may be used if they are selected as adequate for sound educational reasons; (c) a cut score setting process should be educationally justified; (d) a high-stakes testing program can appropriately address unfair access to education; (e) multiple opportunities to pass satisfies the standard that a single test score should not be the sole basis for a high-stakes decision; (f) a conjunctive decision-making model can appropriately motivate both students and schools; and (g) an 80% pass rate criterion applied to eventual, as opposed to initial, success rates for subgroups is a reasonable threshold for adverse impact. Caution is recommended because circumstances in other states may not parallel those in Texas in important ways.     

Keyboarding compared with handwriting on a high-stakes writing assessment: Student choice of composing medium, raters’ perceptions, and text quality

This article examines the influence of keyboarding versus handwriting in a high-stakes writing assessment. Conclusions are based on data collected from a pilot project to move Old Dominion University's Exit Exam of Writing Proficiency from a handwritten format into a dual-option format (i.e., the students may choose to handwrite or keyboard the exam). This test case is used to demonstrate the complex implications for a writing assessment tool when universities aim to accommodate students’ various methods of physically creating a text. Discussion includes speculation about the format of the writing component on the SAT. It is argued that by analyzing the differences in high-stakes exams when students keyboard and when they handwrite, researchers can better understand how the material conditions for composing influence students and raters. It is concluded that if handwriting and keyboarding represent even slightly different composing skills and cognitive processes, then high-stakes writing assessments should aim to accommodate students by providing methods of textual production with which students are most comfortable, even as this accommodation produces new challenges for the creators and managers of large-scale writing assessments.     

‘Because then you could never ever get a job!’: children’s constructions of NAPLAN as high-stakes

In the midst of the debate surrounding the question of whether Australia’s National Assessment Program: Literacy and Numeracy (NAPLAN) test is high-stakes, it is evident that children’s own accounts of their experiences remain sparse. This paper describes the findings of a case study which documented the experiences of 105 children across two Catholic primary schools in Queensland serving different socio-economic status (SES) communities. Analysis of the data revealed that these teachers and principals did not experience NAPLAN as high-stakes. However, the data suggested that the children experienced the tests within a confusing context of contradictions and dissonances emanating from multiple sources; receiving little, if any, clear and consistent information regarding the purpose and significance of NAPLAN. While the children’s responses were varied, many reported NAPLAN as a negative experience, with some constructing the test as high-stakes. These constructions ranged from personal judgement or sense of letting their families down, to failure, and less commonly, grade retention and school exclusion. Some Year 3 children had also constructed good results as vital to future prosperity. These constructions bring into question the assumption that because NAPLAN is designed to be a low-stakes test, that children will necessarily experience it in this way.     

Validity of High‐Stakes Assessment: Are Students Engaged in Complex Thinking?

The validity of high-stakes assessments and accountability systems is discussed in relation to the requirements of No Child Left Behind (NCLB). The extent to which content standards and assessments are cognitively rich, the challenges in setting performance standards, and the impact of high-stakes assessments on instruction and student learning are addressed. The article argues for quality content standards, cognitively rich assessments, and a cohesive, balanced assessment system.     

Beyond translation: adapting a performance-task-based assessment of critical thinking ability for use in Rwanda

Critical thinking is frequently proposed as one of the most important learning outcomes of a university education. However, to date, it has been difficult to ascertain whether university students in low-income contexts are improving in their critical thinking skills, because the limited studies in this domain have relied on instruments developed in Western contexts, despite the clear dangers of such an approach. Cultural bias in assessment can best be overcome by explicitly developing tests for use in specific contexts. However, resource constraints often prevent this possibility. An alternative strategy is to adapt an existing instrument for use in a particular context. Although adaptation is the norm for high-stakes cross-cultural assessments, it is often not attempted for single country research studies. This may be due to an assumption that adaptation is excessively technical or will add significantly to a study timeline. In this article, which relies on data from a recent study in Rwanda, we present a methodology for adapting a performance-task-based assessment of critical thinking. Our experience with this methodology suggests that small teams can adapt instruments in a relatively short time frame, and that the benefits of doing so far outweigh any cost.     

Searching for the Norm in a System of Absolutes: A Case Study of Standards-Based Accountability Reform in Pre-Kindergarten

Research Findings: The emergence of standards-based accountability reforms in early childhood education has created new challenges for the field. This article presents findings from a case study that explored how stakeholders in a large urban pre-kindergarten program struggled to implement an assessment tool that aligned the normative academic achievement expectations found among their teachers and administrators with the absolute measures of this construct found in their state policymakers' high-stakes standards-based accountability reforms. Analyzing the tension that emerged in this process of alignment highlights the challenges early educators face as they fold their child-centered programs into these larger high-stakes standards-based kindergarten through grade 12 education systems. Practice and Policy: The findings from this study illuminate the need for early childhood education programs to understand how high-stakes standards-based accountability reforms define student achievement. Furthermore, as early childhood programs and personnel address these reforms, their responses need to be explicit about how their assessment measures are connected to their normative conceptualizations of student achievement and what this means for the education of children in their programs.     

A Quantum Measurement Paradigm for Educational Predicates: Implications for validity in educational measurement

Abstract

The outcomes of educational assessments undoubtedly have real implications for students, teachers, schools and education in the widest sense. Assessment results are, for example, used to award qualifications that determine future educational or vocational pathways of students. The results obtained by students in assessments are also used to gauge individual teacher quality, to hold schools to account for the standards achieved by their students, and to compare international education systems. Given the current high-stakes nature of educational assessment, it is imperative that the measurement practices involved have stable philosophical foundations. However, this article casts doubt on the theoretical underpinnings of contemporary educational measurement models. Aspects of Wittgenstein’s later philosophy and Bohr’s philosophy of quantum theory are used to argue that a quantum theoretical rather than a Newtonian model is appropriate for educational measurement, and the associated implications for the concept of validity are elucidated. Whilst it is acknowledged that the transition to a quantum theoretical framework would not lead to the demise of educational assessment, it is argued that, where practical, current high-stakes assessments should be reformed to become as ‘low-stakes’ as possible. This article also undermines some of the pro high-stakes testing rhetoric that has a tendency to afflict education.     

Teachers’ involvement in high-stakes language assessment reforms: The case of Test for English Majors (TEM) in China

As important stakeholders in language assessment, teachers’ perceptions of language test design and implementation are key factors influencing test impact. This study investigated how English teachers perceived the recent reform of Test for English Majors Grade 4 (TEM4) - a high-stakes English proficiency test targeting English-major undergraduates in China. Furthermore, this study explored the convergence and divergence between teachers’ perceptions of the reform and the intentions of the TEM4 Committee. Findings of this study indicate that teachers in general commented favourably on the revised TEM4 and felt that most of the reform measures were appropriate; however, some divergence between teachers’ perceptions and the intentions of the TEM4 developer was uncovered, particularly in relation to the listening section. This study underscores the importance of soliciting teachers’ perspectives on high-stakes assessment reforms and signals the need for building more effective communicative mechanisms between language assessment providers and stakeholders.     

The Effect of Stakes on Accountability Test Scores and Pass Rates

Students may not fully demonstrate their knowledge and skills on accountability tests if there are no stakes attached to individual performance. In that case, assessment results may not accurately reflect student achievement, so the validity of score interpretations and uses suffers. For this study, matched samples of students taking state accountability tests under low-stakes and high-stakes conditions were used to estimate the effect of stakes on test performance and subsequent pass rates. Across five assessments, expected performance was greater under high-stakes conditions, with effect sizes ranging from 0.41 to 0.50 standard deviations and with students of lower ability tending to be slightly more affected by stakes. Depending on where cut scores were set, pass rates differed by up to 30% when comparing the low- and high-stakes conditions.     

Tracing the Discourses of Accountability and Equity: The Case of the Grade 4 Literacy Test in Jamaica

In an attempt to understand how a narrowed version of accountability in the form of high-stakes assessment deepens inequity rather than improves educational equity, we examine three education documents in Jamaica using critical discourse analysis. Our two research questions were: How did each government document position the Grade Four Literacy Test? What did that positioning signify in terms of accountability and equity? Our analyses reveal that the discourse of equity becomes silenced or overpowered in the presence of powerful discourses of accountability. It is to this extent, we contend, that a narrower version of accountability in the form of high-stakes testing serves to compound and complicate inequities in education. We offer the view that a more comprehensive framework for accountability in which teachers are supported and literacy projects are comprehensive and sustained moves closer to achieving, over the long run, the equity that is needed in literacy at the primary level in Jamaica.     

Multiple Sources of Evidence: An Analysis of Stakeholders' Perceptions of Various Indicators of Student Learning

This study compared different stakeholders' perceived validity of various indicators of student learning used to judge the quality of students' academic performance. Data were gathered from the questionnaire responses of 314 educators in three states that have implemented comprehensive state-wide assessment programs with high-stakes consequences both for educators and for students. MANOVA results showed that while educators generally hold similar perceptions, significant differences exist between school administrators and teachers. Administrators perceived the results from nationally normed standardized assessments, state assessments, and district assessments to be more valid indicators of student achievement than did teachers. In contrast, teachers granted more validity to classroom observations and homework completion and quality than did administrators. The implications of these differences for reform initiatives are discussed, particularly with regard to teachers' motivation to improve results.     

Teacher change and “high-stakes” assessment: what happens to professional development?

This study explored the impact of learning a new instructional practice in a US school district that places great value in a statewide student assessment. Grounded Theory methodology (Strauss & Corbin, 1998, Basics of Qualitative Research: Techniques and procedures for developing grounded theory, 2nd Edition, Sage Publications, Thousand Oaks, CA) was used to analyze interview and observation data from 20 teachers undergoing professional development in a new reading strategy. We found that teaching in a “high-stakes” assessment environment impacts the implementation, fidelity, and sustainability of new teaching methods. Most importantly, our results suggest that some teachers may use “high-stakes” assessments as their primary reference point in which to gauge the merit of innovative teaching practices. Furthermore, decisions regarding which components to implement, how often to use these components, and what materials to utilize in their implementation may be influenced by the preparation for and expected results of these tests.     

Genres of high-stakes writing assessments and the construct of writing competence

High-stakes writing assessments currently exert a strong influence on the writing curriculum and instruction in schools across the United States. Under these circumstances it is important to examine the construct of writing competence on which these assessments are based, as well as the extent to which this construct supports the goals of secondary education. In this paper we conduct an exploratory analysis of the genre demands of high-stakes writing assessments from three states – California, Texas, and New York – with the aim of discerning, comparing, and evaluating the role that genre knowledge plays in the construct of writing competence measured by these assessments. Our method of inquiry includes both task analysis of the prompts and genre analysis of high-scoring benchmark papers written in response to these prompts. For the analysis of benchmark papers we employed both structural analysis and quantitative counts of key linguistic features to characterize the genres represented in these assessment tasks. Our results suggest a lack of alignment between the genres of the benchmark papers designated as exemplary and the genre demands of the prompts to which they were written. Exceptions to this pattern were most common on the New York assessments, which contextualize writing tasks in tests of subject–matter knowledge. Findings from our exploratory analysis lead us to argue for greater consistency and clarity of expectations in the design of high-stakes writing exams, and for the design of writing tasks that adequately represent the demands of discipline-specific forms of written discourse.