A Validity Argument in Support of the Use of College Admissions Test Scores for Federal Accountability

In 2018, 26 states administered a college admissions test to all public school juniors. Nearly half of those states proposed to use those scores as their academic achievement indicators for federal accountability under the Every Student Succeeds Act (ESSA); many others are planning to use those scores for other accountability purposes. Accountability encompasses a number of different uses and subsumes a variety of claims. For states proposing to use summative tests for accountability, a validity argument needs to be developed, which entails delineating each specific use of test scores associated with accountability, identifying appropriate evidence, and offering a rebuttal to counterclaims. The aim of this article is to support states in developing a validity argument for use of college admission test scores for accountability by identifying claims that are applicable across states, along with summarizing existing evidence as it relates to each of these claims. As outlined by The Standards for Educational and Psychological Testing, multiple sources of evidence are used to address each claim. A series of threats to the validity argument, including weaker alignment with content standards and potential influences in narrowing teaching, are reviewed. Finally, the article contrasts validity evidence, primarily from research on the ACT, with regulatory requirements from ESSA. The Standards and guidance addressing the use of a “nationally recognized high school academic assessment” (Elementary and Secondary Education Act (ESEA), Negotiated Rulemaking Committee; Department of Education) are the primary sources for the organization of validity evidence.     

A General Framework for the Validation of Embedded Formative Assessment

In educational practice, test results are used for several purposes. However, validity research is especially focused on the validity of summative assessment. This article aimed to provide a general framework for validating formative assessment. The authors applied the argument‐based approach to validation to the context of formative assessment. This resulted in a proposed interpretation and use argument consisting of a score interpretation and a score use. The former involves inferences linking specific task performance to an interpretation of a student's general performance. The latter involves inferences regarding decisions about actions and educational consequences. The validity argument should focus on critical claims regarding score interpretation and score use, since both are critical to the effectiveness of formative assessment. The proposed framework is illustrated by an operational example including a presentation of evidence that can be collected on the basis of the framework.     

Strategies for Examining the Consequences of Assessment and Accountability Programs

This article addresses issues in evaluating the consequences of assessment programs that are developed for the purpose of holding schools accountable to state standards. After providing a brief review of research examining consequential evidence, a validation study to obtain consequential evidence for state assessment and accountability programs is proposed. The proposal includes a validity argument, a set of propositions that follow from the validity argument, a delineation of the consequential evidence needed, and a way to model the relationship between performance gains and school, principal, teacher, and student variables.     

Validity of Inferences from Test-based Educational Accountability Systems

Many states have adopted test-based accountability systems before the No Child Left Behind (NCLB) Act of 2001 became law. NCLB has added a new dimension to test-based educational accountability systems. Features of state and NCLB accountability are discussed with an emphasis on questions of the validity of inferences that are made about school quality. It is concluded that none of the current approaches toe test-based accountability support causal inferences about school quality. It is also shown that tracking progress toward the important goal of closing gaps in achievement requires more than just monitoring changes in the percentages of students who are proficient.     

Accountability Reform and Responsive Assessment for Immigrant Youth

Immigrant students, one of the fastest-growing populations in US public schools, have been linguistically and culturally disadvantaged by accountability policies that rely only on standardized tests. Recent changes to these policies allow for the use of performance-based assessment tasks (PBATs) as an assessment indicator to supplement standardized tests. In this article, we explore how 1 highly successful high school that works exclusively with recently arrived immigrant teenagers has incorporated PBATs into its curriculum. We find that school leaders, teachers, and students agree that the use of rigorous performance assessments accomplishes language learning, content mastery, and test preparation simultaneously.     

Student Achievement and School and Teacher Accountability

This article is part of a set of papers generated from a keynote presentation by Dr. Jack Frymier at the 1997 CREATE annual meeting. Dr. Frymier dealt with several reasons, that, as he saw it, invalidate the use of student achievement data in teacher accountability systems. This article first notes problems with Dr. Frymier's conception of accountability. Next, it summarizes some of the recent evidence showing the strong connection between school and teacher effectiveness measures and student achievement. It then notes some of the benefits of school and teacher effectiveness measures external to their function as measures of performance. Next, policy issues arising from the use of student data and the associated research are considered. Finally, it concludes with some cautions about using effectiveness measures in teacher accountability systems.     

Disentangling the predictive validity of high school grades for academic success in university

To refine selective admission models, we investigate which measure of prior achievement has the best predictive validity for academic success in university. We compare the predictive validity of three core high school subjects to the predictive validity of high school grade point average (GPA) for academic achievement in a liberal arts university programme. Predictive validity is compared between the Dutch pre-university (VWO) and the International Baccalaureate (IB) diploma. Moreover, we study how final GPA is predicted by prior achievement after students complete their first year. Path models were separately run for VWO (n = 314) and IB (n = 113) graduates. For VWO graduates, high school GPA explained more variance than core subject grades in first-year GPA and final GPA. For IB graduates, we found the opposite. Subsequent path models showed that after students’ completion of the first year, final GPA is best predicted by a combination of first-year GPA and high school GPA. Based on our small-scale results, we cautiously challenge the use of high school GPA as the norm for measuring prior achievement. Which measure of prior achievement best predicts academic success in university may depend on the diploma students enter with.     

Targeting education: The politics of performance and the prospects of ‘Education For All’

This article offers a critical examination of the public claims about the potential, reach and impact of “target setting in education” within the context of developing countries. The argument is made that the target-setting enterprise is undermined by three fallacies—conceptual, methodological and organizational—with negative consequences for the achievement of education quality in poor countries. While acknowledging the logic of targets on the part of international development agencies, the article nevertheless suggests that target setting could be seen as a transnational system of surveillance that takes measures of control, measurement and accountability beyond the confines of national borders.     

The importance of adjusting for pupil background in school value-added models: A study of Progress 8 and school accountability in England

In the UK, USA and elsewhere, school accountability systems increasingly compare schools using value-added measures of school performance derived from pupil scores in high-stakes standardised tests. Rather than naïvely comparing school average scores, which largely reflect school intake differences in prior attainment, these measures attempt to compare the average progress or improvement pupils make during a year or phase of schooling. Schools, however, also differ in terms of their pupil demographic and socioeconomic characteristics and these factors also predict why some schools subsequently score higher than others. Many therefore argue that value-added measures unadjusted for pupil background are biased in favour of schools with more ‘educationally advantaged’ intakes. But others worry that adjusting for pupil background entrenches socioeconomic inequities and excuses low-performing schools. In this article we explore these theoretical arguments and their practical importance in the context of the ‘Progress 8’ secondary school accountability system in England, which has chosen to ignore pupil background. We reveal how the reported low or high performance of many schools changes dramatically once adjustments are made for pupil background, and these changes also affect the reported differential performances of regions and of different school types. We conclude that accountability systems which choose to ignore pupil background are likely to reward and punish the wrong schools and this will likely have detrimental effects on pupil learning. These findings, especially when coupled with more general concerns surrounding high-stakes testing and school value-added models, raise serious doubts about their use in school accountability systems.     

Four Perspectives on Educational Assessment

Educational achievement testing is considered from four perspectives. First, the political concerns for accountability provide a powerful impetus for state‐wide testing of students. Second, parents want evidence about their children's accomplishments but often do not receive nor understand much information regarding testing. Third, in the USA, teachers often spend 2 to 8 weeks preparing their students for achievement tests and administering them. The “high‐stakes” nature of the tests undermines teachers’ curricula and instruction and, according to critics, may place too much emphasis on discrete skills, rote recall of facts, and esoteric test‐taking strategies. Students’ views of testing are the fourth perspective considered. A series of studies has revealed that the trust placed in testing by young children is gradually eroded until many adolescents are sceptical about the purpose and value of standardized achievement testing. Low achievers are especially prone to give token effort and to use counterproductive tactics. The cumulative, negative impact of standardized testing on many students and teachers threatens the validity of the test results and effective instruction. Seven suggestions are outlined to improve educational assessment for all participants.     

Have State Accountability and High-Stakes Tests Influenced Student Progression Rates in High School?

The question addressed in this paper is whether there are signs that students' progression through high school to graduation has been affected by the efforts to increase school accountability, including high-stakes exit exams. If such accountability systems were able to raise student academic performance significantly in elementary and middle school, these same students should be more likely to succeed academically in high school and graduate with their cohort. If state accountability systems also attempt to raise minimum standards for high school graduation by requiring exit tests, as many do, this could have the opposite effect of lowering high school graduation rates by putting yet one more barrier in the way of lower-performing students completing requirements.     

The Elusive Relationship Between Teacher Characteristics and Student Academic Growth: A Longitudinal Multilevel Model for Change

Teacher characteristics and student achievement growth are currently a significant topic of investigation in the educational accountability arena. Given the environment of high-stakes accountability associated with the No Child Left Behind (NCLB) legislation and state accountability systems, staffing all classrooms with highly qualified teachers is a critical national concern. A new era of research is needed to understand the complexity of teacher quality when defined by student learning growth. The present study evaluated the effects of teacher characteristics (i.e., experience, education, and race) in high school reading achievement gains using a multi-level growth model in an urban school district in Kentucky. Findings showed significant effects of time, but non-significant effects of teacher characteristics in high school reading achievement growth. Implications for educational policy and future research are discussed.     

Can High School Achievement Tests Serve to Select College Students?

Postsecondary schools have traditionally relied on admissions tests such as the SAT and ACT to select students. With high school achievement assessments in place in many states, it is important to ascertain whether scores from those exams can either supplement or supplant conventional admissions tests. In this study we examined whether the Arizona Instrument to Measure Standards (AIMS) high school tests could serve as a useful predictor of college performance. Stepwise regression analyses with a predetermined order of variable entry revealed that AIMS generally did not account for additional performance variation when added to high school grade-point average (HSGPA) and SAT. However, in a cohort of students that took the test for graduation purposes, AIMS did account for about the same proportion of variance as SAT when added to a model that included HSGPA. The predictive value of both SAT and AIMS was generally the same for Caucasian, Hispanic, and Asian American students. The ramifications of universities using high school achievement exams as predictors of college success, in addition to or in lieu of traditional measures, are discussed.     

Predicting College Performance of American Indians: A Large‐Sample Examination of the SAT

Extensive research has examined the validity and fairness of standardized tests in academic admissions. However, due to their underrepresentation in higher education, American Indians have gained much less attention in this research. In the present study, we examined for American Indian students (1) group differences on SAT scores, (2) the predictive and incremental validity of SAT over high school grades, (3) the effect of socioeconomic status on SAT validity, (4) differential prediction in the use of SAT scores, and (5) potential omitted variables that could explain differential prediction for American Indian students. Results provided evidence of predictive and incremental validity of SAT scores, and the validity of SAT scores was largely independent of socioeconomic status. Overprediction was found when using SAT scores to predict college performance and it was reduced when including high school grades as an additional predictor. This study provides substantial evidence of the validity and fairness of SAT scores for American Indians.     

Hurricane Katrina's Impact on Students and Staff Members in the Schools of Mississippi

Following Hurricane Katrina, a team of researchers from the University of Southern Mississippi's Department of Educational Leadership and Research surveyed school administrators regarding the impact of Hurricane Katrina upon students and school staff members in Mississippi public schools. Respondents disclosed concerns over displacement and psychological stressors experienced by both students and staff. Additional concerns were expressed regarding support systems for students, destruction of student and personnel records, student achievement and school accountability, and the employment status of teachers. Implications for policy and practice are discussed.     

Using data for decision-making: perspectives from 16 principals in Michigan, USA

In response to the vast amounts of data associated with the accountability movement and the rhetoric of data-informed decision-making, we interviewed 16 principals to find out what streams of data they used and what decisions they made by using the data. We found that: (a) student achievement data are predominantly used to the extent of neglecting other streams of data such as student and community background data and school process data; (b) student achievement data are used more for accountability purposes—for assessing “of” rather than “for” the learning; (c) different streams of data are rarely used together to derive rich meaning for decision-making; and (d) school districts differ in the extent to which their principals use data to improve curriculum and instruction. The study pointed both to the challenges and to the opportunities of making data-informed decisions to improve our schools.     

Validity considerations ensuing from examinees’ perceptions about high-stakes national examinations in Cyprus

Student examinees are key stakeholders in large-scale, high-stakes, public examination systems. How they perceive the purpose, comprehend the technical characteristics of testing and how they interpret scores influence their response to the system demands and their preparation for the examinations; this information relates to intended and unintended consequences of testing and is a component of an expanded notion of test validity. The research reported in this paper investigates examinees’ perceptions about the secondary school graduation and university-entrance national exams in Cyprus. Interviews with recent examinees reveal the versatility and complexity of their perceptions about the fairness and appropriateness of the system, which are influenced by design features of the exams and by the local context. There are important, mostly unintended, consequences on their in- and out-of-school experience, on school curricula and on instructional practices. Empirical evidence about consequential aspects of examinations contributes to the validity argument needed to support such programmes.     

Defining and Measuring College and Career Readiness: A Validation Framework

This article reviews the intended uses of these college‐ and career‐readiness assessments with the goal of articulating an appropriate validity argument to support such uses. These assessments differ fundamentally from today's state assessments employed for state accountability. Current assessments are used to determine if students have mastered the knowledge and skills articulated in state standards; content standards, performance levels, and student impact often differ across states. College‐ and career‐readiness assessments will be used to determine if students are prepared to succeed in postsecondary education. Do students have a high probability of academic success in college or career‐training programs? As with admissions, placement, and selection tests, the primary interpretations that will be made from test scores concern future performance. Statistical evidence between test scores and performance in postsecondary education will become an important form of evidence. A validation argument should first define the construct (college and career readiness) and then define appropriate criterion measures. This article reviews alternative definitions and measures of college and career readiness and contrasts traditional standard‐setting methods with empirically based approaches to support a validation argument.     

Understanding the Gender Gap in Science and Engineering: Evidence from the Chilean College Admissions Tests

This study seeks to develop a better understanding of the underrepresentation of women in science and engineering by analyzing the gender gaps (a) in the interest in pursuing a science degree and (b) on science achievement. We use national-level college admissions data to examine gender differences and to explore the association between these outcomes and the attendance to single-sex or co-educational schools. The Chilean college admissions system provides a unique context to study these gender differences, since applicants who wish to pursue an undergraduate degree in science or engineering are required to take a high-stakes standardized science achievement test as part of the admission battery. This test has three subjects: biology, physics, and chemistry, and applicants must choose to be tested in only one of them. Significant gender differences exist for the examinees in their choice of subject and achievement on (the tests). Gender gaps favoring males are observed in the three forms. Both interest and achievement in science are associated with the sex composition of the school attended.