Content and alignment of state writing standards and assessments as predictors of student writing achievement: an analysis of 2007 National Assessment of Educational Progress data

We examined the degree to which content of states’ writing standards and assessments (using measures of content range, frequency, balance, and cognitive complexity) and their alignment were related to student writing achievement on the 2007 National Assessment of Educational Progress (NAEP), while controlling for student, school, and state characteristics. We found student demographic characteristics had the largest effect on between-state differences in writing performance, followed by state policy-related variables, then state and school covariates. States with writing tests that exhibited greater alignment with the NAEP writing assessment demonstrated significantly higher writing scores. We discuss plausible implications of these findings.     

Defining and Measuring College and Career Readiness: A Validation Framework

This article reviews the intended uses of these college‐ and career‐readiness assessments with the goal of articulating an appropriate validity argument to support such uses. These assessments differ fundamentally from today's state assessments employed for state accountability. Current assessments are used to determine if students have mastered the knowledge and skills articulated in state standards; content standards, performance levels, and student impact often differ across states. College‐ and career‐readiness assessments will be used to determine if students are prepared to succeed in postsecondary education. Do students have a high probability of academic success in college or career‐training programs? As with admissions, placement, and selection tests, the primary interpretations that will be made from test scores concern future performance. Statistical evidence between test scores and performance in postsecondary education will become an important form of evidence. A validation argument should first define the construct (college and career readiness) and then define appropriate criterion measures. This article reviews alternative definitions and measures of college and career readiness and contrasts traditional standard‐setting methods with empirically based approaches to support a validation argument.     

Aligning Mathematics Teacher Work Sample Content with Selected NCTM Standards: Implications for Preservice Teacher Education

Teacher Work Sample Methodology has been described as an alternative means/set of procedures for assessing teacher effectiveness in producing student learning that are more authentic than traditional means of teacher certification. To investigate the degree to which the methodology aligns with state/national standards, 50 work samples produced by student teachers at Western Oregon University between fall 1991 and spring 1999 were analyzed to determine: (1) the efficiency of Teacher Work Sample Methodology in moving state and national standards, for example, the NCTM standards, into the classroom; and (2) the extent to which Teacher Work Sample Methodology promotes alignment of standards, content, instruction and assessments of instruction. The research found that a majority of the student teacher work samples demonstrated weak alignment or no alignment between stated instructional objectives and selected NCTM Curriculum and Evaluation Standards (Problem Solving, Communication, Reasoning, and Connections). However, in most of the work samples, more than half of the pre/post-assessment methods (performance, knowledge) were aligned with the instructional objectives.     

District Accountability Without a State Assessment: A Proposed Model

Most states have adopted assessment and accountability systems that involve common measures of student performance. A state assessment system that allows school districts to choose the specific strategies they use to measure student performance on state-adopted content standards presents a unique state accountability challenge. The authors propose an accountability model that addresses this challenge using a combination of student performance, technical quality, and noncognitive indicators of performance. They also describe a study that evaluated the proposed model using data from all school districts in a southern state.     

Validity of High‐Stakes Assessment: Are Students Engaged in Complex Thinking?

The validity of high-stakes assessments and accountability systems is discussed in relation to the requirements of No Child Left Behind (NCLB). The extent to which content standards and assessments are cognitively rich, the challenges in setting performance standards, and the impact of high-stakes assessments on instruction and student learning are addressed. The article argues for quality content standards, cognitively rich assessments, and a cohesive, balanced assessment system.     

Linking Results of Distinct Assessments

With increased demands for various types of assessments-from the class- room use of individual student results to international comparisons-has come an expanded desire to use assessments for multiple purposes by linking results from distinct assessments. There is a desire to make comparisons from results on one assessment with those of another (e.g., the results from a state assessment vs. the results on a national or international assessment). The degree to which desired interpretations and inferences are justified, however, depends on the nature of the assessments being compared and the ways in which the linkage occurs. Five different types of linking (equating, calibra- tion, statistical moderation, prediction, and social moderation) are distin- guished. The characteristics of these types of linking, their requirements for the assessments being linked, and the comparative inferences they support are described.     

Instructional Sensitivity of a State's Standards-Based Assessment

Abstract

The accuracy of achievement test score inferences largely depends on the sensitivity of scores to instruction focused on tested objectives. Sensitivity requirements are particularly challenging for standards-based assessments because a variety of plausible instructional differences across classrooms must be detected. For this study, we developed a new method for capturing the alignment between how teachers bring standards to life in their classrooms and how the standards are defined on a test. Teachers were asked to report the degree to which they emphasized the state's academic standards, and to describe how they taught certain objectives from the standards. Two curriculum experts judged the alignment between how teachers brought the objectives to life in their classrooms and how the objectives were operationalized on the state test. Emphasis alone did not account for achievement differences among classrooms. The best predictors of classroom achievement were the match between how the standards were taught and tested, and the interaction between emphasis and match, indicating that test scores were sensitive to instruction of the standards, but in a narrow sense.     

History and Background of Nebraska's School-based Teacher-led Assessment and Reporting System (STARS)

Nebraska's approach to standards, assessment, and accountability, the School-based Teacher-led Assessment and Reporting System (STARS) is based upon local control and the belief that classrooms and teachers must be at the heart of student learning and accountability. STARS relies on locally-developed assessment systems to accurately measure and report student performance on state content standards. Each local system in Nebraska's 500+ school districts is reviewed for technical quality, and districts are publicly rated for assessment quality and student performance. The purpose of this article is to establish the historical background.     

Policy issues: Student Performance and Congressional Support

In the first issue of this journal, I wrote about policy issues with which all stakeholders associated with at-risk children and youth should be involved (Carroll, 1996). Continuing in the policy arena, I now speak to student results. The Title I program serves more than 5 million children with a $7 billion appropriation, and school districts need only report to the state the achievement of Title I participants who are tested as part of the annual state assessment program at three grade groupings--Grades 3 to 5, 6 to 8, and 10 to 12. Districts and states are no longer required to conduct pretest and posttest assessments that show the normal curve equivalent growth of children. Instead, adequate yearly progress toward meeting the states' definitions of advanced, proficient, and partially proficient student performance measures is the new yardstick of accountability and program success. These definitions apply no later than the year 2000-2001, when the states must have their student assessments aligned with their content and student performance standards. Even though the new Title I regulations ease up on frequency and coverage of assessment, Title I schools and programs should not. Schools must assess the performance of all their students and show results if we are to garner continued financial and program support from members of Congress and out constituencies at the state and local levels.     

A Proposed Framework for Evaluating Alignment Studies

Evaluating the multiple characteristics of alignment has taken a prominent role in educational assessment and accountability systems given its attention in the No Child Left Behind legislation (NCLB). Leading to this rise in popularity, alignment methodologies that examined relationships among curriculum, academic content standards, instruction, and assessments were proposed as strategies to evaluate evidence of the intended uses and interpretations of test scores. In this article, we propose a framework for evaluating alignment studies based on similar concepts that have been recommended for standard setting (Kane). This framework provides guidance to practitioners about how to identify sources of validity evidence for an alignment study and make judgments about the strength of the evidence that may impact the interpretation of the results.     

The Rating and Matching Item-Objective Alignment Methods

Central to the standards-based assessment validation process is an examination of the alignment between state standards and test items. Several alignment analysis systems have emerged recently, but most rely on either traditional rating or matching techniques. Little, if any, analyses have been reported on the degree of consistency between the two methods and on the item and objective characteristics that influence judges' decisions. We randomly assigned judges to either rate item-objective links or match items to objectives while reviewing the 2004 Arizona high school mathematics standards and assessment. Across items we found moderate convergence between methods, and we detected apparent reasons for divergently scored items. We also found that judges relied on item and objective content and intellectual skill features to render decisions. Based on our evidence, we contend that a thorough alignment analysis would involve judges using both rating and matching, while focusing on both content and intellectual skill. The findings have important implications for states when examining the alignment between their standards and assessments.     

Benchmarking and Alignment of Standards and Testing

The success of standards-based education systems depends on 2 elements: strong standards, and assessments that measure what the standards expect. States that have or adopt test-based accountability programs claim that their tests are aligned to their standards. But there has been up to now no independent methodology for checking alignment. This article describes and illustrates such a methodology and reports results on a sample of state tests. In general, although individual items align quite well with some standard, the tests as a whole are not well aligned. With few exceptions, the collections of items that make up the tests that we examined do not do a good job of assessing the full range of standards and objectives that states have laid out for their students. This misalignment can have serious consequences for instruction and for the validity of test results.     

Evaluating the Predictive Value of Growth Prediction Models

This study evaluates four growth prediction models—projection, student growth percentile, trajectory, and transition table—commonly used to forecast (and give schools credit for) middle school students' future proficiency. Analyses focused on vertically scaled summative mathematics assessments, and two performance standards conditions (high rigor and low rigor) were examined. Results suggest that, when “status plus growth” is the accountability metric a state uses to reward or sanction schools, growth prediction models offer value above and beyond status‐only accountability systems in most, but not all, circumstances. Predictive growth models offer little value beyond status‐only systems if the future target proficiency cut score is rigorous. Conversely, certain models (e.g., projection) provide substantial additional value when the future target cut score is relatively low. In general, growth prediction models' predictive value is limited by a lack of power to detect students who are truly on‐track. Limitations and policy implications are discussed, including the utility of growth projection models in assessment and accountability systems organized around ambitious college‐readiness goals.     

Evaluating Content-Related Validity Evidence Using a Text-Based Machine Learning Procedure

Validity evidence based on test content is critical to meaningful interpretation of test scores. Within high-stakes testing and accountability frameworks, content-related validity evidence is typically gathered via alignment studies, with panels of experts providing qualitative judgments on the degree to which test items align with the representative content standards. Various summary statistics are then calculated (e.g., categorical concurrence, balance of representation) to aid in decision-making. In this paper, we propose an alternative approach for gathering content-related validity evidence that capitalizes on the overlap in vocabulary used in test items and the corresponding content standards, which we define as textual congruence. We use a text-based, machine learning model, specifically topic modeling, to identify clusters of related content within the standards. This model then serves as the basis from which items are evaluated. We illustrate our method by building a model from the Next Generation Science Standards, with textual congruence evaluated against items within the Oregon statewide alternate assessment. We discuss the utility of this approach as a source of triangulating and diagnostic information and show how visualizations can be used to evaluate the overall coverage of the content standards across the test items.     

Relationship between IEP Characteristics and Test Scores on an Alternate Assessment for Students with Significant Cognitive Disabilities

Although many studies have examined the alignment of state standards with large-scale assessment and instruction, fewer have attended to alignment concerning alternate assessments for students with significant disabilities. This study was designed to (1) compare expectations in one state's alternate assessment (AA) with curricular priorities reflected in students' Individualized Education Programs (IEPs), and (2) consider the effect of this relationship on AA scores. The study was conducted in a state whose AA consisted of standardized performance tasks measuring reading comprehension (RC) and number systems (NUM). Archival data, including AA scores and IEPs for 292 students, were analyzed. The average IEP emphasized speaking, writing, and measurement, and objectives primarily required simple recall skills. Half of IEPs contained no objectives aligned with RC. More than one third of IEPs did not align with NUM. Assessment–IEP alignment had a moderate effect on Reading test score, but not Math test score. Recommendations are made for future investigations of the taught curriculum for this population, and professional development to improve alignment of instruction with assessments.     

Trick or treat: new ecology of education accountability system in the USA

This study tracks American states’ policy choices under the No Child Left Behind Act and explores their consequences for student achievement. Using the path analysis of relationships among state‐level policy input, context, and outcome variables, the study portrays a Halloween‐like ‘trick‐or‐treating’ game between the federal and state governments in the new ecology of the test‐driven education accountability system. States that chose the ‘trick’ path with a calculative policy negotiation and manipulation strategy made significant gains on their own state assessments but not on the national assessment. In contrast, states that followed the ‘treat’ path with a faithful policy implementation for funding strategy have not yet brought about significant gains on either the national or state assessments. The first‐generation accountability states with a prior history of high‐stakes testing tended to employ both strategies at the same time. However, neither effective illusion nor ineffective implementation serves the goal of long‐term, sustainable academic improvement. Implications for research and policy are discussed.     

Evaluating the alignment among curriculum,instruction, and assessments: Implications and applications for research and practice

Alignment has been defined as the extent to which curricular expectations and assessments are in agreement and work together to provide guidance for educators' efforts to facilitate students' progress toward desire academic outcomes. The Council of Chief State School Officers has identified three preferred models as frameworks for evaluating alignment: Webb's alignment model, the Surveys of Enacted Curriculum model, and the Achieve model. Each model consists of a series of indices that summarize or describe the general match or coherence between state standards, large‐scale assessments, and, in some cases, classroom instruction. This article provides an overview of these frameworks for evaluating alignment and their applications in educational practice and the research literature. After providing an introduction to the use of alignment to evaluate large‐scale accountability systems, the article presents potential extensions of alignment for use with vulnerable populations (e.g., students with disabilities, preschoolers), individual students, and classroom teachers. These proposed applications can provide information for facilitating efforts to improve teachers' classroom instruction and students' educational achievement. © 2008 Wiley Periodicals, Inc.     

Examining the Alignment of Chinese National Physics Curriculum Guidelines and 12th‐grade Exit Examinations: A case study

This study reports findings from an analysis of the 2002 Chinese National Physics Curriculum Guidelines and the alignment between the curriculum guidelines and two most recent provincial‐level 12th‐grade exit examinations in China. Both curriculum guidelines and test content were represented using two‐dimensional matrices (i.e., topic by level of cognitive demands) and the Porter’s alignment indices were reported. It appeared that the curriculum documents and the standardized examinations mostly emphasized student understanding of fundamental principles and concepts of physics. Moreover, the two examinations consistently over‐represented the curriculum at both application and analysis cognitive levels. The study also indicated that neither the organization of the current curriculum guidelines nor the exit assessments encourage creativity, critical thinking, and the development of students’ abilities to conduct scientific inquiry. The findings of this study can be used for comparative studies of different countries’ science curriculum standards and assessment systems, and can provide insights into the improvement of science education from an international perspective.