判断评价与课程标准一致性的若干问题

Webb一致性程序是判断评价与课程标准一致性的重工具,它有4个标准:类别一致性、知识深度一致性、知识范围一致性和知识分布平衡性,分别对应着多级可接受水平。一致性研究的挑战在于判断"什么样的一致性才是最好的",这一判断过程虽然可以基于众多实证研究,但也包含某种程度的主观性。     

Issues Related to Judging the Alignment of Curriculum Standards and Assessments

Abstract

A process for judging the alignment between curriculum standards and assessments developed by the author is presented. This process produces information on the relationship of standards and assessments on four alignment criteria: Categorical Concurrence, Depth of Knowledge Consistency, Range of Knowledge Correspondence, and Balance of Representation. Five issues are identified—but not resolved—that have arisen from conducting alignment studies. All of these issues relate to making a decision about what alignment is good enough. Pragmatic decisions have been made to specify acceptable levels for each of the alignment criteria. The assumptions are described. The issues discussed arise from a change in the underlying assumptions and from considering variations in the purpose for an assessment. The existence of such issues reinforces that alignment judgments have an element of subjectivity.     

Methodological Choices in the Content Analysis of Textbooks for Measuring Alignment With Standards

With the recent adoption of the Common Core standards in many states, there is a need for quality information about textbook alignment to standards. While there are many existing content analysis procedures, these generally have little, if any, validity or reliability evidence. One exception is the Surveys of Enacted Curriculum (SEC), which has been widely used to analyze the alignment among standards, assessments, and teachers’ instruction. However, the SEC can be time‐consuming and expensive when used for this purpose. This study extends the SEC to the analysis of entire mathematics textbooks and investigates whether the results of SEC alignment analyses are affected if the content analysis procedure is simplified. The results indicate that analyzing only every fifth item produces nearly identical alignment results with no effect on the reliability of content analyses.     

The alignment between the fourth grade Lebanese science curriculum and classroom assessments

The purpose of this study was to investigate the alignment between the fourth-grade summative classroom assessments and the ‘Matter and Energy’ unit in the Lebanese science curriculum using: content and cognitive levels. Summative assessments were collected from a sample of 17 schools. The alignment between the classroom assessments and the Lebanese science curriculum were examined using three instruments: Curriculum Coding Sheet, Items Coding Sheet, and Item Analysis Sheet. The curricular learning objectives were used for matching the test items to the learning objectives in terms of content. Findings showed moderate content alignment; further analysis revealed that public schools had higher content alignment, and they covered a wider range of topics. In terms of cognitive level, there was a low alignment between the assessments collected and the curriculum; however, there were items that had higher cognitive level. Recommendations for curriculum developers and future research are discussed in light of these findings.     

Aligning Mathematics Teacher Work Sample Content with Selected NCTM Standards: Implications for Preservice Teacher Education

Teacher Work Sample Methodology has been described as an alternative means/set of procedures for assessing teacher effectiveness in producing student learning that are more authentic than traditional means of teacher certification. To investigate the degree to which the methodology aligns with state/national standards, 50 work samples produced by student teachers at Western Oregon University between fall 1991 and spring 1999 were analyzed to determine: (1) the efficiency of Teacher Work Sample Methodology in moving state and national standards, for example, the NCTM standards, into the classroom; and (2) the extent to which Teacher Work Sample Methodology promotes alignment of standards, content, instruction and assessments of instruction. The research found that a majority of the student teacher work samples demonstrated weak alignment or no alignment between stated instructional objectives and selected NCTM Curriculum and Evaluation Standards (Problem Solving, Communication, Reasoning, and Connections). However, in most of the work samples, more than half of the pre/post-assessment methods (performance, knowledge) were aligned with the instructional objectives.     

Comparing State High School Assessments to Standards for Success in Entry-Level University Courses

States have moved rapidly over the past 20 years to institute systems of standards and assessments. State assessments in particular take on added importance at the high school level as they are required for graduation by an increasing number of states. Federal legislation mandating testing in high school also serves to increase the stakes and impact of state exams. Many states are also using high school exams for postsecondary purposes, although the content and criterion validity of these exams in relation to students' post-high school pursuits is not well documented. Though no state exam was developed with the express intent of aligning specifically with postsecondary education, it is nonetheless important to understand this linkage given the wide-ranging use of high school exams across the country. This study analyzed the content of state tests relative to a set of standards that identify knowledge and skills necessary for success in entry-level university courses. A total of 60 math and English assessments from 20 states were analyzed along a number of alignment dimensions. Exams were found to be moderately aligned with a subset of the university standards, but in an uneven fashion. English exams were somewhat more aligned than math exams, but math exams had high alignment in some specific standard areas, and English exams aligned poorly or not at all in areas requiring higher order thinking. In the future, states using high school exams for postsecondary purposes may want to examine the content of state standards and exams to determine their relationship to college-readiness criteria.     

Alignment and Implications for Test Takers

The use of assessment results to inform school accountability relies on the assumption that the test design appropriately represents the content and cognitive emphasis reflected in the state's standards. Since the passage of the Every Student Succeeds Act and the certification of accountability assessments through federal peer review practices, the content validity arguments supporting accountability have relied almost exclusively on the alignment of statewide assessments to state standards. It is assumed that if alignment does not hold, the scores will not provide valid inferences regarding the degree to which test takers have performed. Although alignment results are commonly used as evidence of test appropriateness, Polikoff (this issue) would argue that given the importance of alignment in policy decisions, research related to alignment is surprisingly limited. Few studies have addressed the adequacy of alignment methodologies and results as support for the inferences to be made (i.e., proficient on state standards). This paper uses an example of test taker performance (and common performance indicators) to investigate to what extent the degree of alignment impacts inferences made about performance (i.e., classification into performance levels, estimates of student ability, and student rank order).     

Validity of High‐Stakes Assessment: Are Students Engaged in Complex Thinking?

The validity of high-stakes assessments and accountability systems is discussed in relation to the requirements of No Child Left Behind (NCLB). The extent to which content standards and assessments are cognitively rich, the challenges in setting performance standards, and the impact of high-stakes assessments on instruction and student learning are addressed. The article argues for quality content standards, cognitively rich assessments, and a cohesive, balanced assessment system.     

Aligning an Early Childhood Assessment to State Kindergarten Content Standards: Application of a Nationally Recognized Alignment Framework

This article describes an alignment study conducted to evaluate the alignment between Indiana's Kindergarten content standards and items on the Indiana Standards Tool for Alternate Reporting. Alignment is the extent to which standards and assessments are in agreement, working together to guide educators' efforts to support children's learning and development. The alignment process in this study represented a modification of Webb's nationally recognized method of alignment analysis to early childhood assessments and standards. The alignment panel (N = 13) in this study consisted of early childhood educators and educational leaders from all geographic regions of the state. Panel members were asked to rate the depth of knowledge (DOK) stage of each objective in Kindergarten standards; rate the DOK stage for each item on the ISTAR rating scale; and identify the one or two objectives from the standards to which each ISTAR item corresponded. Analysis of the panel's responses suggested the ISTAR inconsistently conformed to Webb's DOK consistency and ROK correspondence criteria for alignment. A promising finding was the strong alignment of the ISTAR Level F1 and F2 scales to the Kindergarten standards. This result provided evidence of the developmental continuum of skills and knowledge that are assessed by the ISTAR items .     

Evaluating the alignment among curriculum,instruction, and assessments: Implications and applications for research and practice

Alignment has been defined as the extent to which curricular expectations and assessments are in agreement and work together to provide guidance for educators' efforts to facilitate students' progress toward desire academic outcomes. The Council of Chief State School Officers has identified three preferred models as frameworks for evaluating alignment: Webb's alignment model, the Surveys of Enacted Curriculum model, and the Achieve model. Each model consists of a series of indices that summarize or describe the general match or coherence between state standards, large‐scale assessments, and, in some cases, classroom instruction. This article provides an overview of these frameworks for evaluating alignment and their applications in educational practice and the research literature. After providing an introduction to the use of alignment to evaluate large‐scale accountability systems, the article presents potential extensions of alignment for use with vulnerable populations (e.g., students with disabilities, preschoolers), individual students, and classroom teachers. These proposed applications can provide information for facilitating efforts to improve teachers' classroom instruction and students' educational achievement. © 2008 Wiley Periodicals, Inc.     

Content and alignment of state writing standards and assessments as predictors of student writing achievement: an analysis of 2007 National Assessment of Educational Progress data

We examined the degree to which content of states’ writing standards and assessments (using measures of content range, frequency, balance, and cognitive complexity) and their alignment were related to student writing achievement on the 2007 National Assessment of Educational Progress (NAEP), while controlling for student, school, and state characteristics. We found student demographic characteristics had the largest effect on between-state differences in writing performance, followed by state policy-related variables, then state and school covariates. States with writing tests that exhibited greater alignment with the NAEP writing assessment demonstrated significantly higher writing scores. We discuss plausible implications of these findings.     

The Rating and Matching Item-Objective Alignment Methods

Central to the standards-based assessment validation process is an examination of the alignment between state standards and test items. Several alignment analysis systems have emerged recently, but most rely on either traditional rating or matching techniques. Little, if any, analyses have been reported on the degree of consistency between the two methods and on the item and objective characteristics that influence judges' decisions. We randomly assigned judges to either rate item-objective links or match items to objectives while reviewing the 2004 Arizona high school mathematics standards and assessment. Across items we found moderate convergence between methods, and we detected apparent reasons for divergently scored items. We also found that judges relied on item and objective content and intellectual skill features to render decisions. Based on our evidence, we contend that a thorough alignment analysis would involve judges using both rating and matching, while focusing on both content and intellectual skill. The findings have important implications for states when examining the alignment between their standards and assessments.     

Alternate Assessments of Students with Significant Disabilities: Alternative Approaches,Common Technical Challenges

This article examines three typical approaches to alternate assessment for students with significant cognitive disabilities—portfolios, performance assessments, and rating scales. A detailed analysis of common and unique design features of these approaches is provided, including features of each approach that influence the psychometric quality of their results. Validity imperatives for alternate assessments are reviewed, and approaches for addressing the need for validity evidence are outlined. The article concludes with an examination of three technical challenges—alignment, scores and scoring, and standard setting—common to all alternate assessments. In light of these challenges, existing methods and professional testing standards are endorsed as necessary guidance for understanding and advancing alternate assessment practices.     

Teacher beliefs and teacher behaviour in portfolio assessment

This study concerns the relation between teachers’ beliefs towards teaching behaviour and their actual teaching behaviour in teacher portfolio assessment. We analysed the beliefs and behaviour of 18 teachers as described in their portfolios. In addition, each portfolio was independently assessed by two trained raters on eight content standards and the teachers’ classroom behaviour was assessed by their own students in a questionnaire (n=317). Linear multilevel analysis showed that part of the raters’ assessments of the teachers’ beliefs and their behaviour as described in their portfolios was significantly related to the students’ assessments of their teachers’ behaviour. Teachers with high raters’ assessments on the content standard about ‘choosing and arguing for teaching strategies that meet students’ knowledge, abilities and experience’ had significantly higher student assessments than teachers who were judged low on this standard. Implications of the results and suggestions for further research are discussed.     

A Proposed Framework for Evaluating Alignment Studies

Evaluating the multiple characteristics of alignment has taken a prominent role in educational assessment and accountability systems given its attention in the No Child Left Behind legislation (NCLB). Leading to this rise in popularity, alignment methodologies that examined relationships among curriculum, academic content standards, instruction, and assessments were proposed as strategies to evaluate evidence of the intended uses and interpretations of test scores. In this article, we propose a framework for evaluating alignment studies based on similar concepts that have been recommended for standard setting (Kane). This framework provides guidance to practitioners about how to identify sources of validity evidence for an alignment study and make judgments about the strength of the evidence that may impact the interpretation of the results.     

Does Test Item Performance Increase with Test-to-Standards Alignment?

Variation in test performance among examinees from different regions or national jurisdictions is often partially attributed to differences in the degree of content correspondence between local school or training program curricula, and the test of interest. This posited relationship between test-curriculum correspondence, or “alignment,” and test performance is usually inferred from highly distal evidence, rather than directly examined. Utilizing mathematics standards content analysis data and achievement test item data from ten U.S. states, we examine the relationship between topic-specific alignment and test item performance. When a particular item’s content type is emphasized by the standards, we find evidence of a positive relationship between the alignment measure and proportion-correct test item difficulty, although this effect is not consistent across samples. Implications of the results for curricular achievement test development and score interpretation are discussed.     

Refining Methods for Estimating Critical Values for an Alignment Index

Abstract

The alignment among standards, assessments, and teachers’ instruction is an essential element of standards-based educational reforms. The Surveys of Enacted Curriculum (SEC) is the only common tool that can be used to measure the alignment among all three of these sources (Martone & Sireci, 2009). Prior SEC alignment work has been limited by not allowing for significance tests. A recent article (Fulmer, 2011) provided a first attempt to address this shortcoming of the SEC, but that work was limited in several ways. We extend Fulmer's simulation approach by accounting for important elements of the SEC procedures, including the proper framework size, number of standards and assessment points, number of raters, rater cell-splitting rates, and rater agreement results. The results indicate that inferences about relative alignment may be heavily influenced by features of the alignment procedures. Thus, our method should be broadly applied to future SEC alignment investigations.     

Gauging Item Alignment Through Online Systems While Controlling for Rater Effects

The alignment of test items to content standards is critical to the validity of decisions made from standards‐based tests. Generally, alignment is determined based on judgments made by a panel of content experts with either ratings averaged or via a consensus reached through discussion. When the pool of items to be reviewed is large, or the content‐matter experts are broadly distributed geographically, panel methods present significant challenges. This article illustrates the use of an online methodology for gauging item alignment that does not require that raters convene in person, reduces the overall cost of the study, increases time flexibility, and offers an efficient means for reviewing large item banks. Latent trait methods are applied to the data to control for between‐rater severity, evaluate intrarater consistency, and provide item‐level diagnostic statistics. Use of this methodology is illustrated with a large pool (1,345) of interim‐formative mathematics test items. Implications for the field and limitations of this approach are discussed.