Evaluating the alignment among curriculum,instruction, and assessments: Implications and applications for research and practice

Alignment has been defined as the extent to which curricular expectations and assessments are in agreement and work together to provide guidance for educators' efforts to facilitate students' progress toward desire academic outcomes. The Council of Chief State School Officers has identified three preferred models as frameworks for evaluating alignment: Webb's alignment model, the Surveys of Enacted Curriculum model, and the Achieve model. Each model consists of a series of indices that summarize or describe the general match or coherence between state standards, large‐scale assessments, and, in some cases, classroom instruction. This article provides an overview of these frameworks for evaluating alignment and their applications in educational practice and the research literature. After providing an introduction to the use of alignment to evaluate large‐scale accountability systems, the article presents potential extensions of alignment for use with vulnerable populations (e.g., students with disabilities, preschoolers), individual students, and classroom teachers. These proposed applications can provide information for facilitating efforts to improve teachers' classroom instruction and students' educational achievement. © 2008 Wiley Periodicals, Inc.     

Does Test Item Performance Increase with Test-to-Standards Alignment?

Variation in test performance among examinees from different regions or national jurisdictions is often partially attributed to differences in the degree of content correspondence between local school or training program curricula, and the test of interest. This posited relationship between test-curriculum correspondence, or “alignment,” and test performance is usually inferred from highly distal evidence, rather than directly examined. Utilizing mathematics standards content analysis data and achievement test item data from ten U.S. states, we examine the relationship between topic-specific alignment and test item performance. When a particular item’s content type is emphasized by the standards, we find evidence of a positive relationship between the alignment measure and proportion-correct test item difficulty, although this effect is not consistent across samples. Implications of the results for curricular achievement test development and score interpretation are discussed.     

Gauging Item Alignment Through Online Systems While Controlling for Rater Effects

The alignment of test items to content standards is critical to the validity of decisions made from standards‐based tests. Generally, alignment is determined based on judgments made by a panel of content experts with either ratings averaged or via a consensus reached through discussion. When the pool of items to be reviewed is large, or the content‐matter experts are broadly distributed geographically, panel methods present significant challenges. This article illustrates the use of an online methodology for gauging item alignment that does not require that raters convene in person, reduces the overall cost of the study, increases time flexibility, and offers an efficient means for reviewing large item banks. Latent trait methods are applied to the data to control for between‐rater severity, evaluate intrarater consistency, and provide item‐level diagnostic statistics. Use of this methodology is illustrated with a large pool (1,345) of interim‐formative mathematics test items. Implications for the field and limitations of this approach are discussed.     

Comparing the Effectiveness of Self‐Paced and Collaborative Frame‐of‐Reference Training on Rater Accuracy in a Large‐Scale Writing Assessment

There is a large body of research on the effectiveness of rater training methods in the industrial and organizational psychology literature. Less has been reported in the measurement literature on large‐scale writing assessments. This study compared the effectiveness of two widely used rater training methods—self‐paced and collaborative frame‐of‐reference training—in the context of a large‐scale writing assessment. Sixty‐six raters were randomly assigned to the training methods. After training, all raters scored the same 50 representative essays prescored by a group of expert raters. A series of generalized linear mixed models were then fitted to the rating data. Results suggested that the self‐paced method was equivalent in effectiveness to the more time‐intensive and expensive collaborative method. Implications for large‐scale writing assessments and suggestions for further research are discussed.     

Social Equity Theory and Racial‐Ethnic Achievement Gaps

In the United States, racial‐ethnic differences on tests of school readiness and academic achievement continue. A complete understanding of the origins of racial‐ethnic achievement gaps is still lacking. This article describes social equity theory (SET), which proposes that racial‐ethnic achievement gaps originate from two kinds of social process, direct and signal influences, that these two kinds of processes operate across developmental contexts, and that the kind of influence and the setting in which they are enacted change with age. Evidence supporting each of SET's key propositions is discussed in the context of a critical review of research on the Black–White achievement gap. Specific developmental hypotheses derived from SET are described, along with proposed standards of evidence for testing those hypotheses.     

Scale Alignment in Between‐Item Multidimensional Rasch Models

Scores estimated from multidimensional item response theory (IRT) models are not necessarily comparable across dimensions. In this article, the concept of aligned dimensions is formalized in the context of Rasch models, and two methods are described—delta dimensional alignment (DDA) and logistic regression alignment (LRA)—to transform estimated item parameters so that dimensions are aligned. Both the DDA and LRA methods are applied to real and simulated data, and it is demonstrated that both methods are broadly effective for achieving aligned scales. The routine use of scale alignment methods is recommended prior to comparing scores across dimensions.     

Standardized test outcomes for students engaged in inquiry‐based science curricula in the context of urban reform

Considerable effort has been made over the past decade to address the needs of learners in large urban districts through scaleable reform initiatives. We examine the effects of a multifaceted scaling reform that focuses on supporting standards based science teaching in urban middle schools. The effort was one component of a systemic reform effort in the Detroit Public Schools, and was centered on highly specified and developed project‐based inquiry science units supported by aligned professional development and learning technologies. Two cohorts of 7th and 8th graders that participated in the project units are compared with the remainder of the district population, using results from the high‐stakes state standardized test in science. Both the initial and scaled up cohorts show increases in science content understanding and process skills over their peers, and significantly higher pass rates on the statewide test. The relative gains occur up to a year and a half after participation in the curriculum, and show little attenuation with in the second cohort when scaling occurred and the number of teachers involved increased. The effect of participation in units at different grade levels is independent and cumulative, with higher levels of participation associated with similarly higher achievement scores. Examination of results by gender reveals that the curriculum effort succeeds in reducing the gender gap in achievement experienced by urban African‐American boys. These findings demonstrate that standards‐based, inquiry science curriculum can lead to standardized achievement test gains in historically underserved urban students, when the curriculum is highly specified, developed, and aligned with professional development and administrative support. © 2008 Wiley Periodicals, Inc. J Res Sci Teach 45: 922–939, 2008     

Opportunities and Options for Facilitating and Evaluating Access to the General Curriculum for Students With Disabilities

Federal regulations concerning the development and implementation of alternate assessments based on modified achievement standards include a set of safeguards intended to ensure that eligible students have access to grade-level general curriculum. These regulations concerning curricular access and opportunity to learn for students with disabilities may prove difficult for educational researchers and policymakers to operationalize and evaluate. This article provides a historical and policy context for efforts to ensure curricular access. In addition, this article reviews research on potential indicators of (or ways of measuring) access to the general curriculum and opportunity to learn. Finally, best practices for facilitating and evaluating access to the general curriculum are discussed.     

Test Anxiety Interventions for Children and Adolescents: A Systematic Review of Treatment Studies from 2000–2010

High‐stakes tests have played an increasingly important role in how student achievement and school effectiveness are measured. Test anxiety has risen with the use of tests in educational decision making. Students with high test anxiety perform poorly on tests when compared to students with low test anxiety. School psychologists can play an important role as experts both in tests and measurement and mental health in providing consultation and treatment for students with test anxiety. This article describes the results of a systematic literature review of the last 10 years of test‐anxiety interventions. Results indicate that there are few studies that have examined test‐anxiety interventions with elementary and secondary school students. However, techniques including biofeedback, behavior therapy, cognitive behavioral therapy, priming competency, and mixed approaches have demonstrated promising results. Suggestions are made for school psychologists for the delivery of evidenced‐based test anxiety interventions.     

Instructional Sensitivity of a State's Standards-Based Assessment

Abstract

The accuracy of achievement test score inferences largely depends on the sensitivity of scores to instruction focused on tested objectives. Sensitivity requirements are particularly challenging for standards-based assessments because a variety of plausible instructional differences across classrooms must be detected. For this study, we developed a new method for capturing the alignment between how teachers bring standards to life in their classrooms and how the standards are defined on a test. Teachers were asked to report the degree to which they emphasized the state's academic standards, and to describe how they taught certain objectives from the standards. Two curriculum experts judged the alignment between how teachers brought the objectives to life in their classrooms and how the objectives were operationalized on the state test. Emphasis alone did not account for achievement differences among classrooms. The best predictors of classroom achievement were the match between how the standards were taught and tested, and the interaction between emphasis and match, indicating that test scores were sensitive to instruction of the standards, but in a narrow sense.     

The Dependence of Growth‐Model Results on Proficiency Cut Scores

States participating in the Growth Model Pilot Program reference individual student growth against “proficiency” cut scores that conform with the original No Child Left Behind Act (NCLB). Although achievement results from conventional NCLB models are also cut‐score dependent, the functional relationships between cut‐score location and growth results are more complex and are not currently well described. We apply cut‐score scenarios to longitudinal data to demonstrate the dependence of state‐ and school‐level growth results on cut‐score choice. This dependence is examined along three dimensions: 1) rigor, as states set cut scores largely at their discretion, 2) across‐grade articulation, as the rigor of proficiency standards may vary across grades, and 3) the time horizon chosen for growth to proficiency. Results show that the selection of plausible alternative cut scores within a growth model can change the percentage of students “on track to proficiency” by more than 20 percentage points and reverse accountability decisions for more than 40% of schools. We contribute a framework for predicting these dependencies, and we argue that the cut‐score dependence of large‐scale growth statistics must be made transparent, particularly for comparisons of growth results across states.     

“I Know You Don't Know I Know…” Children Use Second‐Order False‐Belief Reasoning for Peer Coordination

Numerous studies have investigated children's abilities to attribute mental states, but few have examined their ability to recruit these abilities in social interactions. Here, 6‐year‐olds (N = 104) were tested on whether they can use first‐ and second‐order false‐belief understanding to coordinate with peers. Children adjusted their decisions in a coordination game in response to either their partner's erroneous belief or their partner's erroneous belief about their own belief—a result that contrasts with previous findings on the use of higher order “theory of mind” (TOM) reasoning at this age. Six‐year‐olds are thus able to use their higher order TOM capacities for peer coordination, which marks an important achievement in becoming competent social collaborators.     

Self‐assessment methods in writing instruction: a conceptual framework,successful practices and essential strategies

Student writing achievement is essential to lifelong learner success, but supporting writing can be challenging for teachers. Several large‐scale analyses of publications on writing have called for further study of instructional methods, as the current literature does not sufficiently address the need to support best teaching practices. Self‐assessment methods in writing instruction present meaningful ways to promote student writing achievement through reflection and meta‐cognition during the writing process. Self‐assessment practices are described in publications across the academic disciplines where writing is required, and close investigations of the methods are also published in the research fields of literacy, English education and composition. This study aims to bridge these fields to construct a clear and comprehensive understanding of self‐assessment methods. Self‐assessment encompasses a wide range of practices and varied terminology. It is essential to clarify effective methods of self‐assessment and to disseminate practical information to educators. This paper presents the findings of an analysis of the literature yielding: (1) a theoretical framework for self‐assessment methods in writing instruction; (2) an overview of concepts/practices; (3) a list of literature‐supported strategies for effective classroom use derived from a study of a vast body of literature. Results strongly support self‐assessment as a means to foster student writing achievement, middle school through higher education. Delineated strategies can assist instructors with self‐assessment implementation.     

Local Observed‐Score Kernel Equating

Three local observed‐score kernel equating methods that integrate methods from the local equating and kernel equating frameworks are proposed. The new methods were compared with their earlier counterparts with respect to such measures as bias—as defined by Lord's criterion of equity—and percent relative error. The local kernel item response theory observed‐score equating method, which can be used for any of the common equating designs, had a small amount of bias, a low percent relative error, and a relatively low kernel standard error of equating, even when the accuracy of the test was reduced. The local kernel equating methods for the nonequivalent groups with anchor test generally had low bias and were quite stable against changes in the accuracy or length of the anchor test. Although all proposed methods showed small percent relative errors, the local kernel equating methods for the nonequivalent groups with anchor test design had somewhat larger standard error of equating than their kernel method counterparts.     

Parents’ self‐efficacy beliefs,parents’ gender,children’s reader self‐perceptions,reading achievement and gender

This study investigated the relationships among parents’ self‐efficacy beliefs, parents’ gender, children’s reader self‐perceptions, reading achievement and gender. This study consisted of 66 students, aged eight and nine, and 92 parents involved in a family literacy project for approximately one year. The study was conducted in a rural area of Eastern Canada. There were three instruments used in this study: a Questionnaire for Parents, a Reader Self‐Perception Scale (RSPS) (Henk & Melnick, 1995), and a standardised reading test (Test of Early Reading Ability‐2 – TERA‐2) (Reid, Hresko & Hammill, 1989). The Pearson‐Product‐Moment method and t‐tests were used to determine relationships in the data and to identify significant differences in scores on the instruments. Significant positive and negative relationships were found between mothers’ and fathers’ self‐efficacy beliefs and children’s reader self‐perceptions. Children’s self‐perceptions as readers significantly related to their reading achievement. Mothers had stronger beliefs than did fathers in their ability to help improve boys’ reading achievement. Significant differences favouring females were found in children’s reader self‐perceptions and their reading achievement. The findings of this study provide a basis for understanding factors related to young children’s reading achievement.     

Using Retest Data to Evaluate and Improve Effort‐Moderated Scoring

There has been a growing research interest in the identification and management of disengaged test taking, which poses a validity threat that is particularly prevalent with low‐stakes tests. This study investigated effort‐moderated (E‐M) scoring, in which item responses classified as rapid guesses are identified and excluded from scoring. Using achievement test data composed of test takers who were quickly retested and showed differential degrees of disengagement, three basic findings emerged. First, standard E‐M scoring accounted for roughly one‐third of the score distortion due to differential disengagement. Second, a modified E‐M scoring method that used more liberal time thresholds performed better—accounting for two‐thirds or more of the distortion. Finally, the inability of E‐M scoring to account for all of the score distortion suggests the additional presence of nonrapid item responses that reflect less‐than‐full engagement by some test takers.     

Urban African‐American middle school science students: Does standards‐based teaching make a difference?

The current reform movement in science education promotes standards‐based teaching, including the use of inquiry, problem solving, and open‐ended questioning, to improve student achievement. This study examines the influence of standards‐based teaching practices on the achievement of urban, African‐American, middle school science students. Science classes of teachers who had participated in the professional development (n = 8) of Ohio's statewide systemic initiative (SSI) were matched with classes of teachers (n = 10) who had not participated. Data were gathered using group‐administered questionnaires and achievement tests that were specifically designed for Ohio's SSI. Analyses indicate that teachers who frequently used standards‐based teaching practices positively influenced urban, African‐American students' science achievement and attitudes, especially for boys. Additionally, teachers' involvement in the SSI's professional development was positively related to the reported use of standards‐based teaching practices in the classroom. The findings support the efficacy of high‐quality professional development to change teaching practices and to enhance student learning. © 2000 John Wiley & Sons, Inc. J Res Sci Teach 37: 1019–1041, 2000     

Computer‐Based Counselor‐in‐Training Supervision: Ethical and Practical Implications for Counselor Educators and Supervisors

Computer‐based clinical supervision of counselors‐in‐training is becoming more prevalent (M. Reisch & L. Jarman‐Rohde, 2000); however, its use is still in its infancy, and ethical standards have not been established regarding its practice. There exists a dearth of literature focusing on the ethical practice and development of supervisees when using computer‐based supervision. This article (a) explores ethical practice when using technology to facilitate counselor supervision, (b) reviews specific types of computer‐based applications, (c) presents potential advantages and disadvantages of computer‐based supervision, and (d) offers implications for counselor educators and supervisors.     

Relations of social‐emotional functioning and kindergarten academic achievement in children of immigrants

Nearly one in four students residing in the United States is from an immigrant family and these children's school readiness is related to their parent's nativity and other sociodemographic characteristics. Social‐emotional skills are an important conduit for academic development, yet these relations have not been explored for children from immigrant families. This study utilized the Early Childhood Longitudinal Study—Kindergarten Class of 2010–2011, a nationally representative sample of 13,400 students in the United States, to compare the social‐emotional development of kindergarten students from immigrant and nonimmigrant families, and to determine the relations of social‐emotional functioning to kindergarten achievement. Results indicate elevated social‐emotional functioning among children from immigrant families, particularly those who emigrated immigrated from Mexico, compared with children of U.S.‐born parents. Parent nativity predicted reading achievement, but not mathematics performance, even when controlling for sociodemographic factors and social‐emotional skills. This study suggests an immigrant advantage in early social‐emotional development. Implications for research and practice are discussed.