State Perspectives on Implementing,or Choosing Not to Implement,an Alternate Assessment Based on Modified Academic Achievement Standards

Since Federal regulations have given states the option to implement alternate assessments based on modified academic achievement standards (AA-MAS) as part of their accountability systems for a small group of students with disabilities, a number of states have made decisions about whether or not to develop and implement such an assessment. State-level directors of assessment and directors of special education were surveyed about their state's decisions on implementing AA-MAS. Improvements in accessibility and appropriateness were reasons given for choosing to implement an AA-MAS, whereas lack of resources and guidance were identified as barriers. This article presents the findings from a survey on 22 states' decisions concerning implementation of AA-MAS.     

Introduction to the Special Issue on Alternate Assessments Based on Modified Academic Achievement Standards: New Policy,New Practices,and Persistent Challenges

This introduction to the special issue titled Alternate Assessments Based on Modified Academic Achievement Standards: New Policy, New Practices, and Persistent Challenges addresses the federal policy introducing the new alternate assessment for students with persistent academic difficulties, as well as related implementation issues that will be more thoroughly considered throughout the journal. Three guidelines are identified within the policy for alternate assessments based on modified academic achievement standards (AA-MASs), including that (a) a state's grade-level academic content standards cannot be modified for an AA-MAS, (b) a state's general test can be modified for an AA-MAS, and (c) a state's achievement standards can be modified for an AA-MAS so long as they remain on grade level. This article introduces key issues including identification of students eligible for an AA-MAS, the degree of modification that can be applied to develop an AA-MAS, and the current state of AA-MAS development across the nation. The article concludes with overviews of each contribution in the journal.     

The “Two Percent Students”: Considerations and Consequences of Eligibility Decisions

Since the 2001–02 school year, the accountability provisions of the No Child Left Behind Act (NCLB) have shaped much of the work of public school teachers and administrators in the United States. NCLB explicitly prohibits schools from excluding students with disabilities from the accountability system and requires not only participation of all students in statewide accountability assessments but also reporting of the results for students with disabilities along with other students and as a disaggregated group. From the beginning of these requirements, lawmakers recognized that there would be a small group of students with disabilities for whom the regular assessment, even with accommodations, would not be appropriate and they authorized states to develop an alternate assessment based on alternate achievement standards (AA-AAS) for this group of students. More recently, responding to pressures from the field, additional flexibility has been granted to develop an additional alternate assessment based on modified grade-level achievement standards (AA-MAS) for students with disabilities who present with persistent academic difficulties. It is expected that approximately 2% of the total student population might be included in this new alternate assessment. This article examines the decisions that need to be made by individual states to determine the target population for this new alternate assessment and the policy implications of these decisions.     

Considering the Use of General and Modified Assessment Items in Computerized Adaptive Testing

This article used several data sets from a large-scale state testing program to examine the feasibility of combining general and modified assessment items in computerized adaptive testing (CAT) for different groups of students. Results suggested that several of the assumptions made when employing this type of mixed-item CAT may not be met for students with disabilities that have typically taken alternate assessments based on modified achievement standards (AA-MAS). A simulation study indicated that the abilities of AA-MAS students can be underestimated or overestimated by the mixed-item CAT, depending on students’ location on the underlying ability scale. These findings held across grade levels and test lengths. The mixed-item CAT appeared to function well for non-AA-MAS students.     

Modifying Achievement Test Items: A Theory-Guided and Data-Based Approach for Better Measurement of What Students With Disabilities Know

Federal regulations allow up to 2% of the student population of a state to achieve proficiency for adequate yearly progress by taking an alternate assessment based on modified academic achievement standards (AA-MAS). Such tests are likely to be easier, but as long as a test is considered a valid measure of grade level content, it is allowable as an AA-MAS (U.S. Department of Education, 2007b U.S. Department of Education. 2007b. Standards and assessments peer review guidance, Washington, DC: Author.  [Google Scholar]). In this article, we examine procedures for developing, modifying, and evaluating items and tests using an evolving modification paradigm, as well as a classic reliability and validity framework. Theoretical influences, such as principles of universal design, cognitive load theory, and item development research, are discussed. The Test Accessibility and Modification Inventory, a tool that provides systematic and comprehensive guidance to help educators modify grade-level tests, is introduced. Cognitive lab methods and experimental field tests are then described, along with examples and key findings from each, relevant to AA-MASs. The article concludes with a discussion of precautions, lessons learned, and questions generated about the methods used to improve both access and test score validity for the students who are eligible for this new alternate assessment.     

Policy issues: Student Performance and Congressional Support

In the first issue of this journal, I wrote about policy issues with which all stakeholders associated with at-risk children and youth should be involved (Carroll, 1996). Continuing in the policy arena, I now speak to student results. The Title I program serves more than 5 million children with a $7 billion appropriation, and school districts need only report to the state the achievement of Title I participants who are tested as part of the annual state assessment program at three grade groupings--Grades 3 to 5, 6 to 8, and 10 to 12. Districts and states are no longer required to conduct pretest and posttest assessments that show the normal curve equivalent growth of children. Instead, adequate yearly progress toward meeting the states' definitions of advanced, proficient, and partially proficient student performance measures is the new yardstick of accountability and program success. These definitions apply no later than the year 2000-2001, when the states must have their student assessments aligned with their content and student performance standards. Even though the new Title I regulations ease up on frequency and coverage of assessment, Title I schools and programs should not. Schools must assess the performance of all their students and show results if we are to garner continued financial and program support from members of Congress and out constituencies at the state and local levels.     

Perceived Competence Scale for Children: Testing for Factorial Validity and Invariance Across Age and Ability

Given the policy imperative of using multiple measures for state education accountability under the No Child Left Behind Act (NCLB), this study examines similarities and discrepancies between the National Assessment of Educational Progress (NAEP) and the states' own math assessment results in Kentucky and Maine, with a focus on 3 major academic performance indicators: proficiency level, achievement gap, and achievement gain. Using meta-analytic techniques, the study synthesizes multiple measures from the two states over the periods of 1992–1996 and 2000–2003. It pinpoints the areas and degrees of the discrepancies and explores contributing factors. It also reports emerging convergence of the NAEP and state assessments under the NCLB.     

Evaluating the Instructional Sensitivity of Four States' Student Achievement Tests

As state tests of student achievement are used for an increasingly wide array of high- and low-stakes purposes, evaluating their instructional sensitivity is essential. This article uses data from the Bill and Melinda Gates Foundation's Measures of Effective Project to examine the instructional sensitivity of 4 states' mathematics and English language arts assessments to 5 measures of pedagogical quality. The results indicate modest overall levels of sensitivity, with some state assessments insensitive to most or all measures of pedagogical quality. Furthermore, there are sometimes substantial differences across states in the sensitivity of their assessments. The article calls for more research on what makes state tests instructionally sensitive and cautions against the use of insensitive assessments for consequential decisions.     

Participation of psychologists in multidisciplinary team evaluations

State implementation plans required by PL 94–142 were collected and reviewed to determine whether psychologists were mentioned as appropriate members of multidisciplinary evaluation teams. Questionnaire data from 625 directors of special education also were analyzed to determine the frequency with which psychologists actually participated on the evaluation teams. The review revealed an apparent reluctance on the part of states to specify psychologists as appropriate team members. When psychologists were mentioned, school psychologists and other types of psychologists were identified with near equal frequencies. Even though states' implementation guidelines infrequently identified psychologists for participation on multidisciplinary teams, 81% of the surveyed teams actually included a psychologist. However, psychologists' rate of participation varied among the states. Implications of the minimum state requirements and varying participation of psychologists among states are discussed.     

The teachers college faces the future in the selection of teachers

This article examines the educational conditions and resources, institutional characteristics, and political factors that contribute to shaping states' educational accountability policies. The policies addressed are identification of underperforming schools, state takeover or reconstitution of failing schools, and state power to replace principals and teachers in underperforming schools. The article addresses accountability for schools and educators, rather than high-stakes tests for students, because of the emphases of the federal No Child Left Behind Act (NCLB). Based on correlations and cross-sectional, logistic regression, the most important predictors of states' accountability policies appear to be National Assessment of Educational Progress NAEP performance, state education agency staff resources, and Democratic party identification. However, the impact of Democratic party identification is reduced when the southern states are excluded. The article concludes with a discussion of directions for further quantitative and qualitative research, as well as predictions of how NCLB is likely to affect states' accountability policies.     

Modified Multiple-Choice Items for Alternate Assessments: Reliability,Difficulty, and Differential Boost

Federal policy on alternate assessment based on modified academic achievement standards (AA-MAS) inspired this research. Specifically, an experimental study was conducted to determine whether tests composed of modified items would have the same level of reliability as tests composed of original items, and whether these modified items helped reduce the performance gap between AA-MAS eligible and ineligible students. Three groups of eighth-grade students (N?=?755) defined by eligibility and disability status took original and modified versions of reading and mathematics tests. In a third condition, the students were provided limited reading support along with the modified items. Changes in reliability across groups and conditions for both the reading and mathematics tests were determined to be minimal. Mean item difficulties within the Rasch model were shown to decrease more for students who would be eligible for the AA-MAS than for non-eligible groups, revealing evidence of differential boost. Exploratory analyses indicated that shortening the question stem may be a highly effective modification, and that adding graphics to reading items may be a poor modification.     

Happy Marriage or Uneasy Alliance? The Relationship Between Comprehensive School Reform and State Accountability Systems

The purpose of this article is to examine how the relationship between comprehensive school reform (CSR) and state accountability systems helps or hinders school improvement efforts. This article draws on case study data collected in schools in 3 states that received funding to implement reforms through the federal CSR program. Findings show that the 3 states all had high-stakes accountability systems in place (albeit of varying levels of maturity) and these accountability systems often had much more of an impact on teacher practice than the CSR models themselves. In some cases, there was synergy between the models and the states' own reform agendas, but in other cases, teachers felt as if they were being asked to achieve 2 independent goals. The states varied in their levels of support of schools' CSR efforts, depending on the primacy of CSR in the state reform picture and the political will and experience of the individuals in charge of CSR at the state level. Implications for the policy and practice of states, reform design teams, and district and school educators are discussed.     

Content and alignment of state writing standards and assessments as predictors of student writing achievement: an analysis of 2007 National Assessment of Educational Progress data

We examined the degree to which content of states’ writing standards and assessments (using measures of content range, frequency, balance, and cognitive complexity) and their alignment were related to student writing achievement on the 2007 National Assessment of Educational Progress (NAEP), while controlling for student, school, and state characteristics. We found student demographic characteristics had the largest effect on between-state differences in writing performance, followed by state policy-related variables, then state and school covariates. States with writing tests that exhibited greater alignment with the NAEP writing assessment demonstrated significantly higher writing scores. We discuss plausible implications of these findings.     

Suggestions for Utilizing the 2008 EPAS in CSWE-Accredited Baccalaureate and Masters Curriculums—Reflections from the Field,Part 2: The Implicit Curriculum

This article is Part 2 in a 2-part series discussing the new guidelines for Educational Policy and Accreditation Standards (EPAS) issued in April 2008 by the Council on Social Work Education. The 2008 EPAS shifted the focus of assessment for accreditation or reaffirmation from the evaluation of program objectives to assessment of educational outcomes and student achievement of practice competencies. Because major accreditation challenges for social work programs derive from the 2008 EPAS, this article continues the discussion of a model for sequencing accreditation tasks that began in Volume 30, Issue 2 of this journal. In this article, Part 2, the authors discuss the program's implicit curriculum and its assessment under the 2008 EPAS. The articles in this 2-part series are intended to be companion pieces.     

Trick or treat: new ecology of education accountability system in the USA

This study tracks American states’ policy choices under the No Child Left Behind Act and explores their consequences for student achievement. Using the path analysis of relationships among state‐level policy input, context, and outcome variables, the study portrays a Halloween‐like ‘trick‐or‐treating’ game between the federal and state governments in the new ecology of the test‐driven education accountability system. States that chose the ‘trick’ path with a calculative policy negotiation and manipulation strategy made significant gains on their own state assessments but not on the national assessment. In contrast, states that followed the ‘treat’ path with a faithful policy implementation for funding strategy have not yet brought about significant gains on either the national or state assessments. The first‐generation accountability states with a prior history of high‐stakes testing tended to employ both strategies at the same time. However, neither effective illusion nor ineffective implementation serves the goal of long‐term, sustainable academic improvement. Implications for research and policy are discussed.     

Aggregating Student Achievement Trends Across States With Different Tests: Using Standardized Slopes as Effect Sizes

The study of federal education initiatives that takes place over multiple years in multiple settings often calls for aggregating and comparing data-in particular, student achievement data-across a broad set of schools, districts, and states. The need to track the trends over time is complicated by the fact that the data from the different schools, districts, and states also may have been based on different achievement tests. This article suggests one approach for defining a common metric: calculating the standardized slope of a time series of datapoints. The standardized slope serves as an effect size statistic. Meta-analytic techniques can be employed because the student achievement test results from different states can therefore be considered equivalent to the findings from different studies.     

An Application of Generalizability Theory to Evaluate the Technical Quality of an Alternate Assessment

Although federal regulations require testing students with severe cognitive disabilities, there is little guidance regarding how technical quality should be established. It is known that challenges exist with documentation of the reliability of scores for alternate assessments. Typical measures of reliability do little in modeling multiple sources of error, which are characteristic of alternate assessments. Instead, Generalizability theory (G-theory) allows researchers to identify sources of error and analyze the relative contribution of each source. This study demonstrates an application of G-theory to examine reliability for an alternate assessment. A G-study with the facets rater type, assessment attempts, and tasks was examined to determine the relative contribution of each to observed score variance. Results were used to determine the reliability of scores. The assessment design was modified to examine how changes might impact reliability. As a final step, designs that were deemed satisfactory were evaluated regarding the feasibility of adapting them into a statewide standardized assessment and accountability program.     

Navigating the Measurement and Monitoring Maze

Assessment of students with disabilities is a critical component of special education. In addition to the complexity of assessment (e.g., high-stakes assessments, progress monitoring, diagnosis), the issue is complicated further by the use of acronyms. This can make both explaining and understanding important information related to students very difficult. Acronyms related to assessment create a virtual alphabet soup that can be confusing for parents, teachers, and school personnel. Assessment acronyms include terms related to both state testing and progress monitoring such as AYP, AA-AAS, AA-MAS, AA-GLAS, RTI, CBA, and CBM. Educators must understand not only the nature and purpose of the various assessments, but they must also be familiar with the assessment acronyms in order to select appropriate assessments to measure the academic progress and performance of students, and to communicate effectively with families and related school professionals. The purpose of this article is to describe common assessments administered in the field of education today as well as to demystify the use of the acronyms related to these assessments and implications for teachers and other educators. Additionally, we point to some resources that can be of value to individuals wishing to establish or expand their repertoires of assessment practices as the current emphasis on progress monitoring continues to unfold.     

Towards a framework for the validation of early childhood assessment systems

American early childhood education is in the midst of drastic change. In recent years, states have begun the process of overhauling early childhood education systems in response to federal grant competitions, bringing an increased focus on assessment and accountability for early learning programs. The assessment of young children is fraught with challenges; psychometricians and educational researchers must work together with the early childhood community to develop these instruments. The purpose of this paper is to present a conceptual framework for the validation of such instrumentation and examine its implications for early childhood educators. We formulate a validity argument for early childhood assessments providing a pivotal link between validity theory and early education practice. Recommendations for the assessment field are also considered.