Reliability and Validity of Bookmark‐Based Methods for Standard Setting: Comparisons to Angoff‐Based Methods in the National Assessment of Educational Progress

Historically, Angoff‐based methods were used to establish cut scores on the National Assessment of Educational Progress (NAEP). In 2005, the National Assessment Governing Board oversaw multiple studies aimed at evaluating the reliability and validity of Bookmark‐based methods via a comparison to Angoff‐based methods. As the Board considered adoption of Bookmark‐based methods, it considered several criteria, including reliability of the cut scores, validity of the cut scores as evidenced by comparability of results to those from Angoff, and procedural validity as evidenced by panelist understanding of the method tasks and instructions and confidence in the results. As a result of their review, a Bookmark‐based method was adopted for NAEP, and has been used since that time. This article goes beyond the Governing Board's initial evaluations to conduct a systematic review of 27 studies in NAEP research conducted over 15 years. This research is used to evaluate Bookmark‐based methods on key criteria originally considered by the Governing Board. Findings suggest that Bookmark‐based methods have comparable reliability, resulting cut scores, and panelist evaluations to Angoff. Given that Bookmark‐based methods are shorter in duration and less costly, Bookmark‐based methods may be preferable to Angoff for NAEP standard setting.     

Impact of Changing Difficulty on Inferences From the National Assessment of Educational Progress

The U.S. Department of Education measures student achievement through the National Assessment of Educational Progress (NAEP). NAEP estimates of population proficiency quantiles are based on a Bayesian multiple-imputation procedure. This article shows (a) that the resulting estimates depend directly on the mix of item difficulties on the test, and (b) the difficulty of items on the NAEP mathematics exam has increased over time. Does the increasing difficulty of the exam lead to observable changes in student performance over time? This study compared the simulated performance of 1990 examinees on the easier 1990 exam and the more difficult 1996 exam. No significant differences were found. While our results instill confidence that these changes have not impacted the NAEP trend line, our findings are both data-specific and limited in scope, and NAEP should carefully evaluate future adjustments to the test in this manner.     

A New Approach to Avoiding Problems of Scale in Interpreting Trends in Mental Measurement Data

Extracting policy-relevant information from large national surveys of educational achievement is ordinarily a nontrivial task. It is made more treacherous when the data are expressed on scales that are not uniquely determined. The paper begins with a critical analysis of a recent attempt to interpret the findings on reading achievement obtained by the National Assessment of Educational Progress (NAEP). It then describes a new approach to the quantification and interpretation of change and demonstrates its appropriateness for repeated cross-sectional designs such as NAEP. Limitations imposed by the survey design and the nature of the measurements are highlighted     

Raising the Stakes of Test Administration: The Impact on Student Performance on the National Assessment of Educational Progress

This article briefly reviews the current discussion of the effects of test administration conditions (i.e., testing stakes), and the motivational levels associated with them, on achievement test performance. The non-experimental study presented here investigates whether differences in test administration conditions and presumed levels of motivation engendered by different testing environments affect student performance on National Assessment of Educational Progress (NAEP) administrations. The testing conditions under study are the "low-stakes" environment of the current NAG administration and a higher stakes environment typified by many state assessment programs. The results suggest that in comparison to a "moderate-stakes" testing environment NAEP does not seriously underestimate achievement levels. However, the results cannot lead to the conclusion that student achievement is unrelated to testing stakes. Nor can one conclude that substantially raising the stakes of NAEP would not be accompanied by an increase in achievement scores.     

NAEP2010公民教育评估框架述评

全国教育进步评估（NAEP）是由美国国会授权的一项调查,以便收集和报告学生各门学科的成绩信息。自1969年以来,公民教育就是其评估项目之一,为了进一步了解4、8、12年级学生公民教育情况,NAEP2009年制订了2010年公民教育评估框架,计划于2010年对全美学生进行公民教育评估。文章主要介绍了2010年公民教育评估框架产生的背景、评估框架设计中考虑的问题以及评估框架的组成部分,以便为我国公民教育评估框架的研制提供借鉴。     

Making Critical Connections Between Social Studies Teaching and Student Achievement Using NAEP Data Explorer

In this analysis of promising practice, we demonstrate how social studies methods instructors can incorporate data analysis of the 2010 United States History National Assessment of Educational Progress (NAEP–USH) to facilitate pedagogical aims, engage teacher candidates in critical discourse, and investigate the contexts of teaching and learning. The NAEP data explorer application is a valuable tool for examining social studies theory and practice in relation to student learning outcomes. Our assessment of teacher candidates' responses to the activity leads to the recommendation that NAEP data analysis and results encourage self-evaluation of instructional practices while simultaneously supporting critical interpretations of the NAEP exam.     

关于三项著名国际学生评价项目的比较

全国教育进展评价 (简称NAEP)、第三次国际数学和科学教育的再研究 (简称TIMSS -R)和国际学生评价项目 (简称PISA)是当前国际间最为著名的学生评价项目 ,本文拟就 2 0 0 0年NAEP的 8年级评估、TIMSS -R的 8年级评估和PISA三项评价项目的数学和科学领域评估做一比较 ,以便我们了解这些评估的实施背景、基本框架和评估内容     

Discrepancies Between Score Trends from NAEP and State Tests: A Scale-Invariant Perspective

State test score trends are widely interpreted as indicators of educational improvement. To validate these interpretations, state test score trends are often compared to trends on other tests such as the National Assessment of Educational Progress (NAEP). These comparisons raise serious technical and substantive concerns. Technically, the most commonly used trend statistics—for example, the change in the percent of proficient students—are misleading in the context of cross-test comparisons. Substantively, it may not be reasonable to expect that NAEP and state test score trends should be similar. This paper motivates then applies a "scale-invariant" framework for cross-test trend comparisons to compare "high-stakes" state test score trends from 2003 to 2005 to NAEP trends over the same period. Results show that state trends are significantly more positive than NAEP trends. The paper concludes with cautions against the positioning of trend discrepancies in a framework where only one trend is considered "true."     

国家教育进展评估的效度研究

作为一项得到广泛认可的教育绩效指标,国家教育进展评估(NAEP)是美国数十年来用于跟踪和了解教育进展的重要工具,也是全美初等教育与中等教育状况的晴雨表。它是美国当前唯一一项定期对小学、初中和高中学生的教育成就进行的全国性调查,在新测试技术的发展过程中发挥着重要的作用。本文对NAEP的发展历史进行综述和总结,同时对当前NAEP所面临的效度问题,如跨年级(纵向)量表以及在分数报告中使用表现水平的做法等进行评论。     

Predicting students’ writing performance on the NAEP from student- and state-level variables

This study examines the relationship between students’ demographic background and their experiences with writing at school, the alignment between state and National Assessment of Educational Progress (NAEP) direct writing assessments, and students’ NAEP writing performance. The study utilizes primary data collection via content analysis of writing assessment prompts and rubrics and secondary analysis with NAEP data through hierarchical linear modeling. Results indicate students from states with writing tests more similar to the NAEP do not perform significantly better than students from states with writing tests less similar to the NAEP. Rather, student demographic characteristics, including gender, ethnicity, SES, disability status, and English learner status significantly predict NAEP writing performance, as do factors related to frequency of writing across subject areas, frequency of writing for varied purposes, frequency of writing process use, and computer use in writing. The implications of the findings for writing instruction are discussed.     

Academic growth trajectories of ELLs in NAEP data: The case of fourth- and eighth-grade ELLs and non-ELLs on mathematics and reading tests

Utilizing the National Assessment of Educational Progress (NAEP) data, this study examined (1) how fourth and eighth-grade ELLs' mathematics and reading scores on national tests compared to their non-ELL peers' scores over the testing period between 2003 and 2011, and (2) if gender and ethnicity contributed to variation in the growth patterns among the student groups across grade levels and content areas. Since the NAEP data, which provides a national sample of 10,000–20,000 students, is collected using a probability sample design, sampling weights are adjusted so inferences can be appropriately made. Sample sizes within NAEP are large enough to generate adequate power for statistical significance. Thus, to display the data in a multivariate mode, Tableau 8.0.0 software was used. Results suggested that the achievement gap between non-ELLs and ELLs is either steady or slightly widening in both mathematics and reading, with multiple paths across the content areas, grade levels, and gender and ethnic groups.     

美国国家教育进展评价NAEP及其借鉴意义

美国国家教育进展评价,即"国家教育报告卡",是美国一个全国性的教育进展评价项目,旨在测量全美中小学生在阅读、写作、数学、科学、社会等学科领域的学术表现及发展趋势。NAEP是美国目前唯一定期在各个学科领域持续测评学生学业的全国性评价项目,它提供了大量有关美国中小学生学业表现的基础数据。NAEP在编制、抽样、组织、报告等方面的一些做法和经验,对于我们研制中小学各科学业评价标准及开展基础教育质量监测,具有借鉴意义。     

Contention and Consensus: The Development of the 1992 National Assessment of Educational Progress in Reading

Because of plans for state-by-state reporting of 1992 reading data from the National Assessment of Educational Progress (NAEP), we investigated the adequacy of the process used to develop the assessment, the degree to which it represents a consensus among professionals in the reading field, and its content and curricular validity. To carry out this investigation, we analyzed documents produced by NAEP, convened a 2-day panel of experts, held two public colloquia, conducted 50 interviews, and analyzed responses to a questionnaire completed by 627 leading educators. We found that the planning process did not include enough time to address some major concerns of the field. Despite this, there was widespread agreement that the 1992 NAEP in Reading represents important advances in reading assessment, including more open-ended responses, more authentic texts, and student choice about passages. But these very advances raise problems for test design and the interpretation and scoring of student responses.     

Effects of Motivational Interventions on the National Assessment of Educational Progress Mathematics Performance

One of the reasons often cited h r the low average level of proficiency demonstrated by U.S. students on national and international assessments is that there are no consequences or stakes attached to performance on the tests and, therefore, students are not motivated to invest their best effort. In this study, money was chosen as an incentive, but we hoped that short written instructions would be almost as powerful as money and easier and more desirable to implement in the National Assessment of Educational Progress (NAEP). Our results indicate that, at least for Grade 8 participants, student effort can be increased by financial rewards offered at the time of test taking, and that such effort can result in an increase in NAEP math test scores. Thus, from a policy perspective, scores from low-stakes tests may not represent what the student knows. Rather, such scores represent what students will demonstrate with minimal effort     

Reporting Minority Students' Test Scores: How Well Can the National Assessment of Educational Progress Account for Differences in Social Context?

This article investigates the adequacy of the National Assessment of Educational Progress (NAEP) for taking into account dissimilarities in students' family, school, and community contexts when reporting test score differences among population groups (i.e., racial and ethnic minorities). This question was addressed by comparing the NAEP to other representative data for Grades 8 and 12--the National Education Longitudinal Study (NELS) and High School and Beyond (HSB)--that contain richer social context measures. Our analyses show that NAEP lacks a number of important social context measures and that the quality of some (but by no means all) of NAEP's measures is low because of reliance on student self-reports and other unreliable data sources. These weaknesses of NAEP have important practical implications: Compared to HSB and NELS, NAEP usually overestimates the achievement differences between students who come from different population groups but similar social contexts. However, at the secondary school level at which these analyses were conducted, these overestimates primarily reflect NAEP's lack of important measures rather than its reliance on student self-reports.     

Benefits of Opportunity to Read and Balanced Instruction on the NAEP

Abstract

The National Assessment of Educational Progress (NAEP) requires reading comprehension processes that may be increased by students' amount of engaged reading, parental education, and gender, along with balanced reading instruction and opportunity to read. To examine the effects of those variables on reading achievement and engagement, the authors analyzed the 1994 Grade 4 Maryland NAEP with hierarchical linear modeling to construct both between-school and between-teacher models. Amount of engaged reading significantly predicted reading achievement on the NAEP, after parental education was statistically controlled. Balanced reading instruction significantly predicted reading achievement after accounting for students' engaged reading and parental education. Findings confirmed expectations from the proposed theoretical perspective on reading engagement. Policy implications included an emphasis on some instructional variables in the reading engagement model.