Teachers' Ability to Estimate Item Difficulty: A Test of the Assumptions in the Angoff Standard Setting Method

The Angoff (1971) standard setting method requires expert panelists to (a) conceptualize candidates who possess the qualifications of interest (e.g., the minimally qualified) and (b) estimate actual item performance for these candidates. Past and current research (Bejar, 1983; Shepard, 1994) suggests that estimating item performance is difficult for panelists. If panelists cannot perform this task, the validity of the standard based on these estimates is in question. This study tested the ability of 26 classroom teachers to estimate item performance for two groups of their students on a locally developed district-wide science test. Teachers were more accurate in estimating the performance of the total group than of the "borderline group," but in neither case was their accuracy level high. Implications of this finding for the validity of item performance estimates by panelists using the Angoff standard setting method are discussed.     

Diagnostic Profiles: A Standard Setting Method for Use With a Cognitive Diagnostic Model

This article introduces the Diagnostic Profiles (DP) standard setting method for setting a performance standard on a test developed from a cognitive diagnostic model (CDM), the outcome of which is a profile of mastered and not‐mastered skills or attributes rather than a single test score. In the DP method, the key judgment task for panelists is a decision on whether or not individual cognitive skill profiles meet the performance standard. A randomized experiment was carried out in which secondary mathematics teachers were randomly assigned to either the DP method or the modified Angoff method. The standard setting methods were applied to a test of student readiness to enter high school algebra (Algebra I). While the DP profile judgments were perceived to be more difficult than the Angoff item judgments, there was a high degree of agreement among the panelists for most of the profiles. In order to compare the methods, cut scores were generated from the DP method. The results of the DP group were comparable to the Angoff group, with less cut score variability in the DP group. The DP method shows promise for testing situations in which diagnostic information is needed about examinees and where that information needs to be linked to a performance standard.     

A Comparison of Angoff and Bookmark Standard Setting Methods

This article presents a comparison of simplified variations on two prevalent methods, Angoff and Bookmark, for setting cut scores on educational assessments. The comparison is presented through an application with a Grade 7 Mathematics Assessment in a midwestem school district. Training and operational methods and procedures for each method are described in detail along with comparative results for the application. An alternative item ordering strategy for the Bookmark method that may increase its usability is also introduced. Although the Angoff method is more widely used, the Bookmark method has some promising features, specifically in educational settings. Teachers are able to focus on the expected performance of the "barely proficient" student without the additional challenge of estimating absolute item dificulty.     

Consistency of Angoff-Based Predictions of Item Performance: Evidence of Technical Quality of Results From the Angoff Standard Setting Method

Judgmental standard-setting methods, such as the Angoff(1971) method, use item performance estimates as the basis for determining the minimum passing score (MPS). Therefore, the accuracy, of these item peformance estimates is crucial to the validity of the resulting MPS. Recent researchers (Shepard, 1995; Impara & Plake, 1998; National Research Council. 1999) have called into question the ability of judges to make accurate item performance estimates for target subgroups of candidates, such as minimally competent candidates. The propose of this study was to examine the intra- and inter-rater consistency of item performance estimates from an Angoff standard setting. Results provide evidence that item pelformance estimates were consistent within and across panels within and across years. Factors that might have influenced this high degree of reliability, in the item performance estimates in a standard setting study are discussed.     

When Dropping Out is Not a Permanent High School Outcome: Student Characteristics,Motivations, and Reenrollment Challenges

The customary perception is that students who drop out vanish from school enrollment rosters for good. This is an incomplete picture of the complex dropout story; dropping out is not necessarily a permanent high school outcome. Following the enrollment and course history of a district cohort of first-time 9th graders, this article documents the cohort's dropout, reenrollment, and graduation outcomes. One-third of dropouts reenrolled in district high schools, and older students and students behind in course credits were less likely to reenroll than other dropouts. Interviews with district administrators, high school principals, and reenrollees examined student motivations for reenrolling and challenges districts face when dropouts reenroll. Dropouts return primarily because of limited employment opportunities and efforts of school leaders to facilitate their return. Yet reenrollment created district challenges related to funding, accountability, and getting students who had dropped out on track to graduate.     

Schoolwide Impact and AVID: How Have Selected Texas High Schools Addressed the New Accountability Measures?

This is a study of high schools and their districts receiving Comprehensive School Reform (CSR) grants that have used those grants to support implementation of the Advancement Via Individual Determination (AVID) model. Over a 4-year period 10 such high schools in 5 districts have been examined to determine if schoolwide or district-wide accountability measures have improved over the period of study, compared to non-AVID high schools and districts. Selected sets of data are presented as the focus of this study, and include graduation or completion rates, advanced course enrollment, Advanced Placement (AP) results, and number of students graduating on advanced graduation plans. Researchers found that AVID had affected the performance profile of the school by leveraging success of disaggregated subgroups of students, particularly African American and Latino students, as well as students from lower income families.     

The administrative role in teacher competency

As managers of teachers, principals must cope with a variety of educational mistakes in their schools. These cumulative mistakes lead to a state of borderline competency in the teacher. From interviews with 30 elementary school principals in a large urban school system, interviews and observations of 19 borderline competent teachers, and review of principals' teacher evaluation records, five states of coping with teachers' educational mistakes emerged: (1) deployment-enlisting the teachers' colleagues to watch over him or her and report back to the principal on the teacher's behavior; (2) detente-bringing the troubled teacher within the society of peers and rallying forces to help solve his or her problems; (3) determination-deciding that the range of the teacher's deviations exceeds the boundaries of normative behavior and there is cause for dismissal; (4) evaluation-assigning an unsatisfactory efficiency rating with extensive documentation and recordkeeping for teachers who have been identified as borderline competent; and (5) formal dismissal-taking action to remove a teacher from the school.     

Public or private? The influence of immigration on native schooling choices in the United States

This study examines whether rising enrollments of foreign-born students in US public schools caused a movement among native children from public schools to private schools, something the literature has labeled native flight to private school. Using data from the National Center of Educational Statistics School District Demographic System, estimates of native flight are constructed using enrollment data on native and foreign-born, school-age children from 1990, 2000, and 2010. Concern regarding omitted variables bias necessitates the use of an instrumental variables technique. An instrument for the foreign-born enrollment is created using information about the ethnic composition of school districts in 1980 to predict the enrollment patterns of foreign-born students in later years. Two-stage least squares estimates confirm the presence of native flight. Flight to private school among white native students is occurring in smaller school districts in non-traditional immigrant receiving states, while flight among native minorities and Hispanics is located in school districts that reside in traditional immigrant receiving states.     

Differential Use of Item Information by Judges Using Angoff and Nedeisky Procedures

Competency examinations in a variety of domains require setting a minimum standard of performance. This study examines the issue of whether judges using the two most popular methods for setting cut scores (Angoff and Nedelsky methods) use different sources of information when making their judgments. Thirty-one judges were assigned randomly to the two methods to set cut scores for a high school graduation test in reading comprehension. These ratings were then related to characteristics of the items as well as to empirically obtained p values. Results indicate that judges using the Angoff method use a wider variety of information and yield estimates closer to the actual p values. The characteristics of items used in the study were effective predictors of judges' ratings, but were far less effective in predicting p values     

Setting Reliable National Curriculum Standards: a guide to the Angoff procedure

The Northern Ireland Curriculum, like the English National Curriculum, records pupil achievement on a 10‐level scale. The level to which a pupil is ‘assigned’ at the end of a Key Stage is based upon two sources of assessment information: classroom‐based measures provided by the teacher and summative information from Common Assessment Instruments (CAIs), which are pen‐and‐paper tests taken at the end of the Key Stage. CAIs play a central role in confirming the accuracy with which teachers judge the level at which a pupil is working. While the teacher might judge a pupil to have mastered level 7 in Algebra, for example, based upon observation in class, test data and homeworks, the CAI will only confirm this level if the pupil scores above the level 7 cutscore on the CAI. If this cutscore does not accord with a reliable measure of what constitutes level 7 performance in Algebra in the classroom, there is likely to be misclassification of pupils with attendant difficulties for the efficient planning of teaching and learning. Misclassifications can be minimised when examiners and teachers interpret level 7 achievement in Algebra similarly. The Angoff standard‐setting procedure was used to establish level 5 cutscores in the Number and Handling Data tests of the mathematics CAI so that comparisons might be made between the published level 5 cutscores and those which result from a judgemental standard‐setting procedure. The 21 teachers involved in the procedure were offered the opportunity to recommend a level 5 ‘standard’ using the Angoff methodology, and to review their recommendations in the light of test data from the February 1993 CAI administration. A further opportunity was offered following a discussion during which individual teachers articulated their reasons for the standards they recommended. The results confirm that the reliability of recommended standards increases both as a consequence of receiving normative data and of discussion. All statistical measures reported in this article indicate that the procedure could command the confidence of examiners, teachers and the public. While the recommended cutscore for Number is in close accord with that published by the examiners, the extent of the mismatch in the Handling Data test is such as might give rise to some misclassification of pupils. It is important to stress that this mismatch had no real consequences since 1993 was a pilot year and no test outcomes were reported. The article concludes with an outline of the contribution which the Angoff methodology can make to the resolution of some of the difficulties faced by English national assessment, as identified in Sir Ron Dealing's interim report “The National Curriculum and its Assessment”.     

College Readiness for English Language Learners (ELLs) in California: Assessing Equity for ELLs under the Local Control Funding Formula

English Learners (ELs) represent critical mass of K-12 students in public schools across the nation. However, states have failed to improve EL student academic outcomes, including college readiness. In 2013, California altered its school funding policy, providing additional support for English learners. Drawing on Vertical Equity and a Critical Community Strength Framework, this study examines the impact of local funding on college readiness among EL students by analyzing accountability plans and secondary data of 13 districts and assessing reforms using markers of college readiness. Document analysis shows: 1) school districts’ prioritization, 2) resource allocation based on those priorities, and 3) different approaches to operationalizing the construct of equity. Secondary data analysis reveals EL students lagged behind peers in performance on state assessments and enrollment in college preparatory and Advanced Placement varied across districts. These data confirm the critical need for intervention and targeted investment to raise overall college readiness rates among EL students in California and have national implications for states with growing Latino and EL student populations.     

A Standard-Setting Method Designed for Complex Performance Assessments: Categorical Assignments of Student Work

Results of variations on a categorical standard-setting procedure for use with performance assessments with multiple performance standards are reported. Panelists were asked to make independent ordinal categorical assignments of student work. Student work came from the administration of 1 of the 1996 Grade 8 National Assessment of Educational Progress Science Booklets. This study focused on the comparability of cutscores using 2 data analysis strategies, the effects of 2 strategies for making the categorical classifications (sorting vs. direct classification), the efficacy of long and short versions of the classification scale, and the effects of discussion on the basic, proficient, and advanced cutscores. The results indicate that there were minimal differences in cutscores across the 2 data analysis strategies. The direct classification approach was more feasible than the sorting approach, at least with the design as it was implemented in this study. The short version of the classification scale yielded comparable results to the longer version and took less time (about 20%) for the panelists to complete. Discussion produced the expected effect of more agreement among panelists. These results should be interpreted with caution due to the small sample sizes.     

The effect of charter schools on districts’ student composition,costs, and efficiency: The case of New York state

Charter schools can influence a school district's costs by reducing economies of scale and by changing the share of high cost students a district serves, but might also increase the district's efficiency through competition. Utilizing data for New York State school districts from 1998/99 to 2013/14, we estimate difference-in-differences models to assess the effect of charter schools on enrollment and student composition. Then, we estimate an expenditure function, using data prior to the charter school program, to measure the costs associated with reaching a given performance standard for students in various need categories and different enrollments. Next, using the entire data set, we run a second expenditure function to determine changes in efficiency associated with charter school entry. We find that charter schools increase the cost of providing education, and that these cost increases are larger than short-run efficiency gains, but are offset by efficiency gains in the long term.     

Do student migrations affect school performance? Evidence from Wisconsin's inter-district public school program

This paper examines the competitive effects of a unique school choice program implemented in the late 1990s, Wisconsin's open enrollment program, which allows families to send their children to schools outside their home district. In contrast to other school choice programs, districts not only face negative consequences from losing students and state funding, but they also stand to gain in the event of student emigration from other districts. The identification approach exploits differences in the number of schools in bordering districts, which affects inter-district ease-of-transfer. Estimates produce three main conclusions. First, districts that experience student out-migration produce higher standardized test scores in the subsequent year. Second, these effects are most evident among districts for which out-migration, expressed as a percentage of enrollment, falls in the upper quartile of all districts under consideration. Third, districts do not appear to respond to in-migration, indicating that districts place more emphasis on (and have more control over) preventing out-migration, as opposed to encouraging in-migration. These findings provide evidence that schools respond to competitive forces by improving quality.     

Going Charter? A Study of School District Competition in Wisconsin

The question that drives this article is why some school districts decide to open up charter schools and others do not. Several answers are plausible: (a) entrepreneurial initiative, (b) structural explanations, and (c) spatial competition. We use data for the state of Wisconsin derived from extensive case studies of 19 charter schools and quantitative data on Wisconsin school district from state files and the U.S. Department of Education common core databases. We find evidence to support all three explanations for why districts “go charter.” First, in almost every school and district we visited for case studies, at the heart of either the district or the charter school, and often both, there were entrepreneurial administrators, school board members, teachers, or parents. Our evidence was anecdotal but very consistent across 19 case studies. Second, there are two general sets of structural characteristics that were shown to be quantitatively correlated with becoming a charter district. The first set comprised resource characteristics (size, federal revenue, and available seats); the second set comprised indicators of unmet students needs (the percentage of students eligible for free lunch). Finally, we argue and believe we provide significant evidence that competition is also a motivation for going charter. We posit that open enrollment and charter schools are working together to enhance the flows of students from homeschooling, private schools, dropouts, and other public school districts into charter school districts. Thus using several different indicators and models, estimating both which districts become charter districts and the flow and net gain directly from open enrollment, there is no question that charter schools are increasing competition for students in Wisconsin.     

Student Decisions to Attend Public Two-Year Community Colleges

While the strategy of funding both systems provides an incentive for both school districts and community colleges to participate with dual enrollment, the current fiscal environment has drawn attention to the inefficient use of the dual funding structure. This article highlights the results of a case study on Florida's dual enrollment program documenting how over a 20-year period policymakers offered different financial incentives as an inducement to greater participation among districts and community colleges. The article documents how the current state fiscal condition has altered lawmaker's motivations for supporting dual enrollment. It concludes by describing the implications of this on community college participation with dual enrollment.     

School Start Time Change: An In‐Depth Examination of School Districts in the United States

In response to the scientific evidence documenting both profound developmental changes in sleep and circadian biology during adolescence and the myriad of negative health, performance, and safety outcomes risks associated with chronic sleep loss, at least 70 public school districts in the United States, representing approximately 1,000 schools, have successfully implemented a delay in high school start times. However, despite the compelling evidence supporting school start time change as a key strategy in addressing the epidemic of adolescent sleep loss, many school districts across the country with early high school start times have not considered the option to implement later bell schedules for adolescents. Moreover, while the current scientific literature has clearly documented the positive outcomes associated with delayed high school start times, these studies contain limited information regarding the process by which school districts consider, approve and implement bell schedule changes. Thus, this in‐depth examination of those school districts that have been successful in changing their bell schedules is intended to support the efforts of other districts in various stages of contemplating this measure. We utilized a multi‐pronged approach (literature review, case studies, telephone interviews, online survey) to summarize the experiences of school districts across the United States in regard to challenges faced, strategies employed, and lessons learned in the hope that this information will be a useful tool for other school districts looking to chart a course forward to promote the health, safety, and academic opportunities of their students.     

Multilevel analysis of student pathways to higher education

This study examined the relationship between academic preparation and post-secondary educational outcomes. To uncover the factors that influence secondary-level school course-taking patterns and outcomes, multi-level modelling was conducted using data from a nationally representative sample of US secondary-level school students from the Education Longitudinal Study of 2002. Results indicated (1) significant effects from student, family and school predictors on course-taking patterns and (2) significant effects from course-taking patterns on enrollment in post-secondary institution types with math course-taking pattern having the largest impact. Implications for educators and administrators are discussed.     

Strategic interaction among public school districts: Evidence on spatial interdependence in school inputs

This paper provides evidence on strategic interaction among public school districts in Massachusetts. The open enrollment program in Massachusetts is unique in testing the strategic competition thesis because it allows students to attend public schools located outside their home district. For each student who transfers out, funds are deducted from the sending district's local aid distribution and added to the receiving district's local aid. Thus, school districts no longer have the assurance of a ready supply of students and funding, instead they have to compete with neighboring districts, and enact strategies that will help retain as well as attract more students. A spatial econometric framework is applied to disentangle the correlation due to strategic interaction from the one that is simply due to spatially correlated error shocks. The results show that public school districts respond positively to the expenditure decisions of neighboring districts, thereby acting strategically when setting their own spending levels.     

College Course Grades for Dual Enrollment Students

This study compared college course grade outcomes, both during and after high school, of dual-enrollment students to those of traditional students. The study was based on a large, multiyear sample of Iowa high school and community college students. The results showed that while in high school, dual-enrollment students consistently outperformed traditional students in community college courses. However, much of the difference might be due to underlying differences in the two groups associated with the type of college the students chose to attend after high school (i.e., four-year vs. two-year). Dual enrollment students tended to perform about the same as traditional students in terms of post-high-school community college course grades. For students who enrolled in four-year institutions after high school, analyses of college course grade data suggested a small positive effect of dual enrollment on first-year college grade point average (GPA).