Sample Exclusion and Student Attrition Effects in the Longitudinal Study of Middle School Mathematics Performance

Achievement data from a longitudinally matched student cohort from a large school district in the southwestern United States were analyzed to investigate sample exclusion and student attrition effects on estimates of student, school, and district mathematics performance. Use of 2- and 3-level longitudinal growth models to estimate the growth trajectories of middle school students revealed that mathematics performance differed across 2 sample conditions. Relative to the achievement outcomes associated with a sample that included all students from the longitudinal cohort, district and school achievement were generally higher and student group performance more similar in the smaller, more advantaged student sample used for district accountability reporting. Further investigation of the school performance estimates showed that cross-sample changes in student achievement outcomes were closely related to the proportion of students from special student populations who were excluded from the district accountability sample. The achievement differences and the differential patterns of association demonstrated in this study suggest that conclusions drawn about district and school performance and relationships between student characteristics and student achievement outcomes may depend to some degree on which students are included in an analytic sample. Investigators seeking to take advantage of longitudinal designs in school effectiveness research are cautioned to closely examine their data for nonrandom student attrition and document the impact of sample exclusion and student attrition effects in the research and accountability reports that are produced from longitudinal data sets.     

A comparison between value-added school estimates and currently used metrics of school accountability in California

This study compared a value-added approach to school accountability to the currently used metrics of accountability in California of Adequate Yearly Progress (AYP) and Academic Performance Index (API). Five-year student panel data (N?=?53,733) from 29 elementary schools in a large California school district were used to address the research questions. Results show the strong relationship between AYP and API to student background measures. Schools with a majority of students from low socioeconomic background lagged far below schools from more affluent context. Results from the value-added approach however, showed a strongly diminished relationship to student background. Under this model, several schools from a low socioeconomic context can be seen as high achieving. Additionally, little evidence was found that high levels of student achievement negatively affect school value-added scores. Schools that enroll large proportions of advanced students, which often do not show positive growth across years are not penalized under a value-added approach. Recommendations for policy and future research are discussed.     

The Sensitivity of Value-Added Teacher Effect Estimates to Different Mathematics Achievement Measures

Using longitudinal data from a cohort of middle school students from a large school district, we estimate separate "value-added" teacher effects for two subscales of a mathematics assessment under a variety of statistical models varying in form and degree of control for student background characteristics. We find that the variation in estimated effects resulting from the different mathematics achievement measures is large relative to variation resulting from choices about model specification, and that the variation within teachers across achievement measures is larger than the variation across teachers. These results suggest that conclusions about individual teachers' performance based on value-added models can be sensitive to the ways in which student achievement is measured.     

Schools, school quality and achievement growth: Evidence from the Philippines

A broad literature seeks to assess the importance of schools, proxies for school quality, and family background on children's achievement growth using the education production function. Using rich data from the Philippines, we introduce and estimate a model that imposes little structure on the relationship between intake achievement and follow-up achievement and evaluate school performance based on this estimated relationship. Our methods nest typical value-added specifications that use test score gains as the outcome variable and models assuming linearity in the relationship between intake and follow-up scores. We find evidence against the use of value-added models for our data and show that such models give very different assessments of school performance in the Philippines. Using a variety of tests we find that schools matter in the production of student achievement, though variation in performance across schools only explains about 4.4–5.3% of the total (conditional) variation in follow-up achievement. Schools providing basic facilities—in particular schools providing electricity—are found to perform much better in the production of achievement growth.     

Use of the probit model to estimate school performance in student attainment of achievement testing standards

In the USA, trends in educational accountability have driven several models attempting to provide quality data for decision making at the national, state, and local levels, regarding the success of schools in meeting standards for competence. Statistical methods to generate data for such decisions have generally included (a) status models that examine simple indications of number of students meeting a criterion level of achievement, (b) growth models that explore change over the course of one or more years, and (c) value-added models that attempt to control for factors deemed relevant to student achievement patterns. This study examined a new strategy for student and school achievement modeling that augments the field through the use of the probit model to estimate the likelihood of students meeting an established level standard and estimating the proportion of individuals within a school meeting the standard. Results of the study showed that the probit model was an effective tool both for providing such adjustments, as well as for adjusting them based upon salient demographic variables. Implications of these results and suggestions for further use of the model are discussed.     

A longitudinal study of math achievement gains for initially low achieving students

Standardized testing has been implemented in most school districts as part of an effort to improve student achievement in mathematics, reading, science, and English. There have been heated debates as to the effects of these improvement efforts on student achievement. In studying these issues, it is important to examine longitudinal growth patterns for individuals. In most of the studies, however, there is a lack of empirical data at the individual student level or the studies are cross-sectional in nature. The current study attempts to examine growth patterns of student math achievement between 1997 and 2000 and individual differences in growth patterns. MDS exploratory growth modeling was used in the investigation based on data from 716 students in a single school district. Individual differences in growth rates were found. Disadvantaged (limited English proficiency and special education) students had lower initial achievement levels and did not seem to be catching up to other students because their average growth rates were similar to those of other students. These results are discussed in light of recent school reform efforts and the goal of closing achievement gaps.     

Test Score or Student Progress? A Value-Added Evaluation of School Effectiveness in Urban China

Outcome-oriented evaluation of school effectiveness is often based on student test scores in certain critical examinations. This study provides another method of evaluation—value-added—which is based on student achievement progress. This paper introduces the method of estimating the value-added score of schools in multi-level models. Based on longitudinal student achievement data, two measures of school effectiveness in one local education authority in China are compared. It is found that the between-school difference in both test-score and value-added is large comparable with that of Western countries. The results of the two measures of school effectiveness are highly different. The value-added measures lack consistency across different subject areas within schools while the test score measures are highly correlated between subjects. Teachers show their preference for value-added measures over test-score measures of education quality. It is suggested that value-added measures of school effectiveness should be used as a complement to rather than a substitute for test-score measures. The shortcomings of value-added approach are also discussed.     

STEM‐focused high schools as a strategy for enhancing readiness for postsecondary STEM programs

The logic underlying inclusive STEM high schools (ISHSs) posits that requiring all students to take advanced college preparatory STEM courses while providing student‐centered, reform‐oriented instruction, ample student supports, and real‐world STEM experiences and role models will prepare and inspire students admitted on the basis of STEM interest rather than prior achievement for postsecondary STEM. This study tests that logic model by comparing the high school experiences and achievement of students in ISHSs and comparison schools in North Carolina. After identifying ISHS and non‐STEM comparison high schools serving students who were similar in terms of socioeconomic status and academic achievement prior to high school entry, we employed propensity‐score weighting and HLM modeling to estimate the impact of attending an ISHS on a set of outcome measures obtained from student surveys and from the state's longitudinal student data system. Analyses of student survey data found that attending an ISHS raises the likelihood that a student will complete pre‐calculus or calculus and chemistry in high school, leads to increased involvement in STEM extracurricular and out‐of‐class activities, and enhances interest in science careers and aspirations to earn a master's or higher degree. Analyses of student outcome data from state administrative records revealed a positive impact of inclusive STEM high school attendance on grade point average (GPA) but not on ACT scores. © 2016 The Authors. Journal of Research in Science Teaching Published by Wiley Periodicals, Inc. J Res Sci Teach 53: 709–736, 2016

     

Conceptual and empirical differences among various value-added models for accountability

Accountability systems in education generally include indicators of student performance. However, these indicators often differ considerably among the various systems. More and more countries try to include value-added measures, mainly because they do not want to hold schools accountable for differences in their initial intake of students. This study presents a conceptual framework of these value-added measures, resulting in an overview of 5 different types. Using data from Dutch secondary schools, we empirically provide estimates of these different measures. Our analyses show that the correlation between the different types of school effects estimated is rather high, but that the different models implicate different results for individual schools. Based on theoretical considerations, arguments are given to use the following indicators in the value-added accountability models: prior achievement, student-level background characteristics, and compositional characteristics of the student population.     

Understanding achievement differences between schools in Ireland – can existing data-sets help?

Recent years have seen an increased focus on school accountability in Ireland and calls for greater use to be made of student achievement data for monitoring student outcomes. In this paper, it is argued that existing data-sets in Ireland offer limited potential for the value-added modelling approaches used for accountability purposes in many other countries. Some reasons for this include: few longitudinal studies of student achievement, few population data-sets and limited data on student home background in the State examination data-sets. A cross-sectional contextualised attainment model (CAM) is presented using Junior Certificate data from 2009. While the model offers some potential for identifying schools performing significantly above or below average, its explanatory power is limited by the availability of appropriate student background data. It is proposed that the use of statistical models for the purposes of identifying high or low performing schools in Ireland would need to take account of findings from other countries regarding technical difficulties such as variable selection, missing data, and the need to avoid aggregated data. Other practical concerns discussed include the appropriateness of publishing league tables and the usefulness of such information for informing parental choice.     

The Dependence of Growth‐Model Results on Proficiency Cut Scores

States participating in the Growth Model Pilot Program reference individual student growth against “proficiency” cut scores that conform with the original No Child Left Behind Act (NCLB). Although achievement results from conventional NCLB models are also cut‐score dependent, the functional relationships between cut‐score location and growth results are more complex and are not currently well described. We apply cut‐score scenarios to longitudinal data to demonstrate the dependence of state‐ and school‐level growth results on cut‐score choice. This dependence is examined along three dimensions: 1) rigor, as states set cut scores largely at their discretion, 2) across‐grade articulation, as the rigor of proficiency standards may vary across grades, and 3) the time horizon chosen for growth to proficiency. Results show that the selection of plausible alternative cut scores within a growth model can change the percentage of students “on track to proficiency” by more than 20 percentage points and reverse accountability decisions for more than 40% of schools. We contribute a framework for predicting these dependencies, and we argue that the cut‐score dependence of large‐scale growth statistics must be made transparent, particularly for comparisons of growth results across states.     

Explaining the substantial inter-domain and over-time correlations in student achievement: the importance of stable student attributes

Multi-domain and longitudinal studies of student achievement routinely find moderate to strong correlations across achievement domains and even stronger within-domain correlations over time. The purpose of this study is to examine the sources of these patterns analysing student achievement in 5 domains across Years 3, 5 and 7. The analysis is of longitudinal population data of over 22,000 students and utilizes fixed-effects models to incorporate stable general and domain-specific latent factors. These latent factors correspond to a general cognitive-ability-like factor and specific aptitudes in particular, or types of, subject areas. The preferred model incorporates both general and domain-specific latent factors with stronger effects for the general factor, although the domain-specific factors are particularly strong for spelling and numeracy. When taking into account general and domain-specific latent factors, the effects of student’s socioeconomic status (SES) and school SES are trivial.     

Value-added models and the measurement of teacher productivity

Research on teacher productivity, as well as recently developed accountability systems for teachers, relies on “value-added” models to estimate the impact of teachers on student performance. We consider six value-added models that encompass most commonly estimated specifications. We test many of the central assumptions required to derive each of the value-added models from an underlying structural cumulative achievement model and reject nearly all of them. While some of the six popular models produce similar estimates, other specifications yield estimates of teacher productivity and other key parameters that are considerably different.     

Math Achievement Trajectories Among Black Male Students in the Elementary- and Middle-School Years

In this article, we analyze the variation in math achievement trajectories of Black male students to understand the different ways these students successfully or unsuccessfully navigate schools and the school characteristics that are associated with their trajectories. Using longitudinal student-level data from a large urban US city (n = 7,039), we analyze Black male students from one cohort to identify trajectories. We find a lack of growth in standardized math scores, suggesting that, on average, math proficiency among Black male students in our sample is declining over time. We found that the 4th-grade standardized math scores of subsidized-lunch students were somewhat lower than those of nonsubsidized students and those of retained students were substantially lower than their counterparts. The average math score of a Black male student's cohort appears to be the only variable amenable to policy manipulation that has a sizeable association with the growth of their standardized math scores, suggesting that putting Black male students in more challenging learning environments may be the best way to increase math proficiency over time. By themselves, other policy decisions (reducing student mobility, teacher turnover, or special education classification; increasing attendance or spending on after-school programming; or hiring more qualified or experienced teachers) all appear to have no or negligible associations with growth in math scores.     

Evaluating the Predictive Value of Growth Prediction Models

This study evaluates four growth prediction models—projection, student growth percentile, trajectory, and transition table—commonly used to forecast (and give schools credit for) middle school students' future proficiency. Analyses focused on vertically scaled summative mathematics assessments, and two performance standards conditions (high rigor and low rigor) were examined. Results suggest that, when “status plus growth” is the accountability metric a state uses to reward or sanction schools, growth prediction models offer value above and beyond status‐only accountability systems in most, but not all, circumstances. Predictive growth models offer little value beyond status‐only systems if the future target proficiency cut score is rigorous. Conversely, certain models (e.g., projection) provide substantial additional value when the future target cut score is relatively low. In general, growth prediction models' predictive value is limited by a lack of power to detect students who are truly on‐track. Limitations and policy implications are discussed, including the utility of growth projection models in assessment and accountability systems organized around ambitious college‐readiness goals.     

教育增值评价先行者——美国田纳西州教育增值评价模式解析

美国田纳西州增值评价模式(TVAAS)通过对学生多年的学业成就数据进行纵向分析,测定学校系统、学校和教师促进学生学业进步的有效性。其研究发现,种族、社会经济因素、班级规模以及班级成员的异质性均不是学生学业增长较好的预测指标,教师效能才是学生学业进步的主要因素;教师的影响是附加的和累积的,且没有补偿效应。TVAAS是对教育评价方法的有益补充。     

School effects on Chilean children’s achievement growth in language and mathematics: An accelerated growth curve model

This study investigates school effects on primary school students’ language and mathematics achievement trajectories in Chile, a context of particular interest given its large between-school variability in educational outcomes. The sample features an accelerated longitudinal design (3 time points, 4 cohorts) together spanning Grades 3 to 8 (n = 19,704 students in 156 schools). The magnitudes of school effects on students’ growth trajectories were found to be sizeable (generally larger than school effects in Western industrialised countries) and moderately consistent across school subjects. School composition effects on student achievement status were found for both school subjects. However, there was no evidence of composition effects on student achievement growth. The study provides new evidence on the size and nature of school effects in a developing country context based on state-of-the-art methods (i.e., accelerated longitudinal and growth curve models).     

Holding Schools Accountable for the Growth of Nonproficient Students: Coordinating Measurement and Accountability

A key intent of the NCLB growth pilot is to reward low‐status schools who are closing the gap to proficiency. In this article, we demonstrate that the capability of proposed models to identify those schools depends on how the growth model is incorporated into accountability decisions. Six pilot‐approved growth models were applied to vertically scaled mathematics assessment data from a single state collected over 2 years. Student and school classifications were compared across models. Accountability classifications using status and growth to proficiency as defined by each model were considered from two perspectives. The first involved adding the number of students moving toward proficiency to the count of proficient students, while the second involved a multitier accountability system where each school was first held accountable for status and then held accountable for the growth of their nonproficient students. Our findings emphasize the importance of evaluating status and growth independently when attempting to identify low‐status schools with insufficient growth among nonproficient students.