Standard Errors of Proportions Used in Reporting Changes in School Performance With Achievement Levels

The primary purpose of this study was to estimate the amount of variability in the proportions of students in a school district, scoring within each of three achievement levels that could be attributed to factors other than random sampling error. The approach taken is based on a general conceptual framework that collectively incorporates five sources of variability: instructional intervention, random sampling error, measurement error, equating error, and systematic error. Statewide school-level assessment data for reading and mathematics in grades four and eight from four consecutive years were used to examine annual grade-group change. The intent was to assess the impact of random sampling error in grade-group change estimates when either single-year proportions or 2-year average proportions are used to report school improvement with achievement levels. Observed variability in change was compared with theoretically-derived estimates of change due to random sampling error to determine the relative influence of sampling error and the aggregate of the other four sources of variability. Results indicate that the error variance of estimates of change at the school level is large enough to interfere with interpretations of annual change estimates. Recommendations are offered for establishing annual improvement goals and for reporting results with achievement levels-all in the context of adequate yearly progress (AYP)-while taking error estimates into account.