Response to Clemans

Clemans (1993) argued that the use of item response theory (IRT) to vertically scale Form E of the California Achievement Tests produces inappropriate results. In this response we show that (a) Cleman's analysis of school district data was incomplete, inconsistent, and did not follow good measurement practice; (b) the simulation he conducted was unfairly stacked against IRT, was unrealistic, and ignored other realistic published simulations that demonstrated the accuracy of IRT scaling procedures; (c) his "common sense" evaluations of student performance ignored basic facts about the measurement of student achievement; and (d) the concerns expressed in his article were irrelevant to the vast majority of uses of norm-referenced tests.     

Rejoinder

In a recent paper, Bifulco and Bretschneider [Econ. Edu. Rev. 20 (2001) 417] analyzed the performance of data envelopment analysis (DEA) and corrected ordinary least squares (COLS) using simulated data. The results were discouraging as both methods performed poorly even in cases when measurement error was assumed away. The authors concluded that current methods of evaluating school district performance are inadequate for performance-based school reform. As will be shown in this comment, the simulation performed was faulty due to inconsistency of the data generating process and the methods employed. After correcting for this inconsistency and re-estimating efficiency, the results are more consistent with the literature and more favorable to both approaches.