Improving Person-Fit Assessment by Correcting the Ability Estimate and Its Reference Distribution

The standardized log-likelihood of a response vector (l_z) is a popular IRT-based person-fit test statistic for identifying model-misfitting response patterns. Traditional use of l_z is overly conservative in detecting aberrance due to its incorrect assumption regarding its theoretical null distribution. This study proposes a method for improving the accuracy of person-fit analysis using l_z which takes into account test unreliability when estimating the ability and constructs the distribution for each l_z through resampling methods. The Type I error and power (or detection rate) of the proposed method were examined at different test lengths, ability levels, and nominal α levels along with other methods, and power to detect three types of aberrance—cheating, lack of motivation, and speeding—was considered. Results indicate that the proposed method is a viable and promising approach. It has Type I error rates close to the nominal value for most ability levels and reasonably good power.