On the Reliability of Categorically Scored Examinations

A decision-theoretic approach to the question of reliability in categorically scored examinations is explored. The concepts of true scores and errors are discussed as they deviate from conventional psychometric definitions and measurement error in categorical scores is cast in terms ofmisclassifications. A reliability measure based on proportional reduction in loss (PRL) is then presented and exemplified with data from a large-scale assessment. The link between the PRL approach and the classical conception of reliability is discussed. Some design considerations for reliability studies are also discussed.