TOWARD AN INTEGRATION OF THEORY AND METHOD FOR CRITERION-REFERENCED TESTS1,2

In this paper, an attempt has been made to synthesize some of the current thinking in the area of criterion-referenced testing as well as to provide the beginning of an integration of theory and method for such testing. Since criterion-referenced testing is viewed from a decision-theoretic point of view, approaches to reliability and validity estimation consistent with this philosophy are suggested. Also, to improve the decision-making accuracy of criterion-referenced tests, a Bayesian procedure for estimating true mastery scores has been proposed. This Bayesian procedure uses information about other members of a student's group (collateral information), but the resulting estimation is still criterion referenced rather than norm referenced in that the student is compared to a standard rather than to other students. In theory, the Bayesian procedure increases the “effective length” of the test by improving the reliability, the validity, and more importantly, the decision-making accuracy of the criterion-referenced test scores.     

IMPLICATIONS OF CRITERION-REFERENCED MEASUREMENT1,2

During the past several years measurement and instructional specialists have distinguished between norm-referenced and criterion-referenced approaches to measurement. More traditional, a norm-referenced measure is used to identify an individual's performance in relation to the performance of others on the same measure. A criterion-referenced test is used to identify an individual's status with respect to an established standard of performance. This discussion examines the implications of these two approaches to measurement, particularly criterion-referenced measurement, with respect to variability, item construction, reliability, validity, item analysis, reporting, and interpretation.     

THE ISSUE OF ITEM AND TEST VARIANCE FOR CRITERION-REFERENCED TESTS1

It has been argued that item variance and test variance are not necessary characteristics for criterion-referenced tests, although they are necessary for normreferenced tests. This position is in error because it considers sample statistics as the criteria for evaluating items and tests. Within a particular sample, an item or test may have no variance, but in the population of observations for which the test was designed, calibrated, and evaluated, both items and tests must have variance.     

EFFECTS OF DIFFERENT SAMPLES ON ITEM AND TEST CHARACTERISTICS OF CRITERION-REFERENCED TESTS

Although many have rejected classical test construction and analysis procedures for criterion-referenced tests, the present study was concerned with the possibility that classical procedures are both applicable and appropriate when samples of both mastery and nonmastery examinees are employed. A rationale for using these samples was presented, and empirical evidence was gathered which supported the practice of combining samples to increase the variance of test scores and thereby permit the proper estimate of reliability and item validities.     

THE ISSUE OF ITEM AND TEST VARIANCE FOR CRITERION-REFERENCED TESTS: A CLARIFICATION

This note contends that item or score variability is an unnecessary characteristic of criterion-referenced tests as they have been traditionally conceived, namely, as measures of well defined classes of examinee behaviors.     

THE ISSUE OF ITEM AND TEST VARIANCE FOR CRITERION-REFERENCED TESTS: A REPLY

It is a necessary condition that items and tests have variance and discrimination in the range of interest (population of observations) for which they are calibrated and selected. The basis for selection of the calibration sample determines the kind of scale which will be developed, A random sample from a population of individuals leads to a norm-referenced scale, and a sample representative of abilities of a range of a characteristic leads to a criterion-referenced scale.     

香农采样理论的扩展及应用

本文基于香农采样理论,通过对离散序列重构信号的研究,给出一种在完备正交积分变换下的广义采样定理及其应用。     

RELIABILITY OF CRITERION-REFERENCED TESTS: A DECISION-THEORETIC FORMULATION

It has been suggested that the primary purpose for criterion-referenced testing in objective-based instructional programs is to classify examinees into mastery states or categories on the objectives included in the test. We have proposed that the reliability of the criterion-referenced test scores be defined in terms of the consistency of the decision-making process across repeated administrations of the test. Specifically, reliability is defined as a measure of agreement over and above that which can be expected by chance between the decisions made about examinee mastery states in repeated test administrations for each objective measured by the criterion-referenced test.     

THE ROLE OF RELIABILITY IN CRITERION-REFERENCED TESTS

In discussion of the properties of criterion-referenced tests, it is often assumed that traditional reliability indices, particularly those based on internal consistency, are not relevant. However, if the measurement errors involved in using an individual's observed score on a criterion-referenced test to estimate his or her universe scores on a domain of items are compared to errors of an a priori procedure that assigns the same universe score (the mean observed test score) to all persons, the test-based procedure is found to improve the accuracy of universe score estimates only if the test reliability is above 0.5. This suggests that criterion-referenced tests with low reliabilities generally will have limited use in estimating universe scores on domains of items.     

AN INTERPRETATION OF LIVINGSTON'S RELIABILITY COEFFICIENT FOR CRITERION-REFERENCED TESTS

An alternative interpretation of Livingston's reliability coefficient is based on the notion of the relation of the size of the reliability coefficient to the range of talent. It is shown that the (generally) larger Livingston coefficient does not imply a smaller standard error of measurement and consequently does not imply a more dependable determination of whether or not a true score falls below (or exceeds) a given criterion value.