Modeling Group Differences in OLS and Orthogonal Regression: Implications for Differential Validity Studies

In evaluating the relationship between two measures across different groups (i.e., in evaluating “differential validity”) it is necessary to examine differences in correlation coefficients and in regression lines. Ordinary least squares (OLS) regression is the standard method for fitting lines to data, but its criterion for optimal fit (minimizing the squared vertical distances between the points and the line) is less natural in many contexts than the criterion used in orthogonal regression (minimizing the squared Euclidean distances of points from the line). OLS regression is appropriate if the goal is to predict some unknown dependent variable from a known independent variable, but in examining the relationship between two variables, which both contain error, OLS regression introduces bias. This bias, associated with regression toward the mean, can suggest that the test scores have different relationships, and therefore different meanings, in two groups, when the two sets of test scores have the same relationship and the same meanings in the two groups. The impact of regression toward the mean in differential validity studies is illustrated with two synthetic and two real data sets. Each of the two real data sets include two measures of competence in applying legal principles to fact situations (an essay test and a multiple-choice test) for candidates in two groups (Black/White in the first example and women/men in the second example).