SAT Validity for Linguistic Minorities at the University of California, Santa Barbara

The validity of the SAT as an admissions criterion for Latinos and Asian Americans who are not native English speakers was examined. The analyses, based on 1997 and 1998 UCSB freshmen, focused on the effectiveness of SAT scores and high school grade-point average (HSGPA) in predicting college freshman grade-point average (FGPA). When regression equations were estimated based on all students combined, some systematic prediction errors occurred. For language minorities, using only high school grades as a predictor led to predicted FGPAs that tended to exceed actual FGPAs, particularly for Latinos. Including SAT scores in the equation notably reduced prediction bias. Further analyses showed that, while HSGPA had the highest correlation with FGPA for most groups, SAT verbal score was the strongest predictor of FGPA for language minorities in 1998. An overriding conclusion is that combining data across language groups can obscure important test validity information.     

THE SAT PREDICTION OF GRADES FOR MEXICAN-AMERICAN VERSUS ANGLO-AMERICAN STUDENTS AT THE UNIVERSITY OF CALIFORNIA, RIVERSIDE

The regression equations for second quarter freshman grade point averages on SAT scores were calculated for Anglo-American and Mexican-American students at the University of California, Riverside. These regression equations differed significantly for the two groups. However, the use of the regression equation derived from the Anglo-American sample to predict grades of Mexican-American students resulted in overprediction. An examination of the standardized regression weights revealed a significant difference in the weight given to SATM. A replication on a much larger sample revealed a similar outcome. These results were considered as a possible heuristic to suggest a scholastic "'strategy" difference between the two ethnic groups.     

Selection bias in college admissions test scores

Data from college admissions tests can provide a valuable measure of student achievement, but the non-representativeness of test-takers is an important concern. We examine selectivity bias in both state-level and school-level SAT and ACT averages. The degree of selectivity may differ importantly across and within schools, and across and within states. To identify within-state selectivity, we use a control function approach that conditions on scores from a representative test. Estimates indicate strong selectivity of test-takers in “ACT states,” where most college-bound students take the ACT, and much less selectivity in SAT states. To identify within- and between-school selectivity, we take advantage of a policy reform in Illinois that made taking the ACT a graduation requirement. Estimates based on this policy change indicate substantial positive selection into test participation both across and within schools. Despite this, school-level averages of observed scores are extremely highly correlated with average latent scores, as across-school variation in sample selectivity is small relative to the underlying signal. As a result, in most contexts the use of observed school mean test scores in place of latent means understates the degree of between-school variation in achievement but is otherwise unlikely to lead to misleading conclusions.     

SAT SCORES AND AMERICAN STATES: SEEKING FOR USEFUL MEANING

Despite difficulties, states frequently seek to compare their test performance with that of other states, or with the published national norms. All states have some students taking the Scholastic Aptitude Test (SAT) exams for college admissions, but because the numbers vary widely, when the state means are made public by the College Boards, there is debate about their interpretation. Here the 50 state SAT means are the criterion in a multivariate model, together with the state variables of percent taking the SATs (very influential), percent of minority in the state, average income, unemployment rate, and per pupil expenditure. Regression is analyzed (without expenditure) to examine the state residuals compared with the "expected" scores. And briefly noted is the difficulty of using alternative tests for such comparisons, as these are currently normed, selected, and used.     

The Primary Classification Test

ABSTRACT

The authors sought to better understand the relationship between students participating in the Advanced Placement (AP) program and subsequent performance on the Scholastic Aptitude Test (SAT). Focusing on students graduating from U.S. public high schools in 2010, the authors used propensity scores to match junior year AP examinees in 3 subjects to similar students who did not take any AP exams in high school. Multilevel regression models with these matched samples demonstrate a mostly positive relationship between AP exam participation and senior year SAT performance, particularly for students who score a 3 or higher. Students who enter into the AP year with relatively lower initial achievement are predicted to perform slightly better on later SAT tests than students with similar initial achievement who do not participate in AP.     

Selectivity bias and the determinants of SAT scores

The determinants of state-wide average SAT scores are estimated for 1982 in a regression analysis which corrects for the proportion of students taking the test. The selectivity correction has a large impact on the estimates of the effects of other variables. Little effect of schooling variables (teachers' salaries, teachers per pupil, other expenditures) is found in the selectivity-adjusted estimates, except that large schools seem to depress SAT scores and private schools enhance scores. State-wide high school graduation standards also do not explain SAT score variation. In contrast, several demographic variables are quite important, including family size, the college-educated fraction of the population, and female-headed households. We can explain some of the SAT score decline in the 1970s with these cross-section estimates, suggesting that the decline over time is not due to changing resources for schooling but, in part, to changing demographics. In particular, a large part of the recent SAT score decline was caused by the large families of the post-war baby boom.     

Can High School Achievement Tests Serve to Select College Students?

Postsecondary schools have traditionally relied on admissions tests such as the SAT and ACT to select students. With high school achievement assessments in place in many states, it is important to ascertain whether scores from those exams can either supplement or supplant conventional admissions tests. In this study we examined whether the Arizona Instrument to Measure Standards (AIMS) high school tests could serve as a useful predictor of college performance. Stepwise regression analyses with a predetermined order of variable entry revealed that AIMS generally did not account for additional performance variation when added to high school grade-point average (HSGPA) and SAT. However, in a cohort of students that took the test for graduation purposes, AIMS did account for about the same proportion of variance as SAT when added to a model that included HSGPA. The predictive value of both SAT and AIMS was generally the same for Caucasian, Hispanic, and Asian American students. The ramifications of universities using high school achievement exams as predictors of college success, in addition to or in lieu of traditional measures, are discussed.     

Segregated school effects on first grade reading gains: Using propensity score matching to disentangle effects for African-American,Latino, and European-American students

Increasing evidence from observational studies indicates that students attending minority segregated schools are at risk for constrained performance in reading. However, analyses of data gathered under observational conditions may yield biased results. Using data from the Early Childhood Longitudinal Study, 1998–1999 Kindergarten Cohort, this study used propensity score matching to address selection bias due to students’ observed socio-economic, literacy, and social-emotional background characteristics, allowing for a less biased estimate of minority segregated schooling on African-American, Latino, and European-American students’ reading gains in first grade. We found that African-American students attending segregated schools made less gain in reading across the first grade year than African-American students in non-segregated schools. There was no evidence for significant negative effects of segregation on reading gains for Latino and European-American students.     

SCHOLASTIC APTITUDE,MATRICULATION GRADES,AND GROUP INTERVIEW AS PREDICTORS OF PERFORMANCE IN NURSING SCHOOL

This study examines the predictive validity of three commonly used nursing school admission indices, that is, scholastic aptitude test scores, matriculation grades, and evaluations of performance in a group interview situation, in a sample of 321 Israeli nursing school students. Grade point average, supervisor evaluation of clinical internship, and scores on a government certification exam served as primary indices of criterion performance. Whereas composite aptitude test scores correlated moder ately with both grade point average and certification exam scores, matriculation grades correlated negligibly with all three criterion measures. Group interview ratings correlated moderately with clinical performance, but negligibly with the remaining criteria. Aptitude test scores were not found to be biased predictors of criterion performance by ethnicity or social background. The implications of these findings for the selection of nursing school candidates in Israel are discussed.     

Measuring teaching quality in higher education: assessing selection bias in course evaluations

Student evaluations of teaching (SETs) are widely used to measure teaching quality in higher education and compare it across different courses, teachers, departments and institutions. Indeed, SETs are of increasing importance for teacher promotion decisions, student course selection, as well as for auditing practices demonstrating institutional performance. However, survey response is typically low, rendering these uses unwarranted if students who respond to the evaluation are not randomly selected along observed and unobserved dimensions. This paper is the first to fully quantify this problem by analyzing the direction and size of selection bias resulting from both observed and unobserved characteristics for over 3000 courses taught in a large European university. We find that course evaluations are upward biased, and that correcting for selection bias has non-negligible effects on the average evaluation score and on the evaluation-based ranking of courses. Moreover, this bias mostly derives from selection on unobserved characteristics, implying that correcting evaluation scores for observed factors such as student grades does not solve the problem. However, we find that adjusting for selection only has small impacts on the measured effects of observables on SETs, validating a large related literature which considers the observable determinants of evaluation scores without correcting for selection bias.     

Simultaneous-equation bias in higher-education cost functions

Ordinary least squares regression (OLS) is often used to estimate the relationship between cost and enrollment. Cost is normally regarded as being a function of enrollment, and the regression equation is structured accordingly. If, however, enrollment is also a function of cost, which some have suggested, then OLS estimates will be biased and inconsistent. The problem is known as simultaneous-equation bias. This study addresses the materiality of this threat in a representative higher-education context. A model is constructed in which it is assumed that cost and enrollment at public two-year colleges are mutually influential. The model is then estimated by both OLS and two-stage least squares regression. The results differ only slightly. This suggests that the prudent use of OLS in similar situations need not be proscribed.     

Substituting SAT II: Subject Tests for SAT I: Reasoning Tests: Impact on Admitted Class Composition and Quality

Using data from a sample of 10 colleges at which most students had taken both SAT I: Reasoning tests and SAT II: Subject tests, we simulated the effects of making selection decisions using SAT II scores in place of SAT I scores. Specifically, we treated the students in each college as forming the applicant pool for a more select college, and then selected the top two thirds (and top one third) of the students using high school grade point average combined with either SAT I scores or the average of SAT II scores. Success rates, in terms of first-year grade point averages, were virtually identical for students selected by the different models. The percentage of African American, Asian American, and White students selected varied only slightly across models. Appreciably more Mexican American and Other Latino students were selected with the model that used SAT II scores in place of SAT I scores because these students submitted subject test scores for the Spanish test on which they had high scores.     

Generalized Linear Factor Score Regression: A Comparison of Four Methods

Factor score regression has recently received growing interest as an alternative for structural equation modeling. However, many applications are left without guidance because of the focus on normally distributed outcomes in the literature. We perform a simulation study to examine how a selection of factor scoring methods compare when estimating regression coefficients in generalized linear factor score regression. The current study evaluates the regression method and the correlation-preserving method as well as two sum score methods in ordinary, logistic, and Poisson factor score regression. Our results show that scoring method performance can differ notably across the considered regression models. In addition, the results indicate that the choice of scoring method can substantially influence research conclusions. The regression method generally performs the best in terms of coefficient and standard error bias, accuracy, and empirical Type I error rates. Moreover, the regression method and the correlation-preserving method mostly outperform the sum score methods.     

Financial Aid and College Persistence: Do Student Loans Help or Hurt?

Using data from two freshmen cohorts at a public research university (N = 3730), this study examines the relationship between loan aid and second-year enrollment persistence. Applying a counterfactual analytical framework that relies on propensity score (PS) weighting and matching to address selection bias associated with treatment status, the study estimates that loan aid exerts a significant negative effect on persistence for students from low-income background (i.e., Pell eligible), and those taking up high amounts of loans in order to meet total cost of attendance, including students who exhausted the available amount of subsidized loan aid. However, no significant incremental effect associated with unsubsidized loan aid, net of subsidized loan aid, could be detected. The estimated effect of loan aid on persistence controls for first-year academic experience and takes into account 26 factors related to loan selection and persistence in order to match students with loan aid to a counterfactual case in covariate adjusted regression. Comparison with results from non-matched-sample analysis suggests selection bias may mask the negative effect of loans detected with matched-sample estimation. Validity of covariates determining the loan selection process and criteria for acceptable balance in the matched data are discussed, and implications for future research are addressed.     

Bunny Hill or Black Diamond: Differences in Advanced Course‐Taking in College as a Function of Cognitive Ability and High School GPA

College students commonly have considerable course choice, and they can differ substantially in the proportion of their coursework taken at an advanced level. While advanced coursework is generally viewed as a desirable component of a student's education, research has rarely explored differences in student course‐taking patterns as a measure of academic success in college. We examined the relationship between the SAT, high school grade point average (HSGPA), and the amount of advanced coursework taken in a sample of 62 colleges and 188,985 students. We found that both the SAT and HSGPA predict enrollment in advanced courses, even after controlling for advanced placement (AP) credits and demographic variables. The SAT subtests of Critical Reading, Writing, and Math displayed differential relationships with advanced course‐taking dependent on student major. Gender and race/ethnicity were also related to advanced course‐taking, with women taking more advanced courses in all major categories except for science, technology, engineering, and mathematics (STEM) where they took fewer, even after controlling for other variables. Socioeconomic status had a negligible relationship with advanced course‐taking. This research broadens our understanding of academic achievement in college and the goals of admissions in higher education.     

Sources of Uncertainty Often Ignored in Adjusting State Mean SAT Scores for Differential Participation Rates: The Rules of the Game

In this article, we examine two attempts to adjust state mean Scholastic Aptitude Test (SAT) scores for differential participation rates. We show that both attempts can be rejected because of overly stringest identifying conditions in the estimation equations, as well as because within-state SAT score distributions indicate that the selection model employed is too restrictive. We suggest that attempts to do such adjustments ought to follow five simple rules. Adherence to these rules will foster follow-up checks on untested assumptions. We also suggest that soon-to-be available National Assessment of Educational Progress OIJAEP) state data is a better way to make state comparisons.     

Selection blas and the analysis of persistence of part-time undergraduate students

Most studies of persistence behavior use path analysis or ordinary least squares regression to estimate unknown coefficients. However, estimates produced by these techniques are biased if selectivity bias contaminates choices made by individuals in the data sample. We explain this problem, argue that it is present in data samples used in persistence studies, and discuss an alternative estimation technique that controls for it. The methodology and the differences in the interpretation of coefficient estimates are illustrated with a data sample of individual students at a single university.     

Using Wherry's Adjusted R 2 and Mallow's Cp for Model Selection From All Possible Regressions

Selecting a subset of predictors from a pool of potential predictors continues to be a common problem encountered by applied researchers in education. Because of several limitations associated with stepwise variable selection procedures, the examination of all possible regression solutions has been recommended. The authors evaluated the use of Mallow's C_p and Wherry's adjusted R ² statistics to select a final model from a pool of model solutions. Neither the C_p nor the adjusted R ² statistic correctly identified the underlying regression model any better and was generally worse than the stepwise selection method, which itself was poor. Using any of the model selection procedures studied here resulted in biased estimates of the authentic regression coefficients and underestimation of their standard errors. The use of theory and professional judgment is recommended for the selection of variables in a prediction equation.     

Detecting bias in selection for higher education: three different methods

This study examined selection bias in Israeli university admissions with respect to test language and gender, using three approaches for the detection of such bias: Cleary’s model of differential prediction, boundary conditions for differential prediction and difference between d’s (the Constant Ratio Model). The university admissions process in Israel, like those in many countries, is based on a combination of school-related achievement and a general scholastic aptitude test. The selection process was found to be biased in favour of Arabic speakers and not biased with respect to gender. The three approaches for detecting selection bias were similar in the pattern of the results they produced, but differed, as expected, in the magnitude of the bias they detected. The discussion focuses on the results both with respect to the specific groups studied (first research question) and with respect to the three approaches for detecting selection bias (second research question).