Repeater Analysis for Combining Information From Different Assessments

Admission decisions frequently rely on multiple assessments. As a consequence, it is important to explore rational approaches to combine the information from different educational tests. For example, U.S. graduate schools usually receive both TOEFL iBT^® scores and GRE^® General scores of foreign applicants for admission; however, little guidance has been given to combine information from these two assessments, even though the relationships between such sections as GRE Verbal and TOEFL iBT Reading are obvious. In this study, principles are provided to explore the extent to which different assessments complement one another and are distinguishable. Augmentation approaches developed for individual tests are applied to provide an accurate evaluation of combined assessments. Because augmentation methods require estimates of measurement error and internal reliability data are unavailable, required estimates of measurement error are obtained from repeaters, examinees who took the same test more than once. Because repeaters are not representative of all examinees in typical assessments, minimum discriminant information adjustment techniques are applied to the available sample of repeaters to treat the effect of selection bias. To illustrate methodology, combining information from TOEFL iBT scores and GRE General scores is examined. Analysis suggests that information from the GRE General and TOEFL iBT assessments is complementary but not redundant, indicating that the two tests measure related but somewhat different constructs. The proposed methodology can be readily applied to other situations where multiple assessments are needed.     

Predictors of Success on the Counselor Preparation Comprehensive Examination

This study examined the relationship between 403 counseling graduate students' scores on the Counselor Preparation Comprehensive Examination (CPCE; Center for Credentialing and Education, n.d.) and 3 admissions requirements used as predictor variables: undergraduate grade point average (UGPA), Graduate Record Examinations (GRE) General Test Verbal Reasoning (GRE‐V) score, and GRE General Test Quantitative Reasoning (GRE‐Q) score. Multiple regression analyses revealed that all predictor variables accounted for somewhat limited, yet significant variations in the CPCE‐Total scores (R² = .21). Results indicated that UGPAs, GRE‐V scores, and GRE‐Q scores are valid criteria for determining counseling graduate student success on the CPCE.     

EFFECTS OF COACHING ON GRE APTITUDE TEST SCORES

Test preparation activities were determined for a large representative sample of Graduate Record Examination (GRE) Aptitude Test takers. About 3% of these examinees had attended formal coaching programs for one or more sections of the test.
After adjusting for differences in the background characteristics of coached and uncoached students, effects on test scores were related to the length and the type of programs offered. The effects on GRE verbal ability scores were not significantly related to the amount of coaching examinees received, and quantitative coaching effects increased slightly but not significantly with additional coaching. Effects on analytical ability scores, on the other hand, were related significantly to the length of coaching programs, through improved performance on two analytical item types, which have since been deleted from the test.
Overall, the data suggest that, when compared with the two highly susceptible item types that have been removed from the GRE Aptitude Test, the test item types in the current version of the test (now called the GRE General Test) appear to show relatively little susceptibility to formal coaching experiences of the kinds considered here.     

Beyond the GRE: Using a Composite Score to Predict ?the Success of Puerto Rican Students in a Biomedical ?PhD Program

The use and validity of the Graduate Record Examination General Test (GRE) to predict the success of graduate school applicants is heavily debated, especially for its possible impact on the selection of underrepresented minorities into science, technology, engineering, and math fields. To better identify candidates who would succeed in our program with less reliance on the GRE and grade point average (GPA), we developed and tested a composite score (CS) that incorporates additional measurable predictors of success to evaluate incoming applicants. Uniform numerical values were assigned to GPA, GRE, research experience, advanced course work or degrees, presentations, and publications. We compared the CS of our students with their achievement of program goals and graduate school outcomes. The average CS was significantly higher in those students completing the graduate program versus dropouts (p < 0.002) and correlated with success in competing for fellowships and a shorter time to thesis defense. In contrast, these outcomes were not predicted by GPA, science GPA, or GRE. Recent implementation of an impromptu writing assessment during the interview suggests the CS can be improved further. We conclude that the CS provides a broader quantitative measure that better predicts success of students in our program and allows improved evaluation and selection of the most promising candidates.     

The Differential Impact of Curriculum on Aptitude Test Scores

A sample of 22, 923 students who had taken the SAT and the GRE General Test was classified by the four general undergraduate fields of study and by sex. The authors performed several analyses to determine the degree of differential impact that sex and field of study might have on GRE-Verbal, GRE-Quantitative, and GRE-Analytical scores after controlling on SAT-Verbal and SAT-Mathematical scores. They found, first, that the correlations of SAT-Verbal with GRE-Verbal scores and SAT-Mathematical with GRE-Quantitative scores were extremely high, .86 in the total sample and ranging from the low to middle .80s in the eight subgroups. The impact of curriculum and sex, after controlling on SAT scores, was found to be low on GRE- Verbal scores but relatively high on GRE-Quantitative scores, with students in heavily quantitative fields enjoying an advantage over their peers in less quantitative fields of study. The impact was moderate on GRE-Analytical scores. Further studies designed to "purify" the fields of study and include only clearly verbal fields and clearly mathematical fields showed small additional impact. An additional study indicated a generally slight effect of the institution attended on GRE-Quantitative scores after controlling for sex, major field of study, and initial ability.     

Validity of GRE General Test scores and TOEFL scores for graduate admission to a technical university in Western Europe*

Graduate admission has become a critical process in tertiary education, whereby selecting valid admissions instruments is key. This study assessed the validity of Graduate Record Examination (GRE) General Test scores for admission to Master’s programmes at a technical university in Europe. We investigated the indicative value of GRE scores for the Master’s programme grade point average (GGPA) with and without the addition of the undergraduate GPA (UGPA) and the TOEFL score, and of GRE scores for study completion and Master’s thesis performance. GRE scores explained 20% of the variation in the GGPA, while additional 7% were explained by the TOEFL score and 3% by the UGPA. Contrary to common belief, the GRE quantitative reasoning score showed only little explanatory power. GRE scores were also weakly related to study progress but not to thesis performance. Nevertheless, GRE and TOEFL scores were found to be sensible admissions instruments. Rigorous methodology was used to obtain highly reliable results.     

GRE一般测验与GMAT定量能力考查之比较研究

GRE一般测验与GMAT定量能力的考查内容均为初等数学,主要强调解决日常生活工作中遇到的数量问题。但因二者的适用范围有所不同,其能力考查存在一定差异,GMAT对数学逻辑思维的考查比较深入,而GRE一般测验对数学概念的准确性以及思维的全面性有较多考查。在我国硕士研究生入学考试中设置一般能力测试数学部分的考查应根据就读研究生学科专业对数学的不同要求及考生的实际能力和水平设置相应考试内容,试题情境设置应联系实际,贴近生活,试题难度不宜太大,以60%的考生通过资格线来控制试题难度。     

The graduate record examination as a predictor of graduate student performance: The case of criminal justice

Research on the Graduate Record Examination (GRE) as a predictor of graduate student performance has been marked by much debate. Thornell and McCoy (1985) have found that the relationship between student performance and the GRE may be sensitive to the discipline being studied. While unexplained to date, this disciplinary variation may be related to the differences in means and/or relative dispersions of GRE scores among disciplines. Further, there has been only one study (McKee, Mallory, and Campbell, in press) for criminal justice, the field with the lowest mean GRE score. This paper focuses on this neglected field. Data were collected from the files of 70 criminal justice majors in a master's program at a large midwestern urban university. The results indicate that GRE scores are largely unrelated to indicators of graduate student performance (GPA, grades in specific classes, and the completion of the M.A. degree). The one exception is a strong relationship between verbal GRE and graduate GPA. This relationship may indicate a nonlinear ceiling effect wherein verbal GRE scores affect GGPA, mainly in disciplines with means at the low end of the verbal GRE score distribution.     

Who Benefits Most From Preparing for a “Coachable” Admissions Test?

A previous study of the initial, preoperational version of the Graduate Record Examinations (GRE) analytical ability measure (Powers & Swinton, 1984) revealed practically and statistically significant effects of test familiarization on analytical test scores. (Two susceptible item types were subsequently removed from the test.) Data from this study were reanalyzed for evidence of differential effects for subgroups of examinees classified by age, ethnicity, degree aspiration, English language dominance, and performance on other sections of the GRE General Test. The results suggested little, if any, difference among subgroups of examinees with respect to their response to the particular kind of test preparation considered in the study. Within the limits of the data, no particular subgroup appeared to benefit significantly more or significantly less than any other subgroup.     

Graphical Modeling: A New Response Type for Measuring the Qualitative Component of Mathematical Reasoning

We investigated the functioning of a new computer-delivered response type for potential use in graduate admissions assessment. This response type, which is open-ended and automatically scorable, presents problems calling for the examinee to draw a graph modeling a given situation. Problem situations can be like the single-best-answer items currently found on the Graduate Record Examinations (GRE) General Test (ETS, 1998) or they can be more loosely defined, allowing for multiple-correct responses. Two graphical modeling (GM) tests differing from one another in the manipulation of specific item features were randomly spiraled among study participants. Results showed that GM scores were very reliable and moderately related to the General Test's quantitative section, suggesting that GM might help broaden the GRE quantitative construct. In exploratory difficulty analyses, 1 of 3 manipulated item features, problem structure, had a dependable effect. No significant gender differences independent of those associated with the GRE quantitative section were detected. Finally, more participants preferred regular multiple-choice graphical reasoning questions to GM items but thought GM was the fairer indicator of their ability to undertake graduate study.     

Improving Measurement for Graduate Admissions

In this study we examined alternative item types and section configurations for improving the discriminant and convergent validity of the GRE General Test. A computer-based test of reasoning items and a generating-explanations measure was administered to a sample of 388 examinees who previously had taken the General Test. Confirmatory factor analyses indicated that three dimensions of reasoning—verbal, analytical, and quantitative—and a fourth dimension of verbal fluency based on the generating-explanations task could be distinguished. Notably, generating explanations was as distinct from new variations of reasoning items as it was from verbal and quantitative reasoning. In the full sample, this differentiation was evident in relation to such external criteria as undergraduate grade point average (UGPA), self-reported accomplishments, and a measure of ideational fluency, with generating explanations relating uniquely to aesthetic and linguistic accomplishments and to ideational fluency. For the subset of participants with undergraduate majors in the humanities and social sciences, generating explanations added to the relationship with UGPA over that contributed by the General Test.     

A Computer-Based Task for Measuring the Representational Component of Quantitative Proficiency

In this study, we created a computer-delivered problem-solving task based on the cognitive research literature and investigated its validity for graduate admissions assessment. The task asked examinees to sort mathematical word problem stems according to prototypes. Data analyses focused on the meaning of sorting scores and examinee perceptions of the task. Results showed that those who sorted well tended to have higher GRE General Test scores and college grades than did examinees who sorted less proficiently. Examinees generally preferred this task to multiple-choice items like those found on the General Test's Quantitative section and felt the task was a fairer measure of their ability to succeed in graduate school. Adaptations of the task might be used in admissions tests, as well as for instructional assessments to help lower- scoring examinees localize and remediate problem-solving difficulties.     

Relative Effects of English Proficiency on General Admissions Tests Versus Subject Tests

The increasing linguistic diversity of the United States student population has brought to the forefront problems in the interpretation of test scores for non-native speakers of English in graduate admissions. The degree to which test scores reflect English proficiency was studied using data on N = 451 students whose native language was Spanish. They had taken the GRE, English proficiency tests, and the PAEG, a test in Spanish used for admission to graduate schools in Puerto Rico. Regression analyses revealed that the proportion of variance explained by the English proficiency terms (independent of developed skills measured in Spanish) was highest for the GRE verbal test (34%), lowest for the quantitative test (8%) and intermediate for the analytical test (16%), the Psychology (18%) and Biology (17%) Subject tests. These findings are discussed in light of efforts to increase access to higher education for Hispanic students.     

Testing the Predictive Validity of the GRE Exam on Communication Graduate Student Success: A Case Study at University at Buffalo

GRE scores for entering M.A. and Ph.D. students in Communication at University at Buffalo from 1990 to 2001 were used to predict graduate student success. Graduate student success was measured in two ways: grade point average (GGPA) and graduation rates. Preliminary analyses on Master's students (M.A.) revealed that international students, compared to domestic students, earned lower GRE verbal (GRE-V) scores, higher GRE quantitative (GRE-Q) scores, had higher undergraduate grade point averages and were more likely to graduate from the program (82% vs. 49%). Only GRE scores were different for International vs. Domestic students when examining doctoral students. For M.A. students, GRE-V was positively correlated with GGPA and GRE-Q was positively related to earning the M.A. degree. Regression analyses controlling for several factors, including domestic student status, found only undergraduate GPA to predict GGPA and graduation for M.A. students. GGPA and GRE-V predicted graduation rates for M.A. students when GGPA was included in logistic regression analysis. GRE, GPA, and GGPA failed to predict Ph.D. success when examined together.     

GRE candidates' perceptions of the importance of graduate admission factors

This study examines the perceptions of a representative sample of GRE test takers who were asked to indicate their views of the importance of eight widely considered factors in graduate admissions. Overall, candidates perceived undergraduate grades as the most important factor in graduate admissions. Recommendations and one's undergraduate field were rated as somewhat less important than undergraduate grades, and GRE Aptitude Test scores were rated even less important. GRE Advanced (Subject) Test scores were perceived as considerably less important than any other factor. Analyses by subgroup revealed that candidates' perceptions differed markedly according to the graduate field they intended to enter. Perceptions also differed by ethnic group (blacks versus whites) but not by sex or age.     

Generalizability, Validity, and Examinee Perceptions of a Computer-Delivered Formulating-Hypotheses Test

The Formulating-Hypotheses (F-H) item presents a situation and asks examinees to generate as many explanations for it as possible. This study examined the generalizability, validity, and examinee perceptions of a computer-delivered version of the task. Eight F-H questions were administered to 192 graduate students. Half of the items restricted examinees to 7 words per explanation, and half allowed up to 15 words. Generalizability results showed high interrater agreement, with tests of between 2 and 4 items scored by one judge achieving coefficients in the .80s. Construct validity analyses found that F-H was only marginally related to the GRE General Test, and more strongly related than the General Test to a measure of ideational fluency. Different response limits tapped somewhat different abilities, with the 15-word constraint appearing more useful for graduate assessment. These items added significantly to conventional measures in explaining school performance and creative expression.     

Estimating graduate admission standards in psychology

Because publicized minimum criteria for graduate school admission are rarely sufficient to gain entrance to the school of one's choice, a study was undertaken to estimate the actual or effective admission standards to graduate school. Booklets composed of 27 hypothetical graduate school applicants conforming to a 3 (GPA) × 3 (GRE) × 3 (trait adjectives) factorial design were evaluated by graduate program chairpersons from the marginal, low, middle, and high categories of the Roose and Andersen (1970), Cox and Catt (1977), and Endler et al. (1978) ranking systems of schools in psychology. Results showed that a linear model of judgment would account for the ratings of potential for success in graduate school and the decision to accept or reject the hypothetical applicants. GRE scores accounted for about twice as much variance (30%) as GPA (16%), which accounted for about twice as much variance (7%) as trait adjectives. The Roose and Andersen listing was shown to be the most acceptable ranking system when estimating effective graduate admission standards.     

Can a Two‐Question Test Be Reliable and Valid for Predicting Academic Outcomes?

Scores on essay‐based assessments that are part of standardized admissions tests are typically given relatively little weight in admissions decisions compared to the weight given to scores from multiple‐choice assessments. Evidence is presented to suggest that more weight should be given to these assessments. The reliability of the writing scores from two of the large volume admissions tests, the GRE General Test (GRE) and the Test of English as a Foreign Language Internet‐based test (TOEFL iBT), based on retesting with a parallel form, is comparable to the reliability of the multiple‐choice Verbal or Reading scores from those tests. Furthermore, and even more important, the writing scores from both tests are as effective as the multiple‐choice scores in predicting academic success and could contribute to fairer admissions decisions.     

Can you earn a Ph.D. in economics in five years?

We investigate graduate school outcomes for students who entered economics Ph.D. programs in fall 2002. Students in Top-15 ranked programs and those with higher verbal and quantitative GRE scores are less likely to have dropped out, but no more likely to have graduated. Those with undergraduate degrees from Top-60 U.S. liberal arts colleges and from foreign universities have lower attrition and higher completion probabilities. There are important differences in the characteristics associated with retention and completion probabilities between U.S. citizens and non-citizens and between men and women.