An Exploratory Analysis of Differential Item Functioning and Its Possible Sources in a Higher Education Admissions Context

We analyzed a pool of items from an admissions test for differential item functioning (DIF) for groups based on age, socioeconomic status, citizenship, or English language status using Mantel-Haenszel and item response theory. DIF items were systematically examined to identify its possible sources by item type, content, and wording. DIF was primarily found in the citizenship group. As suggested by expert reviewers, possible sources of DIF in the direction of U.S. citizens was often in Quantitative Reasoning in items containing figures, charts, tables depicting real-world (as opposed to abstract) contexts. DIF items in the direction of non-U.S. citizens included “mathematical” items containing few words. DIF for the Verbal Reasoning items included geocultural references and proper names that may be differentially familiar for non-U.S. citizens. This study is responsive to foundational changes in the fairness section of the Standards for Educational and Psychological Testing, which now consider additional groups in sensitivity analyses, given the increasing demographic diversity in test-taker populations.     

A Comparison of Quantitative Questions in Open-Ended and Multiple-Choice Formats

Open–ended counterparts to a set of items from the quantitative section of the Graduate Record Examination (GRE–Q) were developed. Examinees responded to these items by gridding a numerical answer on a machine-readable answer sheet or by typing on a computer. The test section with the special answer sheets was administered at the end of a regular GRE administration. Test forms were spiraled so that random groups received either the grid-in questions or the same questions in a multiple-choice format. In a separate data collection effort, 364 paid volunteers who had recently taken the GRE used a computer keyboard to enter answers to the same set of questions. Despite substantial format differences noted for individual items, total scores for the multiple-choice and open-ended tests demonstrated remarkably similar correlational patterns. There were no significant interactions of test format with either gender or ethnicity.     

A Simple Answer to a Simple Question on Changing Answers

In an article in the Winter 2011 issue of the Journal of Educational Measurement, van der Linden, Jeon, and Ferrara suggested that “test takers should trust their initial instincts and retain their initial responses when they have the opportunity to review test items.” They presented a complex IRT model that appeared to show that students would be worse off by changing answers. As noted in a subsequent erratum, this conclusion was based on flawed data, and that the correct data could not be analyzed by their method because the model failed to converge. This left their basic question on the value of answer changing unanswered. A much more direct approach is to simply count the number of examinees whose scores after an opportunity to change answers are higher, lower, or the same as their initial scores. Using the same data set as the original article, an overwhelming majority of the students received higher scores after the opportunity to change answers.     

From knowing to doing: assessing the skills used to teach reading and writing

Reading and Writing - This study presents results from a project to develop and pilot a new type of performance assessment of the skills used when teaching reading and writing. To provide context...     

POTENTIAL IMPACT OF THE ADDITION OF A WRITING ASSESSMENT ON ADMISSIONS DECISIONS

The effects that adding a writing measure to theGraduate Management Admissions Test would have on thegender and ethnic composition of an admissions pool wereevaluated. Data from the first operational administration of the Analytical WritingAssessment in October 1994 were used. Standardizeddifferences from the white male reference group werecomputed separately for men and women in four ethnicgroups: white, Asian-American, African-American, andHispanic/Latino. Within the white, African-American, andHispanic/Latino groups, women received higher scoresthan men on the Analytical Writing Assessment; in the Asian-American group, men receivedhigher writing scores, but the difference was not asgreat as on the Verbal score. Examinees whose bestlanguage was not English scored relatively higher on the Analytical Writing Assessment than on theVerbal measure. Simulations of eligibility for anadmissions pool suggested that the addition of theAnalytical Writing score would noticeably increase thenumber of women in the pool, but would have virtuallyno impact on ethnic minorities. Rater and scorereliability were reasonably consistent across ethnic andgender groups.     

The Relationship of Essay and Multiple-Choice Scores With Grades in College Courses

Essay and multiple-choice scores from Advanced Placement (AP) examinations in American History, European History, English Language and Composition, and Biology were matched with freshman grades in a sample of 32 colleges. Multiple-choice scores from the American History and Biology examinations were more highly correlated with freshman grade point averages than were essay scores from the same examinations, but essay scores were essentially equivalent to multiple-choice scores in correlations with course grades in history, English, and biology. In history courses, men and women received comparable grades and had nearly equal scores on the AP essays, but the multiple-choice scores of men were nearly one half of a standard deviation higher than the scores of women.     

Comparability of essay question variants

Writing task variants can increase test security in high-stakes essay assessments by substantially increasing the pool of available writing stimuli and by making the specific writing task less predictable. A given prompt (parent) may be used as the basis for one or more different variants. Six variant types based on argument essay prompts from a large-scale, high-stakes North American writing assessment and six based on issue prompts from the same test were created and evaluated in the research section of the test administrations in the winter of 2009. Examinees were asked to volunteer to write an essay on one of the new prompt/variant tasks. Essays were obtained from 7573 examinees for argument prompts and 10,827 examinees for issue prompts. Results indicated that all variant types produced reasonably similar means, standard deviations, and rater reliabilities, suggesting that the variant strategy should be useable for operational administrations in high stakes essay assessments. Variant type did not interact with gender, ethnicity, or language (self-report that English or another language is the examinee's “best” language).     

Impact of Fewer Questions per Section on SAT I Scores

The impact of allowing more time for each question on the SAT I: Reasoning Test scores was estimated by embedding sections with a reduced number of questions into the standard 30-minute equating section of two national test administrations. Thus, for example, questions were deleted from a verbal section that contained 35 questions to produce forms that contained 27 or 23 questions. Scores on the 23-question section could then be compared to scores on the same 23 questions when they were embedded in a section that contained 27 or 35 questions. Similarly, questions were deleted from a 25-question math section to form sections of 20 and 17 questions. Allowing more time per question had a minimal impact on verbal scores, producing gains of less than 10 points on the 200–800 SAT scale. Gains for the math score were less than 30 points. High-scoring students tended to benefit more than lower-scoring students, with extra time creating no increase in scores for students with SAT scores of 400 or lower. Ethnic/racial and gender differences were neither increased nor reduced with extra time.