Application of Think Aloud Protocols for Examining and Confirming Sources of Differential Item Functioning Identified by Expert Reviews

This paper demonstrates and discusses the use of think aloud protocols (TAPs) as an approach for examining and confirming sources of differential item functioning (DIF). The TAPs are used to investigate to what extent surface characteristics of the items that are identified by expert reviews as sources of DIF are supported by empirical evidence from examinee thinking processes in the English and French versions of a Canadian national assessment. In this research, the TAPs confirmed sources of DIF identified by expert reviews for 10 out of 20 DIF items. The moderate agreement between TAPs and expert reviews indicates that evidence from expert reviews cannot be considered sufficient in deciding whether DIF items are biased and such judgments need to include evidence from examinee thinking processes.     

Identifying Sources of Differential Item and Bundle Functioning on Translated Achievement Tests: A Confirmatory Analysis

Increasingly, tests are being translated and adapted into different languages. Differential item functioning (DIF) analyses are often used to identify non-equivalent items across language groups. However, few studies have focused on understanding why some translated items produce DIF. The purpose of the current study is to identify sources of differential item and bundle functioning on translated achievement tests using substantive and statistical analyses. A substantive analysis of existing DIF items was conducted by an 11-member committee of testing specialists. In their review, four sources of translation DIF were identified. Two certified translators used these four sources to categorize a new set of DIF items from Grade 6 and 9 Mathematics and Social Studies Achievement Tests. Each item was associated with a specific source of translation DIF and each item was anticipated to favor a specific group of examinees. Then, a statistical analysis was conducted on the items in each category using SIBTEST. The translators sorted the mathematics DIF items into three sources, and they correctly predicted the group that would be favored for seven of the eight items or bundles of items across two grade levels. The translators sorted the social studies DIF items into four sources, and they correctly predicted the group that would be favored for eight of the 13 items or bundles of items across two grade levels. The majority of items in mathematics and social studies were associated with differences in the words, expressions, or sentence structure of items that are not inherent to the language and/or culture. By combining substantive and statistical DIF analyses, researchers can study the sources of DIF and create a body of confirmed DIF hypotheses that may be used to develop guidelines and test construction principles for reducing DIF on translated tests.     

PISA2000阅读素养测验的项目功能差异分析

本文通过对PISA2000阅读素养测验的不同语言组的DIF检测,发现中文版出现DIF标记的项目较之其他语种版本而言,数量较多,在反思评价认知过程中尤其突出。日语与英语、芬兰语版本的DIF项目数量只有中文版本的二分之一左右。本文最后对DIF产生原因与检测方法的局限性进行了讨论。     

The Multidimensionality of Measurement Bias in High-Stakes Testing: Using Machine Learning to Evaluate Complex Sources of Differential Item Functioning

Test developers and psychometricians have historically examined measurement bias and differential item functioning (DIF) across a single categorical variable (e.g., gender), independently of other variables (e.g., race, age, etc.). This is problematic when more complex forms of measurement bias may adversely affect test responses and, ultimately, bias test scores. Complex forms of measurement bias include conditional effects, interactions, and mediation of background information on test responses. I propose a multidimensional, person-specific perspective of measurement bias to explain how complex sources of bias can manifest in the assessment of human knowledge, skills, and abilities. I also describe a data-driven approach for identifying key sources of bias among many possibilities—namely, a machine learning method commonly known as regularization.     

An analysis of Differential Item Functioning on INVALSI tests,designed to explore gender gap in mathematical tasks

This study presents a quantitative approach based on Differential Item Functioning analysis within the Item Response Theory framework, to quantify gender differences in tackling specific mathematical items. We use this approach to explore two crucial topics in mathematics education (misconceptions in decimal numbers, and estimation) by analysing answers given by Italian students to specific mathematical items taken from a sample of 1400 items administered in INVALSI tests since 2008. For each item, we have a sample of 30,000 students per year, statistically representative of the entire Italian student population. The results section presents a didactic interpretation of the statistical evidence and shows how interdisciplinarity between statistics and mathematics education, with a mixed-method approach (Johnson & Onwuegbuzie, 2004), represents a good approach to exploring the gender gap in relation to specific constructs of math education. This approach, common in many disciplines, can also make an interesting contribution to mathematics education research.