Analyzing Fairness Among Linguistic Minority Populations Using a Latent Class Differential Item Functioning Approach

ABSTRACT

Differential item functioning (DIF) analyses have been used as the primary method in large-scale assessments to examine fairness for subgroups. Currently, DIF analyses are conducted utilizing manifest methods using observed characteristics (gender and race/ethnicity) for grouping examinees. Homogeneity of item responses is assumed denoting that all examinees respond to test items using a similar approach. This assumption may not hold with all groups. In this study, we demonstrate the first application of the latent class (LC) approach to investigate DIF and its sources with heterogeneous (linguistic minority groups). We found at least three LCs within each linguistic group, suggesting the need to empirically evaluate this assumption in DIF analysis. We obtained larger proportions of DIF items with larger effect sizes when LCs within language groups versus the overall (majority/minority) language groups were examined. The illustrated approach could be used to improve the ways in which DIF analyses are typically conducted to enhance DIF detection accuracy and score-based inferences when analyzing DIF with heterogeneous populations.     

For Which Boys and Which Girls Are Reading Assessment Items Biased Against? Detection of Differential Item Functioning in Heterogeneous Gender Populations

In gender differential item functioning (DIF) research it is assumed that all members of a gender group have similar item response patterns and therefore generalizations from group level to subgroup and individual levels can be made accurately. However DIF items do not necessarily disadvantage every member of a gender group to the same degree, indicating existence of heterogeneity of response patterns within gender groups. In this article the impact of heterogeneity within gender groups on DIF investigations was investigated. Specifically, it was examined whether DIF results varied when comparing males versus females, gender × socioeconomic status subgroups and latent classes of gender. DIF analyses were conducted on reading achievement data from the Canadian sample of the Programme of International Student Assessment 2009. Results indicated considerable heterogeneity within males and females and DIF results were found to vary when heterogeneity was taken into account versus when it was not.     

An Investigation of the Language Demands in Content Assessments for English Language Learners

The purpose of the present study is to examine the language characteristics of a few states' large-scale assessments of mathematics and science and investigate whether the language demands of the items are associated with the degree of differential item functioning (DIF) for English language learner (ELL) students. A total of 542 items from 11 assessments at Grades 4, 5, 7, and 8 from three states were rated for the linguistic complexity based on a developed linguistic coding scheme. The linguistic ratings were compared to each item's DIF statistics. The results yielded a stronger association between the linguistic rating and DIF statistics for ELL students in the “relatively easy” items than in the “not easy” items. Particularly, general academic vocabulary and the amount of language in an item were found to have the strongest association with the degrees of DIF, particularly for ELL students with low English language proficiency. Furthermore, the items were grouped into four bundles to closely look at the relationship between the varying degrees of language demands and ELL students' performance. Differential bundling functioning (DBF) results indicated that the exhibited DBF was more substantial as the language demands increased. By disentangling linguistic difficulty from content difficulty, the results of the study provide strong evidence of the impact of linguistic complexity on ELL students' performance on tests. The study discusses the implications for the validation of the tests and instructions for ELL students.     

Differential Performance by English Language Learners on an Inquiry-Based Science Assessment

The performance of English language learners (ELLs) has been a concern given the rapidly changing demographics in US K-12 education. This study aimed to examine whether students' English language status has an impact on their inquiry science performance. Differential item functioning (DIF) analysis was conducted with regard to ELL status on an inquiry-based science assessment, using a multifaceted Rasch DIF model. A total of 1,396 seventh- and eighth-grade students took the science test, including 313 ELL students. The results showed that, overall, non-ELLs significantly outperformed ELLs. Of the four items that showed DIF, three favored non-ELLs while one favored ELLs. The item that favored ELLs provided a graphic representation of a science concept within a family context. There is some evidence that constructed-response items may help ELLs articulate scientific reasoning using their own words. Assessment developers and teachers should pay attention to the possible interaction between linguistic challenges and science content when designing assessment for and providing instruction to ELLs.     

Inconsistencies in DIF Detection for Sub-Groups in Heterogeneous Language Groups

Diversity and heterogeneity among language groups have been well documented. Yet most fairness research that focuses on measurement comparability considers linguistic minority students such as English language learners (ELLs) or Francophone students living in minority contexts in Canada as a single group. Our focus in this research is to examine the degree to which measurement comparability, as indicated by differential item functioning (DIF), is consistent for sub-groups among linguistic minority Francophone students in Canada. The findings suggest that the linguistic minority Francophone students who speak French at home and those who do not speak French at home should not be grouped together for investigating measurement comparability or for examining performance gaps. We identified a great degree of differences in DIF identification with a consistency of 7–10% in DIF identification in the separate analyses for the two groups. The findings highlight methodological problems with investigating fairness for diverse linguistic groups that are treated as a single group.     

Predicting ELL students’ beginning first grade English oral reading fluency from initial kindergarten vocabulary, letter naming, and phonological awareness skills

The precursors of early English reading success have been widely studied for native English-speaking students, and those findings have been generalized to the English language learner (ELL) student population. However, the development of English language acquisition may be different for ELL students. The purpose of this study was to investigate the predictive role of English letter naming fluency, initial sound fluency, and vocabulary skills at the time of kindergarten entry for first grade English oral reading fluency and to examine the variability in language and literacy skills of ELL students by their demographic characteristics. The data for this study came from the Progress Monitoring and Reporting Network (PMRN), and were collected from Florida's Reading First schools. Letter Naming Fluency, Initial Sound Fluency, and Oral Reading Fluency components of Dynamic Indicators of Basic Early Literacy Skills (DIBELS) and the Peabody Picture Vocabulary Test were used as measures. Hierarchical Linear Modeling was used to analyze the curvilinear growth of ELL students’ first grade oral reading fluency. The results of this study revealed that kindergarten English letter naming fluency was the best predictor and vocabulary skills were the second best predictor of oral reading fluency in the first grade, followed by initial sound fluency. On average, male ELL students compared to female ELL students, ELL students eligible for free or reduced price lunch eligibility (FRPL) compared to those not eligible for FRPL, and Hispanic ELL students compared to White ELL students read fewer words at the beginning of the first grade and showed a slower growth rate. English oral reading fluency scores of Asian ELL students were the highest.     

Language factors associated with achievement grouping in math classrooms: a cross-sectional and longitudinal study

The study examines the effects of classroom achievement grouping (AG) practices on the early mathematics performance of language-minority students and compares their mathematics achievement to that of English-speaking majority students. Using a nationally representative database of the USA, both cross-sectional and longitudinal analyses were done. In the cross-sectional analyses we explored the direct effect of grouping practice on students performance, while in the longitudinal analysis we looked at the growth trajectory in mathematics learning. The results of cross-sectional analyses indicated that the effect of AG was negative on the math achievement in 1st and 5th graders. The longitudinal analysis showed a significant negative effect of AG for English Language Learners (ELL). The paper provides the basis for practical guidelines for the grouping practices in mathematics.     

Differential Item Functioning for Minority Examinees on the SAT

The standardization approach to assessing differential item functioning (DIF), including standardized distractor analysis, is described. The results of studies conducted on Asian Americans, Hispanics (Mexican Americans and Puerto Ricans), and Blacks on the Scholastic Aptitude Test (SAT) are described and then synthesized across studies. Where the groups were limited to include only examinees who spoke English as their best language, very few items across forms and ethnic groups exhibited large DIF. Major findings include evidence of differential speededness (where minority examinees did not complete SAT-Verbal sections at the same rate as White students with comparable SAT-Verbal scores) for Blacks and Hispanics and, when the item content is of special interest, advantages for the relevant ethnic group. In addition, homographs tend to disadvantage all three ethnic groups, but the effect of vertical relationships in analogy items are not as consistent. Although these findings are important in understanding DIF, they do not seem to account for all differences. Other variables related to DIF still need to be identified. Furthermore, these findings are seen as tentative until corroborated by studies using controlled data collection designs.     

Assessing school supports for ELL students using the ECLS-K

Using a large and nationally representative sample of children, this study examined the association between the school resources available to English Language Learners and their academic trajectories from kindergarten through fifth grade. Particular attention was paid to the differences in services provided by schools with high, low, or no ELL student concentrations. Growth-curve modeling results indicated that ELL children started kindergarten with significantly lower reading and math scores compared to their English-speaking peers. However, ELL children improved their math scores fast enough that they were able to narrow the initial gap in math scores with their English-speaking peers by fifth grade. This was particularly true for ELL children in schools with either a high- or low ELL student concentration. In contrast, ELL children still had lower reading scores by fifth grade compared to their English-speaking peers. The results highlight the importance of school resources and services in improving ELL students’ academic trajectories. Directions for future studies are discussed.     

Impact of an Early Reading First program on the language and literacy achievement of children from diverse language backgrounds

This study used an age-cutoff regression discontinuity design to examine the impact of a well-resourced Early Reading First prekindergarten program designed to foster the language and literacy development of 4-year-old children from low-income homes. A special challenge for the application of the language-rich curriculum and professional development package implemented in this study was the presence of a large proportion of ELL children in essentially English-speaking classrooms. We, therefore, sought to determine whether the program was effective for improving English language and literacy outcomes for English-language learners as well as native English speakers. There were large and significant differences between treatment and control groups on literacy outcomes for all students. On the literacy tasks, ELL students in the treatment groups performed nearly as well or better than non-ELL students at the beginning of kindergarten, and reached national norms on standardized tests. There were also significant program impacts on some language outcomes for all students. ELL students who received the intervention significantly outperformed ELL students in the control groups on English receptive and expressive vocabulary. On the more complex oral comprehension skills, preschool did not have a significant impact for ELL students. Intervention effects on receptive vocabulary and oral comprehension for native speakers were found only for the third cohort and were not found for expressive vocabulary. These results provide evidence that, given material supports, coaching, professional development, and the use of a language and literacy-focused curriculum, prekindergarten classrooms can enable low-SES children from diverse language backgrounds to enter kindergarten with literacy skills at or near national norms and can significantly impact some language skills. While non-native speakers of English continued to score lower on language measures than their native-speaking peers, results show that 1 year of preschool can put all children on a positive trajectory for long-term success in school.     

Stepwise Analysis of Differential Item Functioning Based on Multiple-Group Partial Credit Model

Bock, Muraki, and Pfeiffenberger (1988) proposed a dichotomous item response theory (IRT) model for the detection of differential item functioning (DIF), and they estimated the IRT parameters and the means and standard deviations of the multiple latent trait distributions. This IRT DIF detection method is extended to the partial credit model (Masters, 1982; Muraki, 1993) and presented as one of the multiple-group IRT models. Uniform and non-uniform DIF items and heterogeneous latent trait distributions were used to generate polytomous responses of multiple groups. The DIF method was applied to this simulated data using a stepwise procedure. The standardized DIF measures for slope and item location parameters successfully detected the non-uniform and uniform DIF items as well as recovered the means and standard deviations of the latent trait distributions.This stepwise DIF analysis based on the multiple-group partial credit model was then applied to the National Assessment of Educational Progress (NAEP) writing trend data.     

Relationship Between Language and Concept Science Notebook Scores of English Language Learners and/or Economically Disadvantaged Students

Despite research interest in testing the effects of literacy-infused science interventions in different contexts, research exploring the relationship, if any, between academic language and conceptual understanding is scant. What little research exists does not include English language learners (ELLs) and/or economically disadvantaged (ED) student samples—students most at risk academically. This study quantitatively determined if there exists a relationship, and if so, how strong of a relationship, between ELL and ED students’ academic language and conceptual understanding based on science notebook scores used in a larger science and literacy-infused intervention with a sample of culturally diverse students. The study also considered strengths of relationships between language and concept science notebook scores within student language status groups (ELL, former ELL, and English speaking). Correlational analyses noted positive, large, and significant correlations between students’ language and concept scores overall, with the largest correlations for science notebook entries using more academic language. Large correlations also existed for ELL student entries at the end of the school year. Implications of the findings for future research and practice in science classrooms including literacy interventions, such as science notebooks, with populations of culturally diverse students are discussed.

     

Relationship Between Language and Concept Science Notebook Scores of English Language Learners and/or Economically Disadvantaged Students

Despite research interest in testing the effects of literacy-infused science interventions in different contexts, research exploring the relationship, if any, between academic language and conceptual understanding is scant. What little research exists does not include English language learners (ELLs) and/or economically disadvantaged (ED) student samples—students most at risk academically. This study quantitatively determined if there exists a relationship, and if so, how strong of a relationship, between ELL and ED students’ academic language and conceptual understanding based on science notebook scores used in a larger science and literacy-infused intervention with a sample of culturally diverse students. The study also considered strengths of relationships between language and concept science notebook scores within student language status groups (ELL, former ELL, and English speaking). Correlational analyses noted positive, large, and significant correlations between students’ language and concept scores overall, with the largest correlations for science notebook entries using more academic language. Large correlations also existed for ELL student entries at the end of the school year. Implications of the findings for future research and practice in science classrooms including literacy interventions, such as science notebooks, with populations of culturally diverse students are discussed.     

Predictors and Outcomes of Early vs. Later English Language Proficiency Among English Language Learners

The development of English language learners (ELLs) was explored from kindergarten through eighth grade within a nationally representative sample of first-time kindergartners (N = 19,890). Growth curve analyses indicated that, compared to native English speakers, ELLs were rated by teachers more favorably on approaches to learning, self control, and externalizing behaviors in kindergarten and generally continued to grow in a positive direction on these social/behavioral outcomes at a steeper rate compared to their native English-speaking peers, holding other factors constant. Differences in reading and math achievement between ELLs and native English speakers varied based on the grade at which English proficiency is attained. Specifically, ELLs who were proficient in English by kindergarten entry kept pace with native English speakers in both reading and math initially and over time; ELLs who were proficient by first grade had modest gaps in reading and math achievement compared to native English speakers that closed narrowly or persisted over time; and ELLs who were not proficient by first grade had the largest initial gaps in reading and math achievement compared to native speakers but the gap narrowed over time in reading and grew over time in math. Among those whose home language is not English, acquiring English proficiency by kindergarten entry was associated with better cognitive and behavioral outcomes through eighth grade compared to taking longer to achieve proficiency. Multinomial regression analyses indicated that child, family, and school characteristics predict achieving English proficiency by kindergarten entry compared to achieving proficiency later. Results are discussed in terms of policies and practices that can support ELL children's growth and development.     

Predictors and outcomes of early versus later English language proficiency among English language learners

The development of English language learners (ELLs) was explored from kindergarten through eighth grade within a nationally representative sample of first-time kindergartners (N = 19,890). Growth curve analyses indicated that, compared to native English speakers, ELLs were rated by teachers more favorably on approaches to learning, self-control, and externalizing behaviors in kindergarten and generally continued to grow in a positive direction on these social/behavioral outcomes at a steeper rate compared to their native English-speaking peers, holding other factors constant. Differences in reading and math achievement between ELLs and native English speakers varied based on the grade at which English proficiency is attained. Specifically, ELLs who were proficient in English by kindergarten entry kept pace with native English speakers in both reading and math initially and over time; ELLs who were proficient by first grade had modest gaps in reading and math achievement compared to native English speakers that closed narrowly or persisted over time; and ELLs who were not proficient by first grade had the largest initial gaps in reading and math achievement compared to native speakers but the gap narrowed over time in reading and grew over time in math. Among those whose home language is not English, acquiring English proficiency by kindergarten entry was associated with better cognitive and behavioral outcomes through eighth grade compared to taking longer to achieve proficiency. Multinomial regression analyses indicated that child, family, and school characteristics predict achieving English proficiency by kindergarten entry compared to achieving proficiency later. Results are discussed in terms of policies and practices that can support ELL children's growth and development.     

Test fairness: Examining differential functioning of the reading comprehension section of the GSEEE in China

This study investigated differential item functioning (DIF), differential bundle functioning (DBF), and differential test functioning (DTF) across gender of the reading comprehension section of the Graduate School Entrance English Exam in China. The datasets included 10,000 test-takers’ item-level responses to 6 five-item testlets. Both DIF and DBF were examined by using poly-simultaneous item bias test and item-response-theory-likelihood-ratio test, and DTF was investigated with multi-group confirmatory factor analyses (MG-CFA). The results indicated that although none of the 30 items exhibited statistically and practically significant DIF across gender at the item level, 2 testlets were consistently identified as having significant DBF at the testlet level by the two procedures. Nonetheless, DBF does not manifest itself at the overall test score level to produce DTF based on MG-CFA. This suggests that the relationship between item-level DIF and test-level DTF is a complicated issue with the mediating effect of testlets in testlet-based language assessment.     

First Language of Test Takers and Fairness Assessment Procedures

Over the past few decades, those who take tests in the United States have exhibited increasing diversity with respect to native language. Standard psychometric procedures for ensuring item and test fairness that have existed for some time were developed when test‐taking groups were predominantly native English speakers. A better understanding of the potential influence that insufficient language proficiency may have on the efficacy of these procedures is needed. This paper represents a first step in arriving at this better understanding. We begin by addressing some of the issues that arise in a context in which assessments in a language such as English are taken increasingly by groups that may not possess the language proficiency needed to take the test. For illustrative purposes, we use the first‐language status of a test taker as a surrogate for language proficiency and describe an approach to examining how the results of fairness procedures are affected by inclusion or exclusion of those who report that English is not their first language in the fairness analyses. Furthermore, we explore the sensitivity of the results of these procedures, differential item functioning (DIF) and score equating, to potential shifts in population composition. We employ data from a large‐volume testing program for this illustrative purpose. The equating results were not affected by either inclusion or exclusion of such test takers in the analysis sample, or by shifts in population composition. The effect on DIF results, however, varied across focal groups.     

Detection of Gender-Related Differential Item Functioning in a Mathematics Performance Assessment

This study used three different differential item functioning (DIF) detection proce- dures to examine the extent to which items in a mathematics performance assessment functioned differently for matched gender groups. In addition to examining the appropriateness of individual items in terms of DIF with respect to gender, an attempt was made to identify factors (e.g., content, cognitive processes, differences in ability distributions, etc.) that may be related to DIF. The QUASAR (Quantitative Under- standing: Amplifying Student Achievement and Reasoning) Cognitive Assessment Instrument (QCAI) is designed to measure students' mathematical thinking and reasoning skills and consists of open-ended items that require students to show their solution processes and provide explanations for their answers. In this study, 33 polytomously scored items, which were distributed within four test forms, were evaluated with respect to gender-related DIF. The data source was sixth- and seventh- grade student responses to each of the four test forms administrated in the spring of 1992 at all six school sites participatingin the QUASARproject. The sample consisted of 1,782 students with approximately equal numbers of female and male students. The results indicated that DIF may not be serious for 3 1 of the 33 items (94%) in the QCAI. For the two items that were detected as functioning differently for male and female students, several plausible factors for DIF were discussed. The results from the secondary analyses, which removed the mutual influence of the two items, indicated that DIF in one item, PPPl, which favored female students rather than their matched male students, was of particular concern. These secondary analyses suggest that the detection of DIF in the other item in the original analysis may have been due to the influence of Item PPPl because they were both in the same test form.     

Behavioral Assessment of Coping Strategies in Young Children At-risk,Developmentally Delayed,and Typically Developing

Research Findings: This study investigated the effects of full- and half-day kindergarten programs on classroom instructional quality and children's academic achievement. Considerations were given for how the length of the school day, language status (English language learner [ELL] and non-ELL), and children's attendance patterns influenced achievement. Quantitative and qualitative data were collected concurrently and were interpreted to note the convergence (or lack thereof) of the findings. Quantitative results revealed no difference in the quality of instruction being offered in full- and half-day classrooms. Additionally, full-day kindergarten positively impacted children's academic achievement in literacy but not in mathematics, regardless of children's language status. In regard to language development, ELL children benefited more from full-day kindergarten than did their English-speaking peers, whereas all (ELL and non-ELL) children enrolled in full-day kindergarten made greater language gains when they missed fewer than 10 school days. Practice or Policy: Findings from this study have significant policy and practice implications related to the overall quality, availability, and cultural and developmental appropriateness of kindergarten programming in the United States.     

Developing and Validating a Science Notebook Rubric for Fifth-Grade Non-Mainstream Students

We present the development and validation of a science notebook rubric intended to measure the academic language and conceptual understanding of non-mainstream students, specifically fifth-grade male and female economically disadvantaged Hispanic English language learner (ELL) and African-American or Hispanic native English-speaking students. The science notebook rubric is based on two main constructs: academic language and conceptual understanding. The constructs are grounded in second-language acquisition theory and theories of writing and conceptual understanding. We established content validity and calculated reliability measures using G theory and percent agreement (for comparison) with a sample of approximately 144 unique science notebook entries and 432 data points. Results reveal sufficient reliability estimates, indicating that the instrument is promising for use in future research studies including science notebooks in classrooms with populations of economically disadvantaged Hispanic ELL and African-American or Hispanic native English-speaking students.