Measurement properties as a possible cause of digital device effects on a standardized assessment of learning

Recent research found a substantial effect of digital device type on a computerized, standardized assessment of learning. Findings were consistent across grade and content domain. To investigate the source of these differences, the current research probed whether measurement differences or, equivalently, test bias due to device type could explain achievement or proficiency differences. We used 2018–2019 results on the Indiana Learning Evaluation Assessment Readiness Network (ILEARN). The data are census-based and cover grades 3 and 8 in the content domains of mathematics and English language arts (ELA). For our analysis, we use the root mean-squared deviation (RMSD) for detecting measurement differences across students that took ILEARN on different digital devices. Our findings suggest that few, if any, achievement differences that depend on the type of digital device can be explained by test bias. We discuss our findings in the context of multiple-group versus multiple indicator, multiple cause measurement models. Our findings suggest that digital device operates directly on the construct (math or ELA) rather than on the indicators that measure the construct. Finally, we conclude that a competing hypothesis – that differences are due to a digital device familiarity effect – remains plausible.