Using causal models to bridge the divide between big data and educational theory

An extraordinary amount of data is becoming available in educational settings, collected from a wide range of Educational Technology tools and services. This creates opportunities for using methods from Artificial Intelligence and Learning Analytics (LA) to improve learning and the environments in which it occurs. And yet, analytics results produced using these methods often fail to link to theoretical concepts from the learning sciences, making them difficult for educators to trust, interpret and act upon. At the same time, many of our educational theories are difficult to formalise into testable models that link to educational data. New methodologies are required to formalise the bridge between big data and educational theory. This paper demonstrates how causal modelling can help to close this gap. It introduces the apparatus of causal modelling, and shows how it can be applied to well-known problems in LA to yield new insights. We conclude with a consideration of what causal modelling adds to the theory-versus-data debate in education, and extend an invitation to other investigators to join this exciting programme of research.

Practitioner notes

What is already known about this topic

• ‘Correlation does not equal causation’ is a familiar claim in many fields of research but increasingly we see the need for a causal understanding of our educational systems.
• Big data bring many opportunities for analysis in education, but also a risk that results will fail to replicate in new contexts.
• Causal inference is a well-developed approach for extracting causal relationships from data, but is yet to become widely used in the learning sciences.

What this paper adds

• An overview of causal modelling to support educational data scientists interested in adopting this promising approach.
• A demonstration of how constructing causal models forces us to more explicitly specify the claims of educational theories.
• An understanding of how we can link educational datasets to theoretical constructs represented as causal models so formulating empirical tests of the educational theories that they represent.

Implications for practice and/or policy

• Causal models can help us to explicitly specify educational theories in a testable format.
• It is sometimes possible to make causal inferences from educational data if we understand our system well enough to construct a sufficiently explicit theoretical model.
• Learning Analysts should work to specify more causal models and test their predictions, as this would advance our theoretical understanding of many educational systems.

     

Implementation Factors and Their Influence on Student Mathematics Outcomes

Assessing implementation allows for a better understanding of an intervention's effects and the mechanisms that influence its impact. Two main areas of implementation are (a) the quality with which an intervention is delivered and (b) instructors’ adherence to the programmed intervention. The current study used data from a kindergarten mathematics intervention program to (a) examine if and how treatment adherence was associated with implementation quality and (b) explore implementation measures’ relation to student mathematics outcomes. Results indicated high implementation scores across time for both adherence and quality. Neither treatment adherence nor implementation quality was found to relate to a general outcome measure of student mathematics achievement; however, both were similarly related to the curricular-aligned measure.