Getting ‘What Works’ working: building blocks for the integration of experimental and improvement science

ABSTRACT

As a systemic approach to improving educational practice through research, ‘What Works’ has come under repeated challenge from alternative approaches, most recently that of improvement science. While ‘What Works’ remains a dominant paradigm for centralized knowledge-building efforts, there is need to understand why this alternative has gained support, and what it can contribute. I set out how the core elements of experimental and improvement science can be combined into a strategy to raise educational achievement with the support of evidence from randomized experiments. Central to this combined effort is a focus on identifying and testing mechanisms for improving teaching and learning, as applications of principles from the learning sciences. This article builds on current efforts to strengthen approaches to evidence-based practice and policy in a range of international contexts. It provides a foundation for those who aim to avoid another paradigm war and to accelerate international discussions on the design of systemic education research infrastructure and funding.     

What is meant by “rigour” in evidence-based educational policy and what’s so good about it?

ABSTRACT

Across the evidence-based policy and practice (EBPP) community, including education, randomised controlled trials (RCTS) rank as the most “rigorous” evidence for causal conclusions. This paper argues that that is misleading. Only narrow conclusions about study populations can be warranted with the kind of “rigour” that RCTs excel at. Educators need a great deal more information to predict if a programme will work for their pupils. It is unlikely that that information can be obtained with EBPP-style rigour. So, educators should not be overly optimistic about success with programmes that have been “rigorously” tested. I close with a plea to the EBPP community to take on the job of identifying and vetting the information educators need in practice.     

The key role of representativeness in evidence-based education

ABSTRACT

Within evidence-based education, results from randomised controlled trials (RCTs), and meta-analyses of them, are taken as reliable evidence for effectiveness – they speak to “what works”. Extending RCT results requires establishing that study samples and settings are representative of the intended target. Although widely recognised as important for drawing causal inferences from RCTs, claims regarding representativeness tend to be poorly evidenced. Strategies for demonstrating it typically involve comparing observable characteristics (e.g., race, gender, location) of study samples to those in the population of interest to decision makers. This paper argues that these strategies provide insufficient evidence for establishing representativeness. Characteristics typically used for comparison are unlikely to be causally relevant to all educational interventions. Treating them as evidence that supports extending RCT results without providing evidence demonstrating their relevance undermines the inference. Determining what factors are causally relevant requires studying the causal mechanisms underlying the interventions in question.