The key role of representativeness in evidence-based education

ABSTRACT

Within evidence-based education, results from randomised controlled trials (RCTs), and meta-analyses of them, are taken as reliable evidence for effectiveness – they speak to “what works”. Extending RCT results requires establishing that study samples and settings are representative of the intended target. Although widely recognised as important for drawing causal inferences from RCTs, claims regarding representativeness tend to be poorly evidenced. Strategies for demonstrating it typically involve comparing observable characteristics (e.g., race, gender, location) of study samples to those in the population of interest to decision makers. This paper argues that these strategies provide insufficient evidence for establishing representativeness. Characteristics typically used for comparison are unlikely to be causally relevant to all educational interventions. Treating them as evidence that supports extending RCT results without providing evidence demonstrating their relevance undermines the inference. Determining what factors are causally relevant requires studying the causal mechanisms underlying the interventions in question.     

The Design and Conduct of Randomised Controlled Trials in Education: Lessons from health care

There are increasing calls within the educational research community to undertake more randomised controlled trials (RCTs). Whilst educational researchers turned away from using the RCT in the last 30 years, health care researchers have adopted the method with vigour. However, there have been many methodological innovations of the RCT, developed by health services researchers, in recent years. Many of these innovations are directly applicable to educational research. In this paper we describe some of the important aspects of trial methodology that educational researchers ought to consider when they design and undertake trials. Specifically, we describe the importance of systematic reviews, the need for a trial protocol, sample size requirements, outcome measures, concealment of the randomisation process from the investigators, intention to treat analysis, and the metric--numbers needed to teach (NNT). Addition ally, we argue that there are considerable similarities between health and education allowing the use of RCT methodological innovations to be directly utilised in educational research.     

The Need for Randomised Controlled Trials in Educational Research

This paper argues for more randomised controlled trials in educational research. Educational researchers have largely abandoned the methodology they helped to pioneer. This gold-standard methodology should be more widely used as it is an appropriate and robust research technique. Without subjecting curriculum innovations to a RCT then potentially harmful educational initiatives could be visited upon the nation's children.     

A systematic review of controlled trials evaluating interventions in adult literacy and numeracy

This paper reports a systematic review of the quasi‐experimental literature in the field of adult literacy and numeracy, published between 1980 and 2002. We included 27 controlled trials (CTs) that evaluated strategies and pedagogies designed to increase adult literacy and numeracy: 18 CTs with no effect sizes (incomplete data) and 9 CTs with full data. These nine trials are examined in detail for this paper. Of these nine trials, six evaluated interventions in literacy and three evaluated interventions in literacy and numeracy. Three of the nine trials showed a positive effect for the interventions, five trials showed no difference and one trial showed a positive effect for the control treatment. The quality of the trials was variable, but many of them had some methodological problems. There was no evidence of publication bias in the review. There have been few attempts to expose common adult literacy or numeracy programmes to rigorous evaluation and therefore in terms of policy and practice it is difficult to make any recommendations as to the type of adult education that should be supported. In contrast, however, the review does provide a strong steer for the direction to be taken by educational researchers: because of the present inadequate evidence base rigorously designed randomised controlled trials and quasi‐experiments are required as a matter of urgency.     

Evidence‐based practice in educational research: a critical realist critique of systematic review

The paper argues that a critical realist perspective can contribute to a critique of evidence‐based practice, while at the same time not abandoning the idea of evidence altogether. The paper is structured around a number of related themes: the sociopolitics of ‘evidence‐based’; epistemological roots and a critical realist critique; the debate in action based on the recent systematic review of personal development planning; and theory to practice gaps. The advocacy of evidence‐based practice is currently being used to undermine professional autonomy and to valorise the ‘gold‐standard’ of randomised controlled trials. However, the paper proposes that evidence can properly be claimed for critique and emancipatory projects, and that its current discursive location at the core of New Labour thinking is not the only one available. Moreover, thinking from a critical realist perspective liberates the space for theoretically informed work, whereby arguments about method, and in particular randomised controlled trials, do not become a proxy for the open examination of ontological and epistemological assumptions. One of the underlying themes in critical realism is that of critique and emancipation, and evidence properly understood aligned to clear theoretical argumentation is part of this project. Existing models of systematic review fall far short of this aspiration and are of little help to practitioners. As researchers and practitioners we have every reason to maintain a critical stance towards the way evidence is being deployed in debates about policy and practice.     

Unmasking the unasked: correcting the record about assessor masking as an explanation for effect size differences

Ainsworth et al.’s paper “Sources of Bias in Outcome Assessment in Randomised Controlled Trials: A Case Study” examines alternative accounts for a large difference in effect size between 2 outcomes in the same intervention evaluation. It argues that the probable explanation relates to masking: Only one outcome measure was administered by those aware of participants’ treatment assignment. This paper shows this conclusion is not substantiated by the evidence: The original paper fails to exclude alternative explanations, and what it takes as positive evidence for the preferred explanation is actually negative. While accepting the importance of masking in randomised controlled trials, this paper concludes that the original question was based on a misconception about effect sizes: Seen correctly as a measure of whole study design, the question of effect size difference between different outcome measures does not need asking.     

Evidence-based teaching: a simple view of “science”

ABSTRACT

This paper examines the insistent claims by advocates of evidence-based teaching that it is a rigorous scientific approach. The paper questions the view that randomised controlled trials and meta-analyses are the only truly scientific methods in educational research. It suggests these claims are often based on a rhetorical appeal which relies on too simple a notion of “science”. Exploring the tacit assumptions behind “evidence-based teaching”, the paper identifies an empiricist and reductionist philosophy of science, and a failure to recognise the complexity of education and pedagogy. Following a discussion of large-scale syntheses of evidence (Hattie’s Visible Learning; the Education Endowment Foundation’s Teaching and Learning Toolkit), it examines in detail one strand of the latter concerning sports participation, which is used to illustrate flaws in procedures and the failure to take seriously the need for causal explanations.     

What is meant by “rigour” in evidence-based educational policy and what’s so good about it?

ABSTRACT

Across the evidence-based policy and practice (EBPP) community, including education, randomised controlled trials (RCTS) rank as the most “rigorous” evidence for causal conclusions. This paper argues that that is misleading. Only narrow conclusions about study populations can be warranted with the kind of “rigour” that RCTs excel at. Educators need a great deal more information to predict if a programme will work for their pupils. It is unlikely that that information can be obtained with EBPP-style rigour. So, educators should not be overly optimistic about success with programmes that have been “rigorously” tested. I close with a plea to the EBPP community to take on the job of identifying and vetting the information educators need in practice.     

Experimental trials and ‘what works?’ in education: The case of grammar for writing

The place of evidence to inform educational effectiveness has received increasing attention internationally in the last two decades. An important contribution to evidence‐informed policy has been greater attention to experimental trials including randomised controlled trials (RCTs). The aim of this paper is to examine the use of evidence, particularly the use of evidence from experimental trials, to inform national curriculum policy. To do this the teaching of grammar to help pupils’ writing was selected as a case. Two well‐regarded and influential experimental trials that had a significant effect on policy, and that focused on the effectiveness of grammar teaching to support pupils’ writing, are examined in detail. In addition to the analysis of their methodology, the nature of the two trials is also considered in relation to other key studies in the field of grammar teaching for writing and a recently published robust RCT. The paper shows a significant and persistent mismatch between national curriculum policy in England and the robust evidence that is available with regard to the teaching of writing. It is concluded that there is a need for better evidence‐informed decisions by policy makers to ensure a national curriculum specification for writing that is more likely to have positive impact on pupils.     

Structural Equations and Causal Explanations: Some Challenges for Causal SEM

One common application of structural equation modeling (SEM) involves expressing and empirically investigating causal explanations. Nonetheless, several aspects of causal explanation that have an impact on behavioral science methodology remain poorly understood. It remains unclear whether applications of SEM should attempt to provide complete explanations or partial explanations. Moreover, it remains unclear what sorts of things researchers can best take as causes and effects. Finally, the meaning of causal assertions itself remains poorly understood. Attempting to clarify the use of structural equations as causal explanations by addressing these issues has implications for behavioral science methodology because applications of SEM typically remain vague about causation and thus about their substantive conclusions. Research aimed at clarifying these issues can lead to a sharper and more refined use of SEM for causal explanation, and by extension, clarify behavioral science methodology more generally.     

Avoiding Bias in Randomised Controlled Trials in Educational Research

Randomised controlled trials (RCTs) are often seen as the 'gold standard' of evaluative research. However, whilst randomisation will ensure comparable groups, trials are still vulnerable to a range of biases that can undermine their internal validity. In this paper we describe a number of common threats to the internal validity of RCTs and methods of countering them. We highlight a number of examples from randomised trials in education and health care where problems of execution and analysis of the RCT has undermined their internal validity. However, awareness of these potential biases can lead to careful planning to avoid or reduce their occurrence. If good quality randomised trials are to inform policy and practice in education then rigorous trials need to be designed that are the least susceptible to threats to their validity.     

A systematic review and meta‐analysis of the effectiveness of information and communication technology (ICT) on the teaching of spelling

Recent Government policy in England and Wales on Information and Communication Technology (ICT) in schools is heavily influenced by a series of non‐randomised controlled studies. The evidence from these evaluations is equivocal with respect to the effect of ICT on literacy. In order to ascertain whether there is any effect of ICT on one small area of literacy, spelling, a systematic review of all randomised controlled trials (RCTs) was undertaken. Relevant electronic databases (including BEI, ERIC, Web of Science, PsycINFO, The Cochrane Library) were searched. Seven relevant RCTs were identified and included in the review. When six of the seven studies were pooled in a meta‐analysis there was an effect, not statistically significant, in favour of computer interventions (Effect size = 0.37, 95% confidence interval =–0.02 to 0.77, p = 0.06). Sensitivity and sub‐group analyses of the results did not materially alter findings. This review suggests that the teaching of spelling by using computer software may be as effective as conventional teaching of spelling, although the possibility of computer‐taught spelling being inferior or superior cannot be confidently excluded due to the relatively small sample sizes of the identified studies. Ideally, large pragmatic randomised controlled trials need to be undertaken.     

Critical realist review: exploring the real,beyond the empirical

This article defines the ‘critical realist review’, a literature-based methodological approach to critical analysis of health care studies (or any discipline charged with social interventions) that is robust, insightful and essential for the complexities of twenty-first century evidence-based health and social care. We argue that this approach, underpinned by a critical realist philosophy and methodology, will facilitate students and researchers to employ relevant theoretical insights from a range of disciplines that have necessary contributions to make to our understanding of health and social care practice and provision. We explore the limitations of randomised controlled trials (RCTs) and the systematic review for informing evidence-based health and social care. The article also offers some suggestions on method, although, as we argue, the approach deliberately eschews a ‘cookbook’ approach. A glossary of terms is provided in the Appendix.     

The significance of a process evaluation in interpreting the validity of an RCT evaluation of a complex teaching intervention: the case of Integrated Group Reading (IGR) as a targeted intervention for delayed Year 2 and 3 pupils

The significance of a process evaluation for understanding randomised controlled trials (RCTs) of complex teaching interventions in a classroom setting is discussed in relation to the evaluation of the Integrated Group Reading (IGR) programme trial. This process evaluation is based on realist principles and draws on different data sources. The findings indicate that the efficacy of a classroom-based complex intervention like IGR is hard to evaluate given the nature of the intervention and its context. These findings illustrate that a process evaluation should not be seen as a subsidiary study to the experimental trial, but as an independent study in its own right as it can shed light onto the setting up and running of the trial. The paper argues that RCTs by themselves do not necessarily provide the ‘best evidence’ and need to be supplemented by other evaluation perspectives.     

Human capital theory,the diversity of societies and the problem of quality in education

Taking off from the “human capital theory” vs. “screening theory” debate, the paper seeks to distinguish at least ten different causal strands which potentially enter the explanation of the commonly observed correlation between earnings and educational qualification. Debates on this issue are commonly vitiated by two factors: (i) the failure to distinguish between mechanisms which operate in fluid market situations and those which operate in bureaucratized economies with administered careers, (ii) the assumption that there can be a universally valid answer to the question “Why do earnings and certificates correlate?” Various institutional and historical factors cause considerable differences in the relative importance of the ten linking mechanisms in different countries. Of the greatest importance is one general difference between early and late developers. Schools in developing countries are generally less effective in fostering the sort of mental attitudes and abilities which conduce to economic development chiefly because, for a variety of reasons, schooling is more examination-oriented than in richer countries. These crucial aspects of the quality of education are too often overlooked, presumably for reasons which have to do with the nature of the economics and educational professions.     

Are Attitudes Only of Theoretical Interest?

The concept attitude is often used in general mass media debate as a causal, explaining factor. In contrast, the concept has hardly ever appeared in the headlines of Swedish scientific reports in pedagogics during the last few years. A possible explanation may be the pessimistic opinion about the usefulness of attitude research expressed by researchers in adult education. Studies lending support to this defeatism are critically examined from a methodological point of view without finding support for the general conclusion that the attitude concept is of low interest, as an explanatory and a predicting construct.     

详析模式方法及其在教育科学研究中的应用

详析模式是一种因果分析的多变量检验模型,通过引入检验变量来辨别、阐明和揭示原变量问的真实性因果关系。在教育研究中运用详析模式方法,能够检验教育变量间因果关系的真假、揭示变量间相互关系的县体条件、挖掘教育变量间的潜在关系,从而有效避免由对教育调查资料的感性认识而得出错误结论,有助于抓住教育问题的实质,为正确地进行教育决策提供切实可靠的依据。     

Do Volunteers in Schools Help Children Learn to Read? A Systematic Review of Randomised Controlled Trials

The aim of unpaid volunteer classroom assistants is to give extra support to children learning to read. The impact of using volunteers to improve children's acquisition of reading skills is unknown. To assess whether volunteers are effective in improving children's reading, we undertook a systematic review of all relevant randomised controlled trials (RCTs). An exhaustive search of all the main electronic databases was carried out (i.e. BEI, PsycInfo, ASSIA, PAIS, SSCI, ERIC, SPECTR, SIGLE). We identified eight experimental studies, of which seven were RCTs. One of the RCTs was excluded because it did not meet the inclusion criteria. One RCT randomised intact classes and the other six studies randomised individual children and could therefore be included in a meta-analysis. All of the trials were fairly small, with the largest including 99 pupils. Four of the trials showed a positive outcome, while three showed a negative effect and the remaining study was equivocal. We pooled the four most homogeneous trials. The pooled data indicated an effect size of 0.19, which was not statistically significant ( p = 0.54, 95% confidence interval = -0.31 to 0.68). Overall, volunteering appeared to have a small effect on reading outcomes. However, the confidence intervals were wide, which could conceal a potentially large benefit or a harmful effect. Thus, more good quality RCTs are required in order to provide more conclusive evidence.