If you build it,will they come? Lessons learned in recruiting students for randomized controlled trials in postsecondary settings

Randomized controlled trials (RCTs) are considered the most methodologically sound methods of conducting empirical research, but a successful RCT is contingent on the recruitment of subjects into the intervention and study. Few research articles and evaluations in education discuss recruitment, and even fewer discuss challenges in recruitment. This is worrisome, given that recruitment can jeopardize internal and external validity – the most valuable aspects of RCTs. This paper initiates a conversation about planning and carrying out a successful recruitment campaign, specifically in postsecondary settings where students are more autonomous, programmes are likely to be voluntary, the flow of information is disparate, and incentives are often misaligned for external evaluation. We also share our challenges and lessons learned in recruiting students for a student support programme and its evaluation. We also advocate that more researchers devote attention to, and sufficiently plan for the recruitment of subjects into RCTs.     

Reconsidering assent for randomised control trials in education: Ethical and procedural concerns

Despite ongoing disagreement about what kinds of evidence are most valuable to education, there continues to be an increasing push to make a narrow group of scientific methodologies the basis of educational policy and practice. This has created a growth in the use of randomised control trials (RCTs), which are considered an exemplary example of scientifically rigorous research design. Yet despite the increase in both the prevalence and status of RCTs, this article will argue that the ethics informing this research orientation remains underdeveloped, with the specific need for an agenda that grapples with assent in RCTs in both a philosophical and a methodological way. As a corrective to this, we engage with Biesta’s observation of a lack of explicit engagement with the values informing our decisions about the direction of education practice. We begin by examining assent in existing education research literature, focusing on some of the ways that qualitative and praxis-oriented researchers have grappled with the complexity of assent in research projects involving children and young people in schools. We then consider analogous debates concerning assent in fields such as bioethics and biomedicine, given the established nature of bioethics as a discipline. Finally, we turn our attention to developing an agenda that can work towards a consideration of assent in RCTs in education research, with the purpose of making transparent some of the ethical concerns that warrant attention in the design and conduct of RCT research studies in education.     

Do Volunteers in Schools Help Children Learn to Read? A Systematic Review of Randomised Controlled Trials

The aim of unpaid volunteer classroom assistants is to give extra support to children learning to read. The impact of using volunteers to improve children's acquisition of reading skills is unknown. To assess whether volunteers are effective in improving children's reading, we undertook a systematic review of all relevant randomised controlled trials (RCTs). An exhaustive search of all the main electronic databases was carried out (i.e. BEI, PsycInfo, ASSIA, PAIS, SSCI, ERIC, SPECTR, SIGLE). We identified eight experimental studies, of which seven were RCTs. One of the RCTs was excluded because it did not meet the inclusion criteria. One RCT randomised intact classes and the other six studies randomised individual children and could therefore be included in a meta-analysis. All of the trials were fairly small, with the largest including 99 pupils. Four of the trials showed a positive outcome, while three showed a negative effect and the remaining study was equivocal. We pooled the four most homogeneous trials. The pooled data indicated an effect size of 0.19, which was not statistically significant ( p = 0.54, 95% confidence interval = -0.31 to 0.68). Overall, volunteering appeared to have a small effect on reading outcomes. However, the confidence intervals were wide, which could conceal a potentially large benefit or a harmful effect. Thus, more good quality RCTs are required in order to provide more conclusive evidence.     

Experimental control and threats to internal validity of concurrent and nonconcurrent multiple baseline designs

Single‐case research designs are often applied within school psychology. This article provides a critical review of the scientific merit of both concurrent and nonconcurrent multiple baseline (MB) designs, relative to their capacity to assess threats of internal validity and establish experimental control. Distinctions are established between AB replications and nonconcurrent multiple baseline designed studies using the initial conception proposed by P.J. Watson and E.A. Workman (1981). Despite some previously pessimistic evaluations of nonconcurrent multiple baseline designs, the findings of this review suggest that various threats of internal validity can be assessed and ruled out using either concurrent or nonconcurrent MB designs. It seems that nonconcurrent designs can be used to assess the intervening effects of history, but might be more prone to threats of mortality. These and other threats of internal validity are reviewed and recommendations are provided. © 2007 Wiley Periodicals, Inc. Psychol Schs 44: 451–459, 2007.     

What is meant by “rigour” in evidence-based educational policy and what’s so good about it?

ABSTRACT

Across the evidence-based policy and practice (EBPP) community, including education, randomised controlled trials (RCTS) rank as the most “rigorous” evidence for causal conclusions. This paper argues that that is misleading. Only narrow conclusions about study populations can be warranted with the kind of “rigour” that RCTs excel at. Educators need a great deal more information to predict if a programme will work for their pupils. It is unlikely that that information can be obtained with EBPP-style rigour. So, educators should not be overly optimistic about success with programmes that have been “rigorously” tested. I close with a plea to the EBPP community to take on the job of identifying and vetting the information educators need in practice.     

The Design and Conduct of Randomised Controlled Trials in Education: Lessons from health care

There are increasing calls within the educational research community to undertake more randomised controlled trials (RCTs). Whilst educational researchers turned away from using the RCT in the last 30 years, health care researchers have adopted the method with vigour. However, there have been many methodological innovations of the RCT, developed by health services researchers, in recent years. Many of these innovations are directly applicable to educational research. In this paper we describe some of the important aspects of trial methodology that educational researchers ought to consider when they design and undertake trials. Specifically, we describe the importance of systematic reviews, the need for a trial protocol, sample size requirements, outcome measures, concealment of the randomisation process from the investigators, intention to treat analysis, and the metric--numbers needed to teach (NNT). Addition ally, we argue that there are considerable similarities between health and education allowing the use of RCT methodological innovations to be directly utilised in educational research.     

Practitioner tales: possible roles for research evidence in practice

ABSTRACT

This paper discusses, compares, and contrasts 4 different models for bringing evidence from randomised controlled trials (RCTs) into practice and into practical reasoning. I look at what questions the models can and cannot answer, what role they accord to RCT evidence, and what their possible attraction for practitioners might be. The models are those of Philip Davies, Nancy Cartwright, Stephen Toulmin, and Harald Grimen. The first 2 are constructed for the purpose of using evidence in practice, the 3rd is an adapted argumentation model, and the 4th is based on phronesis and represents an alternative to evidence-based education. It is argued that of these, Toulmin’s model has the narrative qualities that make it the most usable and practitioner-friendly model.     

The key role of representativeness in evidence-based education

ABSTRACT

Within evidence-based education, results from randomised controlled trials (RCTs), and meta-analyses of them, are taken as reliable evidence for effectiveness – they speak to “what works”. Extending RCT results requires establishing that study samples and settings are representative of the intended target. Although widely recognised as important for drawing causal inferences from RCTs, claims regarding representativeness tend to be poorly evidenced. Strategies for demonstrating it typically involve comparing observable characteristics (e.g., race, gender, location) of study samples to those in the population of interest to decision makers. This paper argues that these strategies provide insufficient evidence for establishing representativeness. Characteristics typically used for comparison are unlikely to be causally relevant to all educational interventions. Treating them as evidence that supports extending RCT results without providing evidence demonstrating their relevance undermines the inference. Determining what factors are causally relevant requires studying the causal mechanisms underlying the interventions in question.     

Reassessing RCTs as the ‘gold standard’: synergy not separatism in evaluation designs

ABSTRACT

Randomised controlled trials (RCTs) are increasingly used to evaluate educational interventions in the UK. However, RCTs remain controversial for some elements of the research community. This paper argues that the widespread use of the term ‘gold standard’ to describe RCTs is problematic, as it implies that other research methods are inferior. The usefulness of RCTs can be greatly enhanced when used in conjunction with implementation-specific measures (e.g. observation tools, attitude/engagement surveys and interviews). The proposal is advanced through case studies of two evaluations. One relates to the development of science subject leader skills and expertise at primary school level and the other to co-operative learning of primary maths. Both evaluations randomised schools to the intervention or the business-as-usual control, and compared impact using subject knowledge tests. Integral to each study was a process evaluation which looked at evidence from classroom practice along with feedback from the teachers and pupils themselves. We contend that this enabled much more holistic and richly interpretative pieces of research. The paper concludes that privilege for particular paradigms should be set aside when designing effective evaluations of educational interventions, and that it is insufficient to ask ‘what works?’ without also asking ‘why?’, ‘where?’ and ‘how?’.     

Implementing randomised control trials in open and distance learning: a feasibility study

Randomised control trials (RCTs) are an evidence-based research approach which has not yet been adopted and widely used in open and distance education to inform educational policy and practice. Despite the challenges entailed in their application, RCTs hold the power to robustly evaluate the effects of educational interventions in distance learning and conclude on whether (or not) these interventions should be adopted and used extensively. The aim of this paper is to spark discussions around the use of RCTs in distance learning by illustrating their benefits and drawbacks including challenges in adopting RCTs in education. To achieve this aim, a RCT was implemented to examine whether a small-scale intervention in four language modules could improve attendance at an end-of-module speaking assessment, and in consequence, performance, completion and pass rates. Results raise the need for further research in order to identify what type of interventions should be designed and put into practice to elicit a positive impact on learners. The paper concludes with a discussion on why RCTs should be brought to the forefront as a viable method for the effective evaluation of the impact of open learning analytic interventions.     

The significance of a process evaluation in interpreting the validity of an RCT evaluation of a complex teaching intervention: the case of Integrated Group Reading (IGR) as a targeted intervention for delayed Year 2 and 3 pupils

The significance of a process evaluation for understanding randomised controlled trials (RCTs) of complex teaching interventions in a classroom setting is discussed in relation to the evaluation of the Integrated Group Reading (IGR) programme trial. This process evaluation is based on realist principles and draws on different data sources. The findings indicate that the efficacy of a classroom-based complex intervention like IGR is hard to evaluate given the nature of the intervention and its context. These findings illustrate that a process evaluation should not be seen as a subsidiary study to the experimental trial, but as an independent study in its own right as it can shed light onto the setting up and running of the trial. The paper argues that RCTs by themselves do not necessarily provide the ‘best evidence’ and need to be supplemented by other evaluation perspectives.     

Translating Laboratory Evidence into Classroom Practice with Teacher‐Led Randomized Controlled Trials—A Perspective and Meta‐Analysis

We initiated and structured a single program that supervised teachers, some with neuroscience or psychology degrees, to collaborate and explore the effects of science of learning‐translated pedagogy. This article reports on the 34 findings from teacher‐led randomized controlled trials (RCTs) and replications. Teachers designed trials, looking at areas such as attention, memory, and spaced learning. Overall, positive effects were found over short periods (1–6 weeks; r = 0.15, p < .0001 [d = 0.30], N = 2,157). However, retrieval practice (testing as a learning experience) had differential effects mediated by age, approach, and lesson content. Results suggest science of learning‐translated pedagogy needs extensive replication to establish how best to use laboratory evidence in classrooms. Multiple planned replication of teacher‐led RCTs has potential as an evaluation tool, combining high levels of mundane realism with strong internal validity and the potential to build cost effective large samples for meta‐analysis.     

Evidence of confusion about evidence of causes: comments on the debate about EBP in education

ABSTRACT

EBP – evidence-based policy and practice – has generated intense controversy. A rough continuum of positions can be discerned: At one pole are “tough-minded” commentators distinguished by their support of EBP; however, there are serious internal differences in this camp, for some regard randomised field trials (RFTs) as the gold standard methodology that should generate the evidence required in EBP; while other “softer” commentators have a more nuanced position wherein the RFT is an INUS factor – part of a “causal cake” which requires many other support factors to be present before any generalisation can be made about a policy. The evolution of this softer branch of the tough-minded pole is traced, from Campbell and Stanley, to Cronbach and Associates, to Cartwright and Hardie. At the other pole are the tender-minded, many of whom are members of the philosophy of education community; they are sceptical about EBP, perhaps the most serious of their criticisms being that educational processes are not apt for causal investigation at all. The arguments that are offered in support of this position are judged to be deficient.     

A systematic review and meta‐analysis of the effectiveness of information and communication technology (ICT) on the teaching of spelling

Recent Government policy in England and Wales on Information and Communication Technology (ICT) in schools is heavily influenced by a series of non‐randomised controlled studies. The evidence from these evaluations is equivocal with respect to the effect of ICT on literacy. In order to ascertain whether there is any effect of ICT on one small area of literacy, spelling, a systematic review of all randomised controlled trials (RCTs) was undertaken. Relevant electronic databases (including BEI, ERIC, Web of Science, PsycINFO, The Cochrane Library) were searched. Seven relevant RCTs were identified and included in the review. When six of the seven studies were pooled in a meta‐analysis there was an effect, not statistically significant, in favour of computer interventions (Effect size = 0.37, 95% confidence interval =–0.02 to 0.77, p = 0.06). Sensitivity and sub‐group analyses of the results did not materially alter findings. This review suggests that the teaching of spelling by using computer software may be as effective as conventional teaching of spelling, although the possibility of computer‐taught spelling being inferior or superior cannot be confidently excluded due to the relatively small sample sizes of the identified studies. Ideally, large pragmatic randomised controlled trials need to be undertaken.     

Methodological Issues in Questionnaire-Based Research on Student Learning in Higher Education

Students' scores on questionnaires concerning their approaches to studying in higher education exhibit reasonable stability over time, moderate convergent validity with their scores on other questionnaires, and reasonable levels of discriminating power and criterion-related validity. Nevertheless, the internal consistency of the constituent scales and the construct validity of these instruments are variable, their content validity within contemporary higher education is open to question, and their wording may need to be revised when they are used with students from different social or cultural groups. Future research should investigate the possibility of response bias in such instruments and the validity of self-reports concerning study behavior.     

Experimental trials and ‘what works?’ in education: The case of grammar for writing

The place of evidence to inform educational effectiveness has received increasing attention internationally in the last two decades. An important contribution to evidence‐informed policy has been greater attention to experimental trials including randomised controlled trials (RCTs). The aim of this paper is to examine the use of evidence, particularly the use of evidence from experimental trials, to inform national curriculum policy. To do this the teaching of grammar to help pupils’ writing was selected as a case. Two well‐regarded and influential experimental trials that had a significant effect on policy, and that focused on the effectiveness of grammar teaching to support pupils’ writing, are examined in detail. In addition to the analysis of their methodology, the nature of the two trials is also considered in relation to other key studies in the field of grammar teaching for writing and a recently published robust RCT. The paper shows a significant and persistent mismatch between national curriculum policy in England and the robust evidence that is available with regard to the teaching of writing. It is concluded that there is a need for better evidence‐informed decisions by policy makers to ensure a national curriculum specification for writing that is more likely to have positive impact on pupils.     

用SWOT分析、研究我国高等教育自学考试发展战略

在当前各种教育考试形式迅速发展的形势下,我国高等教育自学考试受到了严峻的挑战;用SWOT分析对我国高等教育自学考试面临外部环境的机会与威胁,内部条件的优势与劣势进行综合分析研究,从而制定新的发展战略。     

Lost in translation: methodological considerations in cross-cultural research

In cross-cultural child development research there is often a need to translate instruments and instructions to languages other than English. Typically, the translation process focuses on ensuring linguistic equivalence. However, establishment of linguistic equivalence through translation techniques is often not sufficient to guard against validity threats. In addition to linguistic equivalence, functional equivalence, cultural equivalence, and metric equivalence are factors that need to be considered when research methods are translated to other languages. This article first examines cross-cultural threats to validity in research. Next, each of the preceding factors is illustrated with examples from the literature. Finally, suggestions for incorporating each factor into research studies of child development are given.