A comparison of randomised controlled trials in health and education

Health care and educational trials face similar methodological challenges. Methodological reviews of health care trials have shown that a significant proportion have methodological flaws. Whether or not educational trials have a similar proportion of poor‐quality trials is unknown. The authors undertook a methodological comparison between health care and educational trials published since 1990. The authors aimed (1) to assess whether the quality of trial reports in education and health care are similar; and (2) to assess whether trial reporting quality is improving. The characteristics of a sample of trials, published since 1990, were taken from health and educational journals. Trials were assessed using the following quality criteria: rationale for sample size; concealment of allocation; blinded follow‐up; use of confidence intervals; adequate sample size. Ninety‐six placebo drug trials and 54 non‐drug trials published in major general journals were identified. These were compared with 54 trials in specialist health journals and 84 trials in educational journals. No educational trial used concealed allocation or reported the rationale for sample size calculation and only one trial used confidence intervals. There was a trend for the reporting of health care trials to improve with time, whilst the reporting quality of educational trials declined. The authors concluded that poor quality of trial reporting is more prevalent in educational journals than in health care journals.     

Reconsidering assent for randomised control trials in education: Ethical and procedural concerns

Despite ongoing disagreement about what kinds of evidence are most valuable to education, there continues to be an increasing push to make a narrow group of scientific methodologies the basis of educational policy and practice. This has created a growth in the use of randomised control trials (RCTs), which are considered an exemplary example of scientifically rigorous research design. Yet despite the increase in both the prevalence and status of RCTs, this article will argue that the ethics informing this research orientation remains underdeveloped, with the specific need for an agenda that grapples with assent in RCTs in both a philosophical and a methodological way. As a corrective to this, we engage with Biesta’s observation of a lack of explicit engagement with the values informing our decisions about the direction of education practice. We begin by examining assent in existing education research literature, focusing on some of the ways that qualitative and praxis-oriented researchers have grappled with the complexity of assent in research projects involving children and young people in schools. We then consider analogous debates concerning assent in fields such as bioethics and biomedicine, given the established nature of bioethics as a discipline. Finally, we turn our attention to developing an agenda that can work towards a consideration of assent in RCTs in education research, with the purpose of making transparent some of the ethical concerns that warrant attention in the design and conduct of RCT research studies in education.     

On the feasibility of conducting randomised trials in education: case study of a sex education intervention

Abstract This article reports on the conduct and results of a randomised controlled trial to test the effectiveness of a teacher‐led intervention to improve teenagers' knowledge of emergency contraception. The trial was successfully conducted in 24 mixed‐sex state secondary schools in Avon, South‐west England. The intervention was popular with both teachers and pupils, and significantly improved pupils' knowledge of emergency contraception. The article uses the emergency contraception trial to demonstrate the feasibility of conducting large‐scale randomised trials in schools. The article focuses particularly on study design, specification of research question, recruitment of schools and method of randomisation to highlight methodological choices that can maximise the feasibility and value of such trials to answer questions regarding the effectiveness of complex educational interventions. The article also highlights the importance of using theory and mixed research methods both to thoroughly develop an intervention before it is tested in a trial, and also to evaluate factors affecting its success within a trial.     

The effect of ability matching and differentiated instruction in financial literacy education. Evidence from two randomised control trials

The present paper examines the impact of ability matching and differentiated instruction on the learning outcomes of eighth and ninth grade students in a financial education programme. In particular, the effect of within-class ability matching is investigated by forming pairs of students either randomly or based on their abilities. In addition, the paper studies whether differentiated instruction, in the form of additional instructions for lower ability students, enhances the impact of the financial education programme. The paper provides evidence on the effects of both practices using two randomised control trials involving 65 schools and 2,407 students. Overall, the results suggest that the programme raises the financial proficiency of students by 0.18 standard deviations. Although the learning outcomes of the average student are not affected by the differentiation practices, non-native students significantly benefit from differentiated instruction.     

Webinars in higher education and professional training: A meta-analysis and systematic review of randomized controlled trials

Digital learning environments are increasingly popular in higher education and professional training. Teaching and learning via webinars, and web conferencing more broadly, represents one widely used approach. Webinars are defined as web-based seminars, in which participants and facilitators communicate live over the Internet across distant geographical locations using shared virtual platforms and interact ubiquitously and synchronously in real time via voice over IP technology and web camera equipment. In the past, studies have reported mixed evidence concerning the effectiveness of webinars in promoting student achievement. As a remedy, this systematic literature review and meta-analysis cumulates observed effect sizes from previously published randomized controlled trials and corrects artifactual variance induced by sampling error. The research questions were: How effective are webinars in promoting student achievement? And which characteristics moderate webinar effectiveness? The findings suggest that webinars were slightly more effective than control conditions (online asynchronous learning management systems and offline face-to-face classroom instruction), but the differences were trivial in size. Differences were moderated by webinar, participant, achievement, and publication characteristics. This meta-analysis is the first to systematically review and meta-analyze the best evidence available for evaluating the effectiveness of webinars and video conferences in promoting student knowledge and skills. The implications of the study's findings can inform school teachers, lecturers, trainers, technologists, and theorists interested in the computer-supported design, implementation, delivery, tutoring, and assessment of webinar-based learning environments.     

The importance of specifying and studying causal mechanisms in school-based randomised controlled trials: lessons from two studies of cross-age peer tutoring

Based on the experience of evaluating 2 cross-age peer-tutoring interventions, we argue that researchers need to pay greater attention to causal mechanisms within the context of school-based randomised controlled trials. Without studying mechanisms, researchers are less able to explain the underlying causal processes that give rise to results from randomised controlled trials. Studying implementation fidelity is necessary but not sufficient for causal explanation; the study of causal mechanisms through the application of mixed methods is also required. Due to the increasingly complicated nature of many classroom-based innovations that are subject to evaluation, and the potentially distal nature of hypothesised effects, particularly on attainment, programme theory and articulation of mechanisms are essential in enhancing causal explanation and promoting the accumulation of knowledge of what works and why in classroom settings.     

Implementing randomised control trials in open and distance learning: a feasibility study

Randomised control trials (RCTs) are an evidence-based research approach which has not yet been adopted and widely used in open and distance education to inform educational policy and practice. Despite the challenges entailed in their application, RCTs hold the power to robustly evaluate the effects of educational interventions in distance learning and conclude on whether (or not) these interventions should be adopted and used extensively. The aim of this paper is to spark discussions around the use of RCTs in distance learning by illustrating their benefits and drawbacks including challenges in adopting RCTs in education. To achieve this aim, a RCT was implemented to examine whether a small-scale intervention in four language modules could improve attendance at an end-of-module speaking assessment, and in consequence, performance, completion and pass rates. Results raise the need for further research in order to identify what type of interventions should be designed and put into practice to elicit a positive impact on learners. The paper concludes with a discussion on why RCTs should be brought to the forefront as a viable method for the effective evaluation of the impact of open learning analytic interventions.     

Intention-to-treat analysis in partially nested randomized controlled trials with real-world complexity

ABSTRACT

Demands for scientific knowledge of what works in educational policy and practice has driven interest in quantitative investigations of educational outcomes, and randomized controlled trials (RCTs) have proliferated under these conditions. In educational settings, even when individuals are randomized, both experimental and control students are often grouped into particular classrooms and schools and share common learning experiences. Analyses that account for these clusters are common. A less common design involves one clustered experimental arm and one unclustered experimental arm, sometimes called a partially clustered design. Analysts do not always use methods that yield valid statistical inferences for such partially clustered designs. Additionally, published methods for handling partially clustered designs may not be flexible enough to handle real-world complications, including treatment non-compliance. In this paper, we illustrate how models that accommodate partial clustering may be used in educational research. We explore the performance of these models using a series of Monte Carlo simulations informed by data taken from a large-scale RCT studying the impacts of a programme designed to decrease summer learning loss. We find that clustering and non-compliance can have substantial impacts on statistical inferences about intent-to-treat effects, and demonstrate methods that show promise for addressing these complications.     

Two models and three uses for mock trials in justice education

This article distinguishes two models that incorporate three uses for mocks trials in criminal justice education. The first model, which I refer to as the Law School Model (LSM), adapts for undergraduate education the method that is commonly used in legal education (Kravetz 2001). It uses a single case and one mock trial during an entire semester, and pursues a deeper understanding of all pretrial and trials stages. However, few mock trial “kits” exist for educators pursuing this model. The second model, which I refer to as the Liberal Education Model (LEM), focuses on the trial itself and pursues understanding of the judicial system and substantial development of communication skills fostered by traditional liberal education. This article suggests that the LEM offers criminal justice educators greater flexibility because (1) it offers many complete mock trial “kits” for instant use, (2) it incorporates all three uses discussed in this article (whereas the LSM is limited to the first use), and (3) many disciplines in over 200 colleges and universities use it in courses and intercollegiate competitions.     

"猴子审判"与美国自由教育原则

历史上的两次“猴子审判”相映成趣,两者都涉及美国的自由教育原则,但结果却大相径庭。第一次“猴子审判”借助自由教育原则帮助达尔文的进化论进入美国学校课堂,而新版“猴子审判”却在自由教育原则下试图用“神创论”和“智能设计论”取代进化论。“猴子审判”向美国自由教育原则提出了挑战,人们对自由教育原则进行实际的理性的反思。     

Multimedia and interactivity in distance learning of resuscitation guidelines: a randomised controlled trial

The aim of the study was to conduct a comparative analysis of three forms of knowledge transfer online, taking into account the long-term knowledge growth rate and the satisfaction of students surveyed in respect to distance learning. The cohort of participants for this survey were students of the first degree, studying in the Emergency Medical Services. The authors created three e-learning courses in basic and advanced resuscitation procedures. The level of course participants’ satisfaction was analysed, as well as the knowledge growth rate measured three times: 7, 30 and 90 days, respectively, after finishing the course. The level of satisfaction in all the groups surveyed was comparable. The growth rate for knowledge was the highest in group C in the second and third survey, respectively (30 and 90 days after the end of the course), and a significant difference in favour of course C was demonstrated one month after finishing classes.     

Using learning styles theory in engineering education

Studies have shown that, while learning different concepts, people sometimes take different approaches (learning styles). Accordingly, their performance reflected differently in their academic studies. With the effect of globalisation to the educational environments, the influence of individual learning styles on educational performance is getting more significant. In this study, a learning style assessment tool was used to examine the relationship between students’ learning styles and their performance in engineering education programmes. At the beginning of the programme, 285 students’ learning styles were measured using a learning style assessment tool. Four years after the engineering education, their performance in the programme was compared with their individual learning styles. This study shows that most of the students are assimilators. Divergers and convergers follow the assimilators. The number of accommodators is very limited. The relationship between engineering students’ learning styles and their performance is found: assimilators and convergers performed better than the divergers and accommodators. The performance difference between assimilators and divergers is statistically significant. The results of this study show that the learning style theory is a potential tool for guiding the design and improvement of courses and helping students to improve their individual performance.     

Using income effects to market undergraduate education abroad participation in higher education

Relying heavily on case or regional studies, higher education institutions market the perceived global skills that students acquire from education abroad as a value-added commodity. As industries increase their global reach, understanding the labor market effects of education abroad from a national perspective is necessary and timely. This study isolates many factors known to influence education abroad using the 2008–2012 Baccalaureate and Beyond Longitudinal Study dataset (B&B: 08/12). Through a 2-level hierarchical linear regression analysis model, earning levels of graduates of undergraduate programs are analyzed, offering a national perspective. The findings revealed that students who participated in education abroad earned more money within the labor market, even during harsh recessionary times. While income differences observed between men and women existed in both samples, initial incomes during the first three years post-graduation demonstrated significantly higher wages for some students who studied abroad when compared to their non-participating peers.     

A systematic review of controlled trials evaluating interventions in adult literacy and numeracy

This paper reports a systematic review of the quasi‐experimental literature in the field of adult literacy and numeracy, published between 1980 and 2002. We included 27 controlled trials (CTs) that evaluated strategies and pedagogies designed to increase adult literacy and numeracy: 18 CTs with no effect sizes (incomplete data) and 9 CTs with full data. These nine trials are examined in detail for this paper. Of these nine trials, six evaluated interventions in literacy and three evaluated interventions in literacy and numeracy. Three of the nine trials showed a positive effect for the interventions, five trials showed no difference and one trial showed a positive effect for the control treatment. The quality of the trials was variable, but many of them had some methodological problems. There was no evidence of publication bias in the review. There have been few attempts to expose common adult literacy or numeracy programmes to rigorous evaluation and therefore in terms of policy and practice it is difficult to make any recommendations as to the type of adult education that should be supported. In contrast, however, the review does provide a strong steer for the direction to be taken by educational researchers: because of the present inadequate evidence base rigorously designed randomised controlled trials and quasi‐experiments are required as a matter of urgency.     

Towards a controlled vocabulary on software engineering education

Software engineering is the discipline that develops all the aspects of the production of software. Although there are guidelines about what topics to include in a software engineering curricula, it is usually unclear which are the best methods to teach them. In any science discipline the construction of a classification schema is a common approach to understand a thematic area. This study examines previous publications in software engineering education to obtain a first controlled vocabulary (a more formal definition of a classification schema) in the field. Publications from 1988 to 2014 were collected and processed using automatic clustering techniques and the outcomes were analysed manually. The result is an initial controlled vocabulary with a taxonomy form with 43 concepts that were identified as the most used in the research publications. We present the classification of the concepts in three facets: ‘what to teach’, ‘how to teach’ and ‘where to teach’ and the evolution of concepts over time.     

Educational benefits of using game consoles in a primary classroom: A randomised controlled trial

It is known that computer games are motivating for children, but there is limited direct evidence of their effects on classroom learning. The studies that are available tend to be limited in terms of output data reported, or small in scale, or both. The aim of this randomised controlled trial was to upscale a recent study by Miller and Robertson investigating the effects of a commercial off‐the‐shelf computer game on children's mental computation skills and self‐perceptions. A pre‐post design was employed, with 634 primary (elementary) school children (10–11 years old) from 32 schools across Scotland. Schools were randomly assigned to experimental or control conditions. In the experimental schools, children used a games console for 20 minutes each day, running a ‘brain training’ game. The controls continued with their normal routine. The treatment period was 9 weeks. Significant pre‐post gains in accuracy and speed of calculations were found in both experimental and control groups over the treatment period. Gains in the experimental group were 50% greater than those of the controls in accuracy, and twice those of the controls in speed. There were no significant changes in two measures of self‐concept for either group. There was a small but statistically significant improvement in attitude towards school among the experimental group but not the controls. When scores were analysed by ability, different patterns were apparent. The design of the study allows a degree of confidence when generalising from these results. Some implications of the findings are discussed.