Individual participant data meta-analysis of the impact of educational interventions on pupils eligible for Free School Meals

Meta-analysis is the synthesis of findings from research projects, which enables an estimate of the average or pooled effect across various studies. This study presents findings from the intention to treat analysis for a series of educational evaluations in England using a two-stage meta-analysis with standardised outcome data and individual participant data meta-analyses. The research estimates the overall impact of educational trials on pupils eligible for Free School Meals (FSM) and the attainment gap in literacy and mathematics performance between FSM and non-FSM pupils based on analysis of 88 trials and data from over half a million pupils. For the meta-analyses, frequentist and Bayesian multilevel models were used to estimate the individual and pooled effect size across categories of explanatory variables such as age groups (key stages in England) and aspects of the type of interventions (one-to-one, small group, whole class). Results indicated that the overall impact of interventions on the literacy outcomes of FSM pupils was positive, with a pooled effect size of 0.06 (0.03, 0.08). However, for mathematics, no overall effect on FSM pupils was observed. Analysis of the attainment gap indicated that literacy outcomes for FSM pupils were improved by interventions marginally more than for non-FSM pupils (pooled attainment gap 0.01 (−0.01, 0.04)). The risk of bias assessment showed that estimates were consistent across different methodological approaches. Overall, evidence from this study can be used to identify, test and scale educational interventions in schools to improve educational outcomes for disadvantaged pupils.     

Communicating comparative findings from meta-analysis in educational research: some examples and suggestions

ABSTRACT

This article reviews some of the strengths and limitations of the comparative use of meta-analysis findings, using examples from the Sutton Trust-Education Endowment Foundation Teaching and Learning ‘Toolkit’ which summarizes a range of educational approaches to improve pupil attainment in schools. This comparative use of quantitative findings has similar characteristics to umbrella reviews which provide a succinct but applicable summary of the current state of evidence to inform practice or policy. Meta-analysis helps to identify which approaches have, on average, made the most difference to tested learning outcomes, in terms of effect size. We suggest that any comparative inferences made between meta-analyses should be treated cautiously, but taken seriously. Additionally, we present alternative ways of interpreting effect sizes, security ratings and cost-estimates to make research findings accessible, whilst retaining appropriate accuracy which is discussed using the ‘Toolkit’ as an example. We conclude by arguing that we should consider the available information not as ‘what works’, but ‘what has worked’ to understand its value and limits in terms of supporting the development of research-based practice.     

Accounting for the Relationship Between Initial Status and Growth in Regression Models

Educational analysts studying achievement and other educational outcomes frequently encounter an association between initial status and growth, which has important implications for the analysis of covariate effects, including group differences in growth. As explicated by Allison (1990 Allison, P. D. (1990). Change scores as dependent variables in regression analyses. Sociological Methodology, 20, 93–114.[Crossref] , [Google Scholar]), where only two time points of data are available, identifying a preferred model can be difficult or impossible. In this paper we extend Allison's inquiry by considering multiple sources of the association between initial status and growth simultaneously, including measurement error but also intrinsic associations between initial status and growth. We illustrate the potential trade-offs between the change-score model specifications (models without a control for initial status) and regressor-variable specifications (with a control for initial status) using simulated data.     

Retrieval Practice Benefits Deductive Inference

Retrieval practice has been shown to benefit learning. However, the benefit has sometimes been attenuated with more complex materials that require integrating multiple units of information. Critically, Tran et al. Psychonomic Bulletin & Review, 22, 135–140 (2015) found that retrieval practice improves sentence memory but not the drawing of inferences from the same sentences. In three experiments, we investigated whether this lack of benefit of retrieval practice for inferential ability was due to the presentation format of the material. Participants studied four sets of seven to nine related sentences by practicing retrieval for two sets and rereading the other two sets. A final test was given 2 days later. When sentences were presented one at a time during study/practice as in Tran et al., we found no effect of retrieval practice on a test requiring inferential reasoning. When sentences in a set were presented simultaneously during study/practice, retrieval practice in the form of fill-in-the-blank testing (experiments 1 and 2) and free recall (experiment 3) aided later deductive inference more than rereading. Our findings suggest that retrieval practice can improve deductive inference, but in order to optimize its utility, the format in which the material is presented during practice must not hinder relational processing of the individual sentences.     

Culturally responsive peacebuilding pedagogy: a case study of Fambul Tok Peace Clubs in conflict-affected Sierre Leone

Recent research on the peace-promoting agency of teachers in conflict-affected contexts has critiqued the dominance of a menu of curriculum initiatives circulated within aid organizations which ignore or dismiss indigenous cultural resources available within local communities. On the other hand, the turn to local culture in peacebuilding interventions has also been critiqued for idealizing the local. This paper presents a case study of the work of Fambul Tok Peace Clubs in conflict-affected Sierra Leone which aim to support teachers in drawing on local practices, rituals and symbols. Using an interpretative approach, alert to the meanings teachers ascribe to their pedagogical practices, it finds that teachers respond to local culture in multiple, sometimes contradictory ways. These range from celebration of indigenous practices as resources for conflict resolution that resonate with their pupils and communities to critique of their oppressive gender norms and inability to address structural drivers of conflict.     

Accounting for Model Uncertainty in the Prediction of University Graduation Rates

Empirical analysis requires researchers to choose which variables to use as controls in their models. Theory should dictate this choice, yet often in social science there are several theories that may suggest the inclusion or exclusion of certain variables as controls. The result of this is that researchers may use different variables in their models and come to disparate conclusions with respect to predicted effects and their statistical significance. In such cases one is uncertain of which particular set of regressors forms the model that represents the data. The approach used below accounts for uncertainty in variable selection by using Bayesian model averaging (BMA). Accounting for uncertainty, we demonstrate that BMA provides better out-of-sample prediction for university graduation rates than results based on alternative variable selection methods.