Communicating comparative findings from meta-analysis in educational research: some examples and suggestions

ABSTRACT

This article reviews some of the strengths and limitations of the comparative use of meta-analysis findings, using examples from the Sutton Trust-Education Endowment Foundation Teaching and Learning ‘Toolkit’ which summarizes a range of educational approaches to improve pupil attainment in schools. This comparative use of quantitative findings has similar characteristics to umbrella reviews which provide a succinct but applicable summary of the current state of evidence to inform practice or policy. Meta-analysis helps to identify which approaches have, on average, made the most difference to tested learning outcomes, in terms of effect size. We suggest that any comparative inferences made between meta-analyses should be treated cautiously, but taken seriously. Additionally, we present alternative ways of interpreting effect sizes, security ratings and cost-estimates to make research findings accessible, whilst retaining appropriate accuracy which is discussed using the ‘Toolkit’ as an example. We conclude by arguing that we should consider the available information not as ‘what works’, but ‘what has worked’ to understand its value and limits in terms of supporting the development of research-based practice.     

Professionalising Ph.D. education to enhance the quality of the student experience

Ph.D. student numbers have been increasing in Australia as has government interest in the economic and social outcomes of graduate education (Moses 1994, Cullen et al. 1994). Yet the position of Ph.D. students can still be seen as marginal within universities and the institutional organisation of Ph.D. education as problematic. This situation can be related to the highly individualistic nature of research and supervision which is both a barrier to and part of the argument against efforts to examine and discuss in general the processes of research training and supervisory interactions across disciplinary and departmental boundaries. This article positions all Ph.D. students, in all fields of study, as learners in a form of professional education -a perspective which offers a way out of this apparent dilemma. Drawing on data from a study reported in Cullen et al. (1994) the following topics are explored: Ph.D. students as learners of the knowledge and skill of the professional practice of research and scholarship; the role of the supervisor in assisting students to become independent practitioners; and the complementary professional role of student participation in the academic community. Implications for policy and practice to professionalise Ph.D. education and enhance the quality of the student experience are discussed.     

Translating Laboratory Evidence into Classroom Practice with Teacher‐Led Randomized Controlled Trials—A Perspective and Meta‐Analysis

We initiated and structured a single program that supervised teachers, some with neuroscience or psychology degrees, to collaborate and explore the effects of science of learning‐translated pedagogy. This article reports on the 34 findings from teacher‐led randomized controlled trials (RCTs) and replications. Teachers designed trials, looking at areas such as attention, memory, and spaced learning. Overall, positive effects were found over short periods (1–6 weeks; r = 0.15, p < .0001 [d = 0.30], N = 2,157). However, retrieval practice (testing as a learning experience) had differential effects mediated by age, approach, and lesson content. Results suggest science of learning‐translated pedagogy needs extensive replication to establish how best to use laboratory evidence in classrooms. Multiple planned replication of teacher‐led RCTs has potential as an evaluation tool, combining high levels of mundane realism with strong internal validity and the potential to build cost effective large samples for meta‐analysis.     

Culturally responsive peacebuilding pedagogy: a case study of Fambul Tok Peace Clubs in conflict-affected Sierre Leone

Recent research on the peace-promoting agency of teachers in conflict-affected contexts has critiqued the dominance of a menu of curriculum initiatives circulated within aid organizations which ignore or dismiss indigenous cultural resources available within local communities. On the other hand, the turn to local culture in peacebuilding interventions has also been critiqued for idealizing the local. This paper presents a case study of the work of Fambul Tok Peace Clubs in conflict-affected Sierra Leone which aim to support teachers in drawing on local practices, rituals and symbols. Using an interpretative approach, alert to the meanings teachers ascribe to their pedagogical practices, it finds that teachers respond to local culture in multiple, sometimes contradictory ways. These range from celebration of indigenous practices as resources for conflict resolution that resonate with their pupils and communities to critique of their oppressive gender norms and inability to address structural drivers of conflict.     

Developmental Changes in Achievement Evaluation: Motivational Implications of Self-Other Differences

In order to explore the development of self-evaluative biases, children at 3 age levels (5–6, 7–8, 9–10) evaluated themselves or another child when given social or temporal comparison feedback. Evaluative biases were indicated by higher evaluations for the self than another, especially after failure. Children at different ages were not differentially responsive to temporal vs. social comparison information. However, evaluative bias in response to the type of evaluation differed by age; there was greater bias for general ability evaluations by older children and greater bias for specific performance evaluations by younger children. Alternative explanations for these self-other differences were minimized, as differences in visual orientation were held constant and differences in knowledge of effort were controlled statistically. Other aspects of the design as well as sex differences support a motivational interpretation of these biases. How these biases are manifested is related to children's level of understanding and concerns at different ages.     

MacIntyre's Moral Theory and the Possibility of an Aretaic Ethics of Teaching

In this paper, I reconstruct Alasdair MacIntyre's aretaic, practical philosophy, drawing out its implications for professional ethics in general and the practice of teaching in particular. After reviewing the moral theory as a whole, I examine MacIntyre's notion of internal goods. Defined within the context of practices, such goods give us reason to reject the very idea of applied ethics. Being goods for the practitioner, they suggest that the eudaimonia of the practitioner is central to professional ethics. In this way, MacIntyre's moral theory helps us recover the untimely question, how does teaching contribute to the flourishing of the teacher?     

The functional use of a mathematical sign

The question of how a mathematics student at university-level makes sense of a new mathematical sign, presented to her or him in the form of a definition, is a fundamental problem in mathematics education. Using an analogy with Vygotsky's theory (1986, 1994) of how a child learns a new word, I argue that a learner uses a new mathematical sign both as an object with which to communicate (like a word is used) and as an object on which to focus and to organise her or his mathematical ideas (again as a word is used) even before she or he fully comprehends the meaning of this sign. Through this sign usage, I claim that the mathematical concept evolves for that learner so that it eventually has personal meaning, like the meaning of a new word does for a child; furthermore, because the usage is socially regulated, I claim that the concept evolves for the learner so that its usage concurs with its usage in the mathematical community. In line with Vygotsky, I call this usage of the mathematical sign before mature understanding, ‘functional use’. I demonstrate ‘functional use’ of signs (manipulations, imitations, template-matching and associations) through an analysis of an interview in which a mathematics university student engages with a ‘new’ mathematical sign, the improper integral, using pedagogically designed tasks and a standard Calculus textbook as resources.     

Assessing the unassessable: making learning visible in undergraduates’ experiences of scientific research

We suggest that academics involved in the provision of research experiences to undergraduate science students may benefit by reconceptualising these experiences as work-based learning. In particular, drawing on the widespread use of reflective practice in work-based learning allows for a more effective focus on process-related learning. We describe an empirical study involving the implementation of reflective blogs in which science students responded to prompt questions designed to focus their attention on how their project was progressing, the process of research and their own learning. The blogs reveal examples of students engaging in independent thinking, creative thinking and developing a sense of themselves as scientists, all qualities valued by supervisors. They also show how these combine as ‘wicked competences’ that allow students to address the complex problems and uncertainties of research. We identify characteristic patterns of reflection that indicate the exercise of such competences and suggest that these could be used as explicit evidence by academics making judgements for assessment purposes.