Beyond synthesis: augmenting systematic review procedures with practical principles to optimise impact and uptake in educational policy and practice

ABSTRACT

Whilst systematic reviews, meta-analyses and other forms of synthesis are considered amongst the most valuable forms of research evidence, their limited impact on educational policy and practice has been criticised. In this article, we analyse why systematic reviews do not benefit users of evidence more consistently and suggest how review teams can optimise the impact of their work. We introduce the Beyond Synthesis Impact Chain (BSIC), an integrated framework of practical strategies for enhancing the impact of systematic reviews. Using examples from health professions education, we propose that review teams can optimise the impact of their work by employing strategies that (1) focus on practical problems and mindful planning in collaboration with users; (2) ensure reviews are relevant and syntheses reflexively account for users’ needs; and (3) couch reports in terms that resonate with users’ needs and increase access through targeted and strategic dissemination. We argue that combining practical principles with robust and transparent procedures can purposefully account for impact, and foster the uptake of review evidence in educational policy and practice. For systematic review teams, this paper offers strategies for enhancing the practical utility and potential impact of systematic reviews and other forms of synthesis.     

Findings From a Multiyear Scale-Up Effectiveness Trial of Open Court Reading

This multiyear scale-up effectiveness study of Open Court Reading (OCR) involved approximately 4,500 students and more than 1,000 teachers per year in Grades K–5 from 49 elementary schools in seven districts across the country. Using a school-level cluster randomized trial design, we assessed the implementation and effectiveness of Open Court Reading over two years. Implementation study results demonstrated adequate to high levels of fidelity across the treatment schools. Intent-to-treat analyses revealed no statistically significant main effects on students’ reading performance in Year 1 and a small negative effect (d = – .09) in Year 2. There were positive impacts for particular subgroups, including kindergarten (d = .12) and Hispanic (d = .10) students in the first year. However, there were negative impacts for first grade (d = –.13), females (d = –.11), students who were not eligible for free or reduced-price lunch (d = –.19), and non-English language learners (d = –.10) in the second year of the study. Thus, relative to the “business-as-usual” reading curricula, no positive overall impacts of OCR and mixed impacts for student subgroups were found.     

Perpetuating academic capitalism and maintaining gender orders through career practices in STEM in universities

Academic capitalism is an outcome of the interplay between neoliberalism, globalisation, markets and universities. Universities have embraced the commercialisation of knowledge, technology transfer and research funding as well as introducing performance and audit practices. Academic capitalism has become internalised as a regulatory mechanism by academics who attempt to accumulate academic capital. Universities are traditionally gendered organisations, reflecting the societal gender order. Despite fears regarding the feminisation of the academy, the embrace of academic capitalism is contributing to its re-masculinisation and exercises an incidental gender effect. Practicing is the means by which the gender order is constituted at work. Three practices in which academics engage are examined as exemplars of the way academics increase their academic capital stock in Science, Technology, Engineering and Maths (STEM) faculties in four European universities, in Bulgaria, Denmark, Ireland and Turkey. These practices tend to be more achievable and likely to be engaged in by men, thus, career practices are the mechanism through which the gender effect of academic capitalism is achieved, academic capitalism perpetuated and the gender order maintained in STEM in academia.     

Pygmalion and the gender gap: do teacher expectations contribute to differences in achievement between boys and girls at the beginning of schooling?

ABSTRACT

This study examined the role of teacher expectations in the emerging gender gaps in reading and mathematics in the first year of schooling. Therefore, we first investigated whether boys and girls differ in their vulnerability to teacher expectancy effects. Second, we analysed whether gender-specific effects of teacher expectations contribute to gender achievement gaps. Our analyses were based on 1,025 first-grade students in Germany. Among the majority of the students, boys and girls did not differ in their vulnerability to teacher expectancy effects. Further analyses examined a subgroup of students who were targets of relatively strong teacher expectation bias and who showed unexpectedly high or low achievement gains. In this specific subgroup, girls’ mathematics achievement was more adversely affected by negatively biased expectations and benefitted less from positive bias than boys’ achievement. Mediation analyses revealed that teacher expectation bias did not substantially contribute to gender gaps in reading or mathematics.     

Mixture Simultaneous Factor Analysis for Capturing Differences in Latent Variables Between Higher Level Units of Multilevel Data

Given multivariate data, many research questions pertain to the covariance structure: whether and how the variables (e.g., personality measures) covary. Exploratory factor analysis (EFA) is often used to look for latent variables that might explain the covariances among variables; for example, the Big Five personality structure. In the case of multilevel data, one might wonder whether or not the same covariance (factor) structure holds for each so-called data block (containing data of 1 higher level unit). For instance, is the Big Five personality structure found in each country or do cross-cultural differences exist? The well-known multigroup EFA framework falls short in answering such questions, especially for numerous groups or blocks. We introduce mixture simultaneous factor analysis (MSFA), performing a mixture model clustering of data blocks, based on their factor structure. A simulation study shows excellent results with respect to parameter recovery and an empirical example is included to illustrate the value of MSFA.     

How do apes ape?

In the wake of telling critiques of the foundations on which earlier conclusions were based, the last 15 years have witnessed a renaissance in the study of social learning in apes. As a result, we are able to review 31 experimental studies from this period in which social learning in chimpanzees, gorillas, and orangutans has been investigated. The principal question framed at the beginning of this era, Do apes ape? has been answered in the affirmative, at least in certain conditions. The more interesting question now is, thus, How do apes ape? Answering this question has engendered richer taxonomies of the range of social-learning processes at work and new methodologies to uncover them. Together, these studies suggest that apes ape by employing a portfolio of alternative social-learning processes in flexibly adaptive ways, in conjunction with nonsocial learning. We conclude by sketching the kind of decision tree that appears to underlie the deployment of these alternatives.     

Using Group Viewing of Television to Teach Preschool Children Social Skills

This quasi‐experimental field study assessed whether group viewing of television in a day‐care setting can be used to encourage prosocial behavior in young children. Drawing from social learning theory and cognitive‐developmental stage theory, four short‐ term programs were designed and tested. Children (aged 3‐5 years) from eight day‐care centers in Montreal (n = 150) were part of the study. Intact groups (two settings randomly assigned to condition) participated in 8 days of intervention consisting of group viewing of video‐taped segments of Sesame Street, followed by participation in activities. The programs were evaluated using a 2 X 2 X 2 factorial pretest—post‐test design with video‐type (prosocial, cognitive), activity‐type (cooperative, individualistic), and gender as the three factors. Measures included free‐play observations, a perspective‐taking ability test, and two qualitative measures describing the context. Analysis of covariance (pretest and age combined as covariates) revealed a significant main effect for video‐type on prosocial behavior, and a significant interaction between video‐type and activity‐type on antisocial behavior. Results suggest that prosocial modelling using television can encourage prosocial behavior in the day‐care setting. A group‐viewing context, with or without post‐viewing enactive prosocial training, may enhance prosocial modeling effects.     

Differentiating between distance/open education systems-parameters for comparison

A wide range of institutions all over the world utilize various forms of distance and/or open learning. This paper suggests eight parameters for categorizing distance/open education systems: (1) target population; (2) dimensions of openness; (3) organizational structure; (4) design and development of learning materials; (5) use of advanced technology; (6) teaching/tutoring system; (7) student-support system; and (8) inter-institutional collaboration. These parameters provide criteria for comparing a broad spectrum of distance/open learning institutions, and assist in depicting the unique characteristics of each distance/open education system. The discussion is preceded by a conceptual examination of the relations and interrelations between thedistance education andopen learning terms.

---

Zusammenfassung Eine große Anzahl von Institutionen in der ganzen Welt wendet verschiedene Formen von Fernunterricht oder offenem Lernen an. In diesem Artikel werden acht Parameter zur Kategorisierung von Fernunterricht und offenem Lernen vorgeschlagen: (1) Zielgruppe; (2) Dimensionen des Zugangs; (3) organisatorische Struktur; (4) Design und Entwicklung von Lernmaterialien; (5) Anwendung fortgeschrittener Techniken; (6) Lehr- und Überprüfungssystem; (7) System zur Unterstützung der Studenten; und (8) Zusammenarbeit zwischen den Institutionen. Diese Parameter stellen Kriterien für den Vergleich eines breiten Spektrums von Institutionen im Bereich des Fernunterrichts und des offenen Lernens zur Verfügung und helfen bei der Identifikation einzigartiger Eigenheiten jedes betreffenden Systems. Der Diskussion geht eine konzeptionelle Untersuchung der Beziehungen zwischen den BegriffenFernunterricht undoffenes Lernen voraus.

---

Résumé Un grand nombre d'institutions de par le monde utilise diverses modalités d'apprentissage ouvert et/ou à distance. Le présent article suggère huit paramètres pour classer les systèmes d'enseignement ouvert ou à distance: (1) la population cible, (2) les dimensions de l'ouverture, (3) la structure organisationnelle, (4) la conception et la mise en oeuvre des matériaux didactiques, (5) l'utilisation de technologies modernes, (6) le système d'enseignement et de tutorat, (7) le système de soutien aux élèves et (8) la collaboration interinstitutionnelle. Ces paramètres fournissent des critères de comparaison pour un grand nombre d'institutions diversifiées d'enseignement ouvert ou à distance, et permettent de décrire les caractéristiques majeures de chacun de ces systèmes. Un examen conceptuel des relations et interrelations entre les termes d'enseignement à distance et d'apprentissage ouvert, précède cette discussion.