Beyond synthesis: augmenting systematic review procedures with practical principles to optimise impact and uptake in educational policy and practice

ABSTRACT

Whilst systematic reviews, meta-analyses and other forms of synthesis are considered amongst the most valuable forms of research evidence, their limited impact on educational policy and practice has been criticised. In this article, we analyse why systematic reviews do not benefit users of evidence more consistently and suggest how review teams can optimise the impact of their work. We introduce the Beyond Synthesis Impact Chain (BSIC), an integrated framework of practical strategies for enhancing the impact of systematic reviews. Using examples from health professions education, we propose that review teams can optimise the impact of their work by employing strategies that (1) focus on practical problems and mindful planning in collaboration with users; (2) ensure reviews are relevant and syntheses reflexively account for users’ needs; and (3) couch reports in terms that resonate with users’ needs and increase access through targeted and strategic dissemination. We argue that combining practical principles with robust and transparent procedures can purposefully account for impact, and foster the uptake of review evidence in educational policy and practice. For systematic review teams, this paper offers strategies for enhancing the practical utility and potential impact of systematic reviews and other forms of synthesis.     

THE PLACE OF SYSTEMATIC REVIEWS IN EDUCATION RESEARCH

ABSTRACT:  The use of systematic reviews in educational research is a growing phenomenon in the UK, but is highly controversial. This article argues that such reviews have a useful place in a research cycle that wishes to inform and be informed by practice and policy. It proposes and discusses a model of educational research, showing how reviews relate to small or large-scale primary studies.     

Work-Based Knowledge,Evidence-Informed Practice and Education

ABSTRACT:?

This paper starts from an examination of an epistemological framework that underpins practice in particular educational contexts. It examines work-based knowledge, relating this to practitioner research and evidence informed practice. This is followed by an exploration of arguments that call for increased rigour in educational research as well as the use of systematic reviews. The paper examines tensions within educational research located in particular institutional contexts which draw upon ‘post-modern’ conceptualisations of practice, setting these against research concerned with generalisability that veers towards traditional positivist claims. The paper concludes by suggesting that such arguments readily fold back into a conservative empiricism and a more appropriate position should be based upon dialogue across a range of constituents. However, such a notion needs to recognise social antagonism as well as aspirations towards the deepening of notions of social justice.     

Communicating comparative findings from meta-analysis in educational research: some examples and suggestions

ABSTRACT

This article reviews some of the strengths and limitations of the comparative use of meta-analysis findings, using examples from the Sutton Trust-Education Endowment Foundation Teaching and Learning ‘Toolkit’ which summarizes a range of educational approaches to improve pupil attainment in schools. This comparative use of quantitative findings has similar characteristics to umbrella reviews which provide a succinct but applicable summary of the current state of evidence to inform practice or policy. Meta-analysis helps to identify which approaches have, on average, made the most difference to tested learning outcomes, in terms of effect size. We suggest that any comparative inferences made between meta-analyses should be treated cautiously, but taken seriously. Additionally, we present alternative ways of interpreting effect sizes, security ratings and cost-estimates to make research findings accessible, whilst retaining appropriate accuracy which is discussed using the ‘Toolkit’ as an example. We conclude by arguing that we should consider the available information not as ‘what works’, but ‘what has worked’ to understand its value and limits in terms of supporting the development of research-based practice.     

What is Evidence-based Education?

This paper argues that education should become more evidence-based. The distinction is made between using existing research and establishing high-quality educational research. The need for high-quality systematic reviews and appraisals of educational research is clear. Evidence-based education is not a panacea, but is a set of principles and practices for enhancing educational policy and practice.     

Research and the Development of Educational Assessment

This paper discusses significant examples of the impact of research studies on practice and policy in educational assessment. The discussion focuses on selected issues in three main areas, concerned respectively with formative assessment, with high-stakes summative tests, and with assessment in large-scale surveys. It is concluded that the effects of research have been uneven and weak, in part because of lack of co-ordination of studies and of any one strong institutional centre. However, researchers are also faced with the difficult task of changing the understanding of assessment issues, both amongst the general public and amongst policy makers.     

What can evidence-use in practice learn from evidence-use in policy?

Abstract

Background: This paper approaches evidence-informed practice from the perspective of evidence-informed policy-making. Using the findings of a recent study of evidence-use by educational policy-makers to raise questions about evidence-use by educational practitioners, it seeks to explore what such a study might tell us about how to understand and improve evidence-use by educational practitioners.

Purpose: The paper aims, therefore, to identify potential connections, shared insights and common issues between evidence-use in policy and evidence-use in practice. It does this by focusing on two specific areas: the nature of the evidence (i.e. what evidence is used) and the nature of the use (i.e. how evidence is used). The paper outlines what was found about each of these aspects of evidence-use in policy, and then considers what questions and issues these findings might raise for evidence-use in educational practice.

Sample: The empirical study on which this paper is based was an in-depth study of the use of evidence within educational policy development in Australia. It focused on the development of three specific education policies within one Australian state education department and involved interviews with 25 policy-makers who were actively involved in the development of these policies.

Design and methods: The policy-based study involved the following data collection processes: (i) in-depth semi-structured interviews with 25 policy-makers who were involved in the development of the selected policies; (ii) documentary analysis of policy documents, background research reports and other relevant papers relating to the selected policies; (iii) unstructured observation (where possible) of meetings and events connected with the development of the selected policies; and (iv) feedback from 40 wider policy staff who took part in a verification workshop to discuss the project’s emerging findings.

Findings: Drawing on the findings from the original policy study, two areas of potential connection to evidence-use in practice are explored. First, in relation to ‘varieties of evidence and uses’, the negotiation of diverse evidence types and the potential for using evidence in multiple and varied ways appear to be features of evidence-use that are common to educational policy-makers as well as educational practitioners. Secondly, in relation to ‘narrowness of evidence sources’, there is potential for both policy-makers and practitioners to use a narrow (rather than broad) selection of evidence, due to a tendency to work with certain evidence types as a starting point (e.g. performance data) and a tendency to draw on certain evidence sources more frequently (e.g. well-known, familiar research sources).

Conclusions: This paper emphasises: (i) the need for more integrated (or joined-up) understandings of evidence-use across contexts of practice and contexts of policy; (ii) the importance of continued efforts to understand and represent evidence-use more effectively within educational practices; and (iii) the value of paying careful attention to the quality and qualities of evidence-use within and across the different settings of educational practice and policy.     

'All the Evidence Shows …': Reasonable expectations of educational research

After years of being criticised in Britain as irrelevant to the task of raising educational standards, educational researchers have been invited to join in the improvement of policy and practice. Welcome though the invitation is, some views of how they should deserve influence reflect unrealistic or narrow definitions of really useful research. This paper argues against exaggerating the prospects for a science of teaching; against tying the national research agenda tightly to what are currently identified as 'central issues'; and for a broader approach to relevance than direct applicability to improving educational practice.     

From policy to practice: The challenges to educational access for non-nationals in South Africa

This paper argues that the problems of educational access for non-nationals in South Africa lie not simply in failures of the current policies, although there are certainly instances where policies need modification, but largely in the implementation of existing policies, and the ways in which they are developed and modified. The paper reviews evidence in the international literature and draws on empirical evidence from a small study of a group of Zimbabwean migrant children to illustrate more clearly the dynamics that serve to exclude them from access to schooling, despite official policy commitments. Key research questions that are addressed are what main barriers to educational access exist for non-national and are these a result of policy gaps; how does the implementation of existing educational policies affect the educational access of non-nationals; what approaches to policy and practice would be more effective in ensuring non-nationals participate fully in basic education? The paper ends with some observations on how to address the policy gaps and how to develop a more effective approach to policy formation and implementation in order to improve both policy and practice.     

International large-scale assessments: what uses,what consequences?

Abstract

Background: International large-scale assessments (ILSAs) are a much-debated phenomenon in education. Increasingly, their outcomes attract considerable media attention and influence educational policies in many jurisdictions worldwide. The relevance, uses and consequences of these assessments are often the focus of research scrutiny. Whilst some argue that the assessment outcomes provide an effective basis for informed policy-making, critics claim that the use of international assessment data can result in a range of unintended consequences, such as the shaping and governing of school systems ‘by numbers’.

Purpose: This article explores and analyses the arguments about the uses and consequences of ILSAs. In particular, the discourse about the assessments’ consequential validity will be discussed and evaluated.

Sources of evidence: Literature relating to the uses and consequences of large-scale assessment was analysed, with a focus on research on the consequential aspects of validity.

Main argument: Much research suggests that ILSAs have unintended consequences that affect and influence educational policy. However, the influences on educational policy are complex and interwoven: for example, it is not clear-cut whether effects such as converging curricular are, necessarily, direct consequences of large-scale assessments. Further, it is suggested that a beneficial consequence of large-scale assessment is the infrastructure they provide for studies in the social sciences, although caution must be applied to causal claims, in particular because of the cross-sectional design of the assessments.

Conclusions: The considerable literature discussing the uses and consequences of large-scale assessments tends to point out potential negative aspects of the studies. However, it is also apparent that large-scale international assessments can be a valuable resource for studying global trends and evolving systems in education. Despite the extensive debates around large-scale assessment outcomes both in the media and in educational policy arenas, empirical educational research all too often appears underused in the discussion.     

Current State and Development Trends of Education Policy Research in China in the Last Decade (2004–2013): A Statistical Analysis of Papers from Eight Core Chinese Journals

Abstract

The author conducted sampling and statistical analysis of papers on education policy research collected by the China National Knowledge Infrastructure in the period from the years 2004–2013. Under the current state of education policy research in China, the number of papers correlates positively with the year; the papers are concentrated in education journals; the researchers are primarily associated with institutions of higher education; and the studies cover a broad range of topics, while emphasizing the practice of education policy. The development trends of education policy research in China include a general rising trend for research on education policy; in the future, researchers from institutions of higher education will continue to be a dominant force in education policy research, and the authors’ regional distribution will continue to be imbalanced; education policy research will focus more on educational equity and improved quality, and studies of foreign education policies will continue to be a topic of interest.     

The Purpose and Functions of Budgets – Some Theoretical Perspectives for Education Managers

While educational leadership and management has experienced impressive development over the last three decades the fact that a robust comparative branch of the field has failed to emerge is equally conspicuous. This article builds a case for comparative and international educational leadership and management, arguing that the development of conceptual frameworks and instrumentation are imperative if the field is to keep abreast of globalisation of policy and practice. Accordingly, a conceptual framework is described and justified based on a cultural and cross-cultural approach focusing on the school level as the baseline unit for analysis. Specifically, the proposed framework is architectured around the interrelationship between two levels of culture, societal and organisational, and four elements comprising schooling and school-based management, namely, organisational structures, leadership and management processes, curriculum, and teaching and learning. Finally, limitations and implications of the model are discussed, including the need for the framework to be operationalised by developing appropriate research instruments.     

The Role of School Exclusion Processes in the Re-Production of Social and Educational Disadvantage

ABSTRACT

English education policy has increasingly focused on the need to intervene in an intergenerational cycle of poverty and low attainment. The accompanying policy discourse has tended to emphasise the impact of family background on educational outcomes. However, as the capacity of parents to secure positive educational outcomes for their children is closely linked to the quality of their own education, low attainment is rather more closely connected to what happens in schools than this focus suggests. Pupils from groups known to be at increased risk of low attainment are also known to be at increased risk of involvement in the disciplinary processes of schools. This paper draws on the findings of a small-scale qualitative study to highlight some of the limitations in the educational provision accessed by Secondary age pupils involved in school exclusion processes. The assumptions and tensions at practice level that underpinned this provision are also discussed. In the conclusion it is argued that a much stronger focus on the learning of these pupils could improve their attainment and contribute to a reduction in social and educational inequalities in the future.     

The school meals service: a necessity or a luxury good?

This article looks at the school meals service and argues that it is an important topic which lies at the interface of educational and social policy. It reviews the history of the service as one of changing ideas and emphases, particularly, but not exclusively, influenced at various times by changing knowledge about the relationship between nutrition and health. It discusses when, and under what circumstances, school meals are seen as a legitimate part of social and educational policy. It discusses the impact of current policies in relation to nutrition and health, with particular reference to changing modes of provision. Finally, using a small piece of empirical research done in Yorkshire, it discusses the impact on the domestic economy of family households of policy since the 1980 Act.     

Engaging with Bourdieu’s theory of practice: an empirical tool for exploring school students’ technology practice

ABSTRACT

This article presents Bourdieu’s theory of practice as a tool for exploring school students’ technology practice in empirical research. The authors provide educational technology researchers with an accessible introduction to the theory of practice. They then detail the conceptual, methodological and analytic application of the theory of practice in two educational technology studies. The application of the theory in the two studies highlights the potential of the sociological framing for informing a robust critical research agenda and understanding the circumstances that can contribute to digital inequalities. Practically, knowledge gained through theoretically informed research is critical for researchers, governments, schools and teachers in working to overcome digital inequalities.     

Glennie,Jenny et al ed (2012) Open educational resources and change in higher education Commonwealth of learning (Vancouver) isbn 978‐1‐894975‐53‐7 291 pp cdn$12 (or free download) http://www.col.org/resources/publications/Pages/detail.aspx?PID=412

This engaging book reviews current policy and practice in the development and use of open educational resources (oer ) internationally. It includes chapters that examine the current environment for oer development and use, including government and institutional policy, as well as a range of case studies that describe the development, re‐use and implementation of oer with a particular emphasis on developing countries. The book is useful for educators, educational leaders, learning resource developers, and government policymakers. Sara Hammer     

Autobiographies and interviews as means of ‘access’ to elite policy making in education 1

Abstract:

This paper explores questions of access to perceptions of educational policy by members of policy elites. In particular, it reviews some possibilities of broadening how national education policy is contructed by examining the utility of published autobiographical tests.     

Qualified to lead? A comparative,contextual and cultural view of educational policy borrowing

Abstract

Background: Around the globe, education policy borrowing remains pervasive and prevalent. The strategies, interventions and innovations of education systems that perform well, in international assessments, are enthusiastically borrowed and copied in the anticipation of similar educational performance and outcomes.

Purpose: This purpose of the article is to highlight some of the implications and consequences of policy borrowing by comparing the preparation and development programmes for school leaders in various education systems. The article explores the way in which the leadership programmes and models are being adopted and developed by different countries.

Sources of evidence: The article draws upon findings from a contemporary, comparative study of leadership preparation and development across seven different education systems (Seven System Leadership Study). This large-scale, mixed-methods, comparative research study is collecting quantitative and qualitative data about the nature and impact of leadership development and preparation programmes in seven different education systems.

Main argument: The findings from the study reinforce how leadership preparation and development programmes are increasingly becoming standardised as a result of education systems borrowing and adapting from each other. This article also argues that in the contemporary policy discourse, important cultural and contextual influences that significantly affect subsequent implementation and outcomes are increasingly being side-lined and ignored.

Conclusions: This article highlights some of the limitations and unintended consequences of educational policy borrowing. It concludes that adopting some of the design features of effective interventions rather than simply borrowing policies or strategies, in part or in their entirety from very different contexts, may be a more productive way forward.     

Discourse analysis as theory,method, and epistemology in studies of education policy

ABSTRACT

Discourse has featured in studies of educational policy as an analytic and methodological tool, theoretical frame, realm of implication, and even a foundational definition of educational policy itself (e.g.) Despite the centrality of discourse as a frame for exploring educational policy and its implications, the ways that discourse is defined or operationalized in educational policy research are often left implicit which can lead to murky relations to larger onto-epistemological questions of how we construct findings from data as well as the nature of policy. In this interpretive analysis, we synthesize a corpus of 37 peer-reviewed journal articles that bring together educational policy and analyses of discourse from varying theoretical and methodological perspectives in order to better understand the breadth and scope of how discourse is defined and operationalized in studies of educational policy, including in ways that are sometimes incommensurate with authors' stated theoretical and methodological positions. After first laying the theoretical groundwork for analyses of discourse in the field of educational policy, we then illustrate how discourse analysis is used differently, and sometimes inconsistently, within contested paradigmatic landscapes. We conclude with an argument for discussions across theoretical frameworks and methodological paradigms about how the concept of discourse lends itself to different epistemological vantage points on educational policy.     

For whom does “what works” work? The political economy of evidence-based education

ABSTRACT

What role does scientific evidence play in educational practice? Supporters of evidence-based education (EBE) see it as a powerful way of improving the quality of public services which is readily applicable to the education sector. Academic scholarship, however, points out important limits to this applicability. I offer an account inspired by Tullock’s theory of bureaucracy that helps explain EBE’s influence despite these limits. Recent configurations of EBE are an imperfect solution to 2 imperatives where policymakers are at an informational disadvantage: (a) guiding professionals working in the field and (b) evaluating evidence from academic researchers. EBE, especially in the form of RCTs and systematic reviews, offers a way of filtering a complex range of research to produce a determinate result that is transparent to policymakers. However, this impression of research transparency is misleading as it omits theoretical background that is critical for successfully interpreting the results of particular interventions. This comes at a cost of relevance to the frontline professionals whom this research evidence is supposed to inform and help.