Detecting Intervention Effects Across Context: An Examination of the Precision of Cluster Randomized Trials

The Institute of Education Sciences has funded more than 100 experiments to evaluate educational interventions in an effort to generate scientific evidence of program effectiveness on which to base education policy and practice. In general, these studies are designed with the goal of having adequate statistical power to detect the average treatment effect. However, the average treatment effect may be less informative if the treatment effects vary substantially from site to site or if the intervention effects differ across context or subpopulations. This article considers the precision of studies to detect different types of treatment effect heterogeneity. Calculations are demonstrated using a set of Institute of Education Sciences funded cluster randomized trials. Strategies for planning future studies with adequate precision for estimating treatment effect heterogeneity are discussed.     

Is college remedial education a worthy investment? New evidence from a sharp regression discontinuity design

To enhance college completion, many institutions have introduced college remedial programs. Yet, till now there is little evidence that this policy helps raise students’ persistence and performance in college. To better understand how to design cost-effective remedial education, this paper studies the impact of an intervention implemented in an undergraduate economics program in Italy. This remedial policy aims at raising students’ effort and performance by combining a short remedial course with the threat of re-enrolling them in the first year in case of a failure in the remedial exam. To estimate causal effects, I implement a sharp regression discontinuity design that exploits the cutoff rule used to assign students to remediation. Results indicate that this nudge-type policy fails to obtain any positive and significant effect on either persistence or performance in college.     

Princesses are bigger than elephants: Effect size as a category error in evidence‐based education

Much of the evidential basis for recent policy decisions is grounded in effect size: the standardised mean difference in outcome scores between a study's intervention and comparison groups. This is interpreted as measuring educational influence, importance or effectiveness of the intervention. This article shows this is a category error at two levels. At the individual study level, the intervention plays only a partial role in effect size, so treating effect size as a measure of the intervention is a mistake. At the meta‐analytic level, the assumptions needed for a valid comparison of the relative effectiveness of interventions on the basis of relative effect size are absurd. While effect size continues to have a role in research design, as a measure of the clarity of a study, policy makers should recognise the lack of a valid role for it in practical decision‐making.     

Statistical Power in Evaluations That Investigate Effects on Multiple Outcomes: A Guide for Researchers

Researchers are often interested in testing the effectiveness of an intervention on multiple outcomes, for multiple subgroups, at multiple points in time, or across multiple treatment groups. The resulting multiplicity of statistical hypothesis tests can lead to spurious findings of effects. Multiple testing procedures (MTPs) are statistical procedures that counteract this problem by adjusting p values for effect estimates upward. Although MTPs are increasingly used in impact evaluations in education and other areas, an important consequence of their use is a change in statistical power that can be substantial. Unfortunately, researchers frequently ignore the power implications of MTPs when designing studies. Consequently, in some cases, sample sizes may be too small, and studies may be underpowered to detect effects as small as a desired size. In other cases, sample sizes may be larger than needed, or studies may be powered to detect smaller effects than anticipated. This paper presents methods for estimating statistical power for multiple definitions of statistical power and presents empirical findings on how power is affected by the use of MTPs.     

Qualified to lead? A comparative,contextual and cultural view of educational policy borrowing

Abstract

Background: Around the globe, education policy borrowing remains pervasive and prevalent. The strategies, interventions and innovations of education systems that perform well, in international assessments, are enthusiastically borrowed and copied in the anticipation of similar educational performance and outcomes.

Purpose: This purpose of the article is to highlight some of the implications and consequences of policy borrowing by comparing the preparation and development programmes for school leaders in various education systems. The article explores the way in which the leadership programmes and models are being adopted and developed by different countries.

Sources of evidence: The article draws upon findings from a contemporary, comparative study of leadership preparation and development across seven different education systems (Seven System Leadership Study). This large-scale, mixed-methods, comparative research study is collecting quantitative and qualitative data about the nature and impact of leadership development and preparation programmes in seven different education systems.

Main argument: The findings from the study reinforce how leadership preparation and development programmes are increasingly becoming standardised as a result of education systems borrowing and adapting from each other. This article also argues that in the contemporary policy discourse, important cultural and contextual influences that significantly affect subsequent implementation and outcomes are increasingly being side-lined and ignored.

Conclusions: This article highlights some of the limitations and unintended consequences of educational policy borrowing. It concludes that adopting some of the design features of effective interventions rather than simply borrowing policies or strategies, in part or in their entirety from very different contexts, may be a more productive way forward.     

Foucault,Docile Bodies and Post-Compulsory Education in Australia

Australian post-compulsory education policies have been subjected to theoretical critiques which question assumptions behind the articulation of policy. One line of critique derives from Foucault's analysis of power, and makes particular use of the theme of ‘docile bodies’. A limitation of these critiques is the adoption of a ‘top-down’ model of policy which fails to test the model against empirical evidence. This article draws upon recent research evidence ‘at the extremities’ to test the applicability of Foucault's ideas within the Australian context. The research challenges the monolithic image that results from a Hop down’ analysis and counteracts untested assumptions about the inevitability of current policies. While it suggests ways in which Foucault's ideas might enables us to lay bare the disjunctures between policy and its plurality in practice, it also highlights the limitations of Foucault's imagery of power.     

Children's rights and educational policy in Europe: the implementation of the United Nations Convention on the Rights of the Child

This article aims to shed light on the impact of the United Nations Convention on the Rights of the Child (CRC) on education policy in Europe. The findings are based on a documentary analysis of the published reports of the Committee on the Rights of the Child (the Committee) on the implementation of the education rights in the CRC in every EU state. This included: a review of the state of children's rights to education in Europe as perceived by the Committee; a summary of the Committee's key recommendations for governments; and an assessment of whether the CRC can be considered to have influenced domestic education law and policies. The findings suggest that the CRC is having an impact on domestic education policy and that the child rights framework could be harnessed further by those seeking to influence government. The article concludes by reflecting on the factors which affect the processes of translating the CRC into policy and practice and explores the role that educationalists, both academic and practitioners, might play in its implementation.     

Shaping education policy research in an Asia-Pacific context

Globalization increasingly calls for comparing educational policies across countries. In this study, we assemble and analyze academic journal publications of the past decade in order to shape education policy research within an Asia-Pacific context. After examining Asia-Pacific research publication data from the Web of Science, we find a few stylized facts: (1) increasing research collaboration, (2) growing policy evaluation research, (3) swelling empirical research with quantitative methodology and (4) growing attention to higher education. Moreover, education stakeholders show exclusive interest on globalization, private tutoring, and language education policies within and across countries. Finally, we find a significant difference in research impact and diffusion between Asia-Pacific and American education policy studies. Further, we examine what determines research impact and diffusion. Our results indicate that research collaboration, national context, publication year and issue, policy area, and research methodology are significantly associated with publication citations. Based on the findings, we suggest useful implications for future directions on education policy research in an Asia-Pacific context.     

Implementing curriculum reform in Wales: the case of the Foundation Phase

Abstract

The Foundation Phase is a Welsh Government flagship policy of early years education (for 3–7 year-old children) in Wales. Marking a radical departure from the more formal, competency-based approach associated with the previous Key Stage 1 National Curriculum, it advocates a developmental, experiential, play-based approach to teaching and learning. The learning country: A paving document (NAfW, 2001) notes that, following devolution, Wales intended to take its own policy direction in order to ‘get the best for Wales’. Building on a three-year mixed methods independent evaluation of the Foundation Phase we discuss in detail the aims and objectives of the Foundation Phase, including the context to its introduction, the theory, assumptions and evidence underlying its rationale, and its content and key inputs. We then contrast this with how the Foundation Phase was received by practitioners and parents, how it has been implemented in classrooms and non-maintained settings, and what discernible impact it has had on young children’s educational outcomes. The paper concludes with a critical analysis of the policy process and identifies a number of contextual issues during the inception of the Foundation Phase that has, it could be argued, constrained its development and subsequent impact. We argue that these constraints are associated with an educational policy landscape that was still in its infancy. In order for future education policy to ‘get the best for Wales’ a number of important lessons must be learnt.     

Women,literacy and health: comparing health and education sectoral approaches in Nepal

Functional adult literacy interventions have been regarded for many decades by policy makers as an effective way of imparting health knowledge. Supported by research on the statistical relationships between women’s literacy rates and health indicators, this dominant policy discourse is based on assumptions that non-literate women lack understanding and confidence, and that formal programmes and institutions constitute the main sites of learning. Proposing a broader conceptualisation of literacy as a social practice and of health as connected with social justice, this article draws on policy analysis and the authors’ earlier research in Nepal to re-examine the relationship between gender, literacy and health. By comparing health and literacy approaches used within the education and health sectors and taking account of new and indigenous informal learning practices, the article points to ways of investigating the complex interaction of factors that influence inequalities in gender and health at community level.     

Comparing Models of Change to Estimate the Mediated Effect in the Pretest–Posttest Control Group Design

Models to assess mediation in the pretest–posttest control group design are understudied in the behavioral sciences even though it is the design of choice for evaluating experimental manipulations. The article provides analytical comparisons of the four most commonly used models to estimate the mediated effect in this design: analysis of covariance (ANCOVA), difference score, residualized change score, and cross-sectional model. Each of these models is fitted using a latent change score specification and a simulation study assessed bias, Type I error, power, and confidence interval coverage of the four models. All but the ANCOVA model make stringent assumptions about the stability and cross-lagged relations of the mediator and outcome that might not be plausible in real-world applications. When these assumptions do not hold, Type I error and statistical power results suggest that only the ANCOVA model has good performance. The four models are applied to an empirical example.     

Achieving quality and equity through inclusive education in an era of high-stakes testing

While great progress has been made by the international community to promote inclusive education for all children, regardless of race, ethnicity, socio-economic status, gender or disability, many countries still continue to marginalize and exclude students in educational systems across the globe. High-stakes assessments in market-driven economies have increased exclusionary practices. Using international databases and research studies, this paper provides evidence of the poor performance of high-stakes assessment policies, particularly in the United States. The authors analyse and compare the key assumptions and consequences of a market-based system of education with those of a system that is based upon the principles of inclusive education through a school-community model and examples from Europe and Latin America. These models demonstrate that the twin goals of quality and equity can be achieved within a system that addresses educational policy and practices more broadly than market-based reforms. Conclusions call for policy-makers to respond to the discrimination and exclusion of various populations around the world by considering the impact of current educational models and the potential they have to support genuinely inclusive education for everyone.     

Evidence,Evaluation and the ‘Tyranny of Effect Size’: a proposal to more accurately measure programme impacts in adult and family literacy

As literacy grows in importance, policymakers’ demands for programme quality grow, too. Evidence on the effectiveness of adult and family literacy programmes is limited at best: research gaps abound, and programme evaluations are more often than not based on flawed theories of programme impact. In the absence of robust evidence on the full range of short- and long-term programme impacts, it is difficult to accurately measure intervention effectiveness. Too frequently, researchers and policymakers focus only on short-term, easily measured outcomes, creating a ‘tyranny of effect size’ that may systematically underestimate impact while simultaneously distorting practice. However, the answer does not lie in turning away from quantitative research. Doing so will consign adult and family literacy to the margins of public policy, when they should be in the mainstream. Longitudinal research from Turkey and the US suggests a need for revised, more subtle theories of how adult literacy and family literacy programmes work, and the diverse ways they benefit participants. By working together more closely and intelligently, researchers, policymakers and practitioners can develop evaluation strategies that more accurately measure programme effects. The key is combining methodological rigour with fully fleshed out theories of literacy development and programme impact.     

Standards-based accountability: reification,responsibility and the ethical subject

Over the last two decades, teachers in Australia have witnessed multiple incarnations of the idea of ‘educational accountability’ and its enactment. Research into this phenomenon of educational policy and practice has revealed various layers of the concept, particularly its professional, bureaucratic, political and cultural dimensions that are central to the restructuring of educational governance and the reorganization of teachers’ work. Today, accountability constitutes a core concept of neoliberal policy-making in education, both fashioning and normalizing what counts as teacher professionalism in the ‘audit society.’ This article focuses specifically on the recent introduction by the Australian Federal Government of standardised literacy testing in all states across Australia, and raises questions about the impact of this reform on the work practices of English literacy teachers in primary and secondary schools. We draw on data collected as part of a major research project funded by the Australian Research Council, involving interviews with teachers about their experiences of implementing standardised testing. The article traces the ways in which teachers’ work is increasingly being mediated by standardised literacy testing to show how these teachers grapple with the tensions between state-wide mandates and a sense of their professional responsibility for their students.     

Progression and Continuity in Early Childhood Education: Tensions and contradictions

This article focuses on the current policy context in early childhood education in England and Wales. The provision of high quality preschool education is regarded internationally as a significant factor in raising educational standards. Such aspirations are dependent on achieving effective patterns of continuity and progression from preschool to school. Despite the centrality of these terms in policy rhetoric progression and continuity are disputed and under‐conceptualized concepts. Furthermore, there is a dearth of empirical studies that explore the relationships between progression in learning and progression in the curriculum. The central aim of this article is to clarify some of the tensions and contradictions inherent in current conceptions of progression and continuity, drawing on perspectives from theory, policy and practice. These perspectives suggest that achieving progression and continuity through curricula which secure high quality outcomes is a difficult and, potentially, elusive aspiration. In theoretical terms, assumptions about progression in learning vary according to different orientations. In policy terms, the current subject‐oriented curriculum framework assumes that progression is an issue of content, and is based on the structures of the subjects. Research evidence indicates that educators in different settings are experiencing problems in providing the conditions which might achieve effective patterns of progression and continuity. The authors identify three key areas for further research in order to advance the development of theories of progression and continuity which might usefully inform policy, practice and curriculum development.     

Effects of resource concentration on research performance

This paper reports the results of a study commissioned by the Australian National Board of Employment, Education and Training, which examines in detail the effect of resource concentration on research performance, and the basis for critical mass, economies of scale, critical time and risk strategy hypotheses. The widespread introduction of policies of resource concentration around the world are found to have been based on little examined assumptions, and in operation to be at times counter-productive. In general relationships between group size and productivity are found to be linear, though there does appear to be evidence for an optimal size of 5–8. Detailed results and policy implications of these findings are presented.     

Higher Education in Further Education: Northern Ireland

This paper reviews the provision of higher education courses in the further education sector in Northern Ireland. It charts the evolution of policy through the 1990s by examining major policy documents. It notes the chronic absence of reliable statistical and research evidence during this period. The available statistical evidence is reviewed. Key themes are identified including: localism, ‘non‐evidence based policy making’, higher education versus the demand for skills and community relations and equality. The innovative concept, now it would appear doomed, of the Springvale campus for west Belfast is examined. Finally, the paper notes that there is an urgent need to provide strategic direction to the further education sector as a whole.     

A review of research on Project STAR and path ahead

Understanding the effects of class size reduction (CSR) has been an enduring issue in education. For the past 3 decades, Project STAR has stimulated research and policy discussions regarding the effects of CSR on a variety of outcomes. Schanzenbach (2007) reviewed STAR studies and concluded that small classes improved student academic outcomes. Since her review, however, some research has challenged this conclusion. This paper reviews these recent challenges, comparing past and recent studies. This review is considered together with studies on large-scale CSR policies in the US and CSR studies outside the US. We conclude by suggesting a direction for future research.     

I Can't Make Heads Or Tails Out Of What You Are Saying,So Let's Just Agree To Be Fair

Assuming a coin is fair is common place in introductory statistical education. This article offers three approaches to test if a coin is fair. The approaches lend themselves to straightforward simulation studies that can enrich student understanding of joint probability and sample size requirements. Simulation studies comparing the relative merits of the three, or potential other, approaches are an example of problem‐based learning.