Separating arguments from conclusions: the mistaken role of effect size in educational policy research

ABSTRACT

Effect size is the basis of much evidence-based education policymaking. In particular, it is assumed to measure the educational effectiveness of interventions. Policy is being driven by the influential work of John Hattie, the Education Endowment Foundation, and others, which is grounded in this assumption. This article demonstrates the assumption is false and notes that, when criticized, proponents either attempt to inoculate themselves by listing (without checking) assumptions or use the specious reasoning that, however flawed their argument, no-one has disproved their conclusions.     

An analysis of Social StoriesTM research using an evidence‐based practice model

The number of Social Stories^TM studies and reviews has increased in recent years, yet concerns regarding quality and effect sizes continue to be expressed. With the emphasis on evidence‐based practices (EBPs) for the education and treatment of people with autism spectrum disorders (ASD), this issue becomes of paramount importance as professionals and parents attempt to select interventions for the people with ASD in their care. The current study makes a unique contribution in its use of an extensive EBP evaluation model to examine 33 single‐subject studies across 13 peer‐reviewed journals, a 12‐year period, and a wide range of grouping variables. Using the Mayton, Wheeler, Menendez and Zhang (2010) EBP evaluation protocol, studies were investigated in terms of eight quality indicators comprised 23 operationally defined standards. Studies included in this analysis met the following criteria: (1) they were intervention studies using single‐subject research designs; (2) they included only participants with disorders on the autism spectrum; and (3) the primary intervention was the use of a Social Story. Findings included on‐ or above‐standard acceptability in EBP indicators related to important aspects of dependent variables within studies and below‐standard acceptability in indicators related to both internal and external validity of studies.     

Education for children with a chronic health condition: an evidence‐informed approach to policy and practice decision making

Children managing chronic health conditions face many obstacles which can impede their learning during periods of hospitalisation. In one particular hospital, a team of educators deemed it necessary to take a personalised learning approach in order to maintain students’ educational progress, namely making use of individual learning plans (ILPs). This team adopted an evidence‐informed practice (EIP) approach to the issue in order to persuade administrators of the need for change. The successful implementation of the EIP approach led to the inclusion of the ILP form in patients’ medical records, which is thought to be a first for Australia. Although EIP is regarded highly by practitioners and policy makers, there can be difficulties when implementing this approach. This study aims to identify the enabling features that permit EIP to be successfully implemented and to examine the ways in which EIP can lead to improved practice.     

Unmasking the unasked: correcting the record about assessor masking as an explanation for effect size differences

Ainsworth et al.’s paper “Sources of Bias in Outcome Assessment in Randomised Controlled Trials: A Case Study” examines alternative accounts for a large difference in effect size between 2 outcomes in the same intervention evaluation. It argues that the probable explanation relates to masking: Only one outcome measure was administered by those aware of participants’ treatment assignment. This paper shows this conclusion is not substantiated by the evidence: The original paper fails to exclude alternative explanations, and what it takes as positive evidence for the preferred explanation is actually negative. While accepting the importance of masking in randomised controlled trials, this paper concludes that the original question was based on a misconception about effect sizes: Seen correctly as a measure of whole study design, the question of effect size difference between different outcome measures does not need asking.     

Single‐subject designs in special education: advantages and limitations

Single‐subject designs provide the special education field with an alternative to group designs. The purpose of this paper is to introduce the topic of single‐subject designs by providing an overview of the relevant literature and a discussion of the major issues and applications of these designs in the field of special education. This paper is divided into four main sections: (1) characteristics of single‐subject designs; (2) types of single‐subject designs; (3) advantages of single‐subject designs; and (4) issues and concerns regarding single‐subject designs. It was concluded that a minimum of three measurements in each phase is typically required until there is a clear pattern. Flexibility and cost‐effectiveness are among the main advantages of these designs. External validity and generalisability are the main concerns. However, meta‐analytic studies can enhance the generalisability of single‐subject designs findings within similar context. Conclusions and recommendations to improve these designs are discussed.     

Evaluating the readiness of special education doctoral students to apply the standards of evidence‐based practice to single‐case research

How well doctoral students in special education are prepared to evaluate research as evidence‐based practice (EBP) is likely to impact their careers, as well as the teachers they will train. In developing a method for evaluating the readiness of small cohort groups of doctoral students to apply a research‐based model of EBP, an instrument and procedure were refined in a pilot evaluation and implemented within a multiple baseline design across participants. Participants’ independent and instrument‐guided performance in rating published research was compared to the ratings of two experts in single‐case research design, yielding proportions of agreement across evaluation conditions. Results indicated group readiness to independently conduct the EBP evaluation and individual differences in readiness indicating the need for remediation.     

Effect sizes as result interpretation aids in single‐subject experimental research: description and application of four nonoverlap methods

Single‐subject experimental research (SSER), one of the most commonly used research methods in special education and applied behaviour analysis, is a scientific, rigorous and valid method to evaluate the effectiveness of behavioural, educational and psychological treatments. However, studies using single‐subject experimental research designs are often excluded from meta‐analyses of evidence‐based practices due to the lack of methodological consensus on the type of effect size indices to be used to determine treatment effect. To promote the use of effect size indices as an adjunct to visual analysis, this article describes four nonoverlap methods (PND, IRD, PEM‐T and Tau‐U) and demonstrates their application to data obtained from studies employing different SSER designs. Advantages and disadvantages of each method are highlighted and considerations for selecting the most appropriate method are provided for researchers and practitioners.     

Of myths and monitoring: learner-centred education as a political project in Scotland

ABSTRACT

Prevalent constructions of best practice at the global level include learner-centred education as an emancipatory and holistic approach across the life course. However, competing discourses of standardisation and preparation for the workforce are also at play. As a small state constructing an image and role for itself on a global stage, Scotland draws aspirationally on learner-centredness in its current Curriculum for Excellence governing education in schools, and in the Statement of Ambition for Adult Education, aligning it with apparently indigenous ideas of good practice in education while distancing it from prevalent patterns in neighbouring England. However, in operationalising these, competing agendas and versions of best practice interrupt these policy narratives and prove difficult to resist. Using a combination of document analysis, observations of consultation processes, and interviews with policy actors, this article explores these tensions in policy content and process.     

WebQuests in special primary education: Learning in a web‐based environment

The present study investigated the differences in learning gain when performing a WebQuest with a well‐defined versus an ill‐defined assignment. Twenty boys and twenty girls (mean age 11; 10), attending a special primary education school, performed two WebQuests. In each WebQuest, they performed either a well‐defined or an ill‐defined assignment. Knowledge acquisition was assessed by means of a concept map (ie, associative knowledge) and a knowledge test, based on facts concerning the subject (ie, factual knowledge). Results indicated that the learning gain on both WebQuests was higher for participants who worked on the ill‐defined assignments. In the long term, factual knowledge remained intact. The results of this study indicated that the use of WebQuest can be a valuable tool to enrich the educational curriculum in special education.     

A better IDEA: best practice and outcomes‐oriented remedies in public impact litigation for students with disabilities

Class action lawsuits filed in violation of Individuals with Disabilities Education Act (IDEA) raise questions for those concerned with improving the education system for students with disabilities. First, do the lawsuits result in changes that can be directly linked to improved student outcomes? Second, do these lawsuits and the ‘consent decrees’ that settle them refer to best practices in educating children with disabilities? To date, no study has examined the remedies ordered as a result of these lawsuits in the disability education context, or proven how different types of remedies can impede or facilitate progress for students with disabilities. Class action lawsuits under IDEA filed in large, urban school districts between 1990 and 2011 and the remedies ordered under the final agreement were reviewed. This review suggests that the lawsuits against large, urban school districts tend to result in remedial activities that focus more on planning, supporting and monitoring than on outcomes or evidence‐based practice.     

Vulnerable children of mentally ill parents: towards evidence‐based support for improving resilience

The increasing prevalence of mental illness among parents always represents a stressor affecting the biopsychosocial development of a child. However, due to varying inherent resilience factors, not all children are affected to the same extent. The presence of evidence‐based resilience factors is able to minimise or prevent the adverse effects associated with the parental disorder. Fostering an attachment to a healthy adult – including professionals in the social field – and providing information about the disorder are regarded as two important preventive factors. The need for specific individualised intervention programmes for this group of forgotten children is highlighted, and a specific method of assessment is presented.     

Journal Editorial Policies Regarding Statistical Significance Tests: Heat Is to Fire as p Is to Importance

The present paper responds to defenses of statistical significance testing offered by Levin and Robinson. First, some inaccurate perceptions of contemporary criticisms of statistical tests are noted. Second, areas of disagreement are explored. For example, it is noted that all nine empirical studies of reporting practices since 1994 show that encouraging (per the 1994 APA style manual) authors to report effect sizes has not worked; two reasons for this failure are explored. Finally, two important areas of agreement regarding needed improvements in contemporary practices are noted.     

Science education as a civil right: Urban schools and opportunity‐to‐learn considerations

In this article I make the case that urban science education is a civil rights issue and that to effectively address it as such we must shift from arguments for civil rights as shared physical space in schools to demands for high‐quality academic preparation that includes the opportunity to learn science. The argument is organized into two sections: first, a review of the school desegregation literature to make the case that urban science education for all is a civil rights issue; and second, an examination and critique of opportunity‐to‐learn literature, including an analysis of three opportunity‐to‐learn constructs to illustrate their potential as civil rights tools in science education. © 2001 John Wiley & Sons, Inc. J Res Sci Teach 38: 1015–1028, 2001     

Critical arts‐based research in education: Performing undocumented historias

The article seeks to elucidate and academically position the genre of critical arts‐based research in education. The article fuses Critical Race Theory (CRT), life history and performance, alongside work with undocumented American students of Mexican origin, to show how a politicised qualitative paradigmatic re envisioning can occur in which counter‐histories and counter‐stories can be co‐created into a powerful, evocative, and transformative arts‐based performance text: Undocumented Historias. The article reflects on how critical arts‐based research in education can function as a means to legitimise, empower and promote the voices of the educationally and socially marginalised; evoking an experiential and sensual means of feeling and knowing by which researcher and researched may co‐recover, interrogate and enrich an anti‐colonialist critique of the dominant social order.     

Generic skills for graduate accountants: the bigger picture,a social and economic imperative in the new knowledge economy

The case for integrating generic skills in university accounting programmes is well documented in the literature, but the implementation of strategies designed to teach generic skills in the context of accounting courses has posed ongoing challenges for academics and course administrators. The imperative for generic skills in accounting programmes derives from an economic view of the role of universities, reflecting the views of government and employers who perceive graduates as economic assets to business and the economy. It is argued that the role of universities extends beyond the economic imperative to encompass a greater social and cultural role. This paper traces the historical evolution of the generic skills discourse with an emphasis on accounting and places it in the broader context of the social and economic roles of universities in an era of transformation in the sector. The generic skills discourse, however, transcends disciplinary boundaries and international borders. The new knowledge economy, emerging as a result of technological advancement, needs graduates across disciplines with flexible mindsets and transferable skill sets, capable of innovating and adapting to a dynamic work environment. Consequently, universities must develop the transferable, generic skills required by graduates to advance their careers and contribute to economic innovation and social development. The generic skills debate must, therefore, be addressed from a systemic perspective, reaching beyond national and disciplinary borders. The lessons learned from the generic skills debate in accounting have wider interdisciplinary application for university policy-makers and educators facing the challenges of a new era in higher education.     

Fifteen years looking at the mirror: On the presence of PISA in education policy processes (Portugal, 2000‐2016)

This article describes and discusses what happens when knowledge for policy generated within PISA is received by its target audience: what have the Portuguese policy actors been doing with PISA data and analysis when they consider, express and justify their choices? Drawing on previous and current studies, using interview materials and formal and informal policy documents, as well as texts published in the written press, the article analyses two main phenomena related to the reception of PISA and how this has evolved between 2001 and 2012 in Portugal: the consolidation of PISA's credibility as a source for policy processes and texts; the emergence of new actors and modes of intervention in the production of knowledge for national policy, drawing on PISA. Finally, it presents an analysis of the reception of PISA 2015 in the Portuguese media, focusing on the interventions by political actors in the Portuguese daily and weekly written press. Two main elements emerge from our content analysis as the main common elements of that reception: the consecration of PISA's credibility; and the practices of qualification and disqualification of educational policies and perspectives. The article concludes by emphasising the regulatory role of PISA in Portuguese policy processes and the relevant contribution played by the politics of reception in legitimising this role.