Policy,philanthropy and profit: the OECD’s PISA for Schools and new modes of heterarchical educational governance

This paper examines the development and administration of Programme for International Student Assessment (PISA) for Schools – a new testing instrument of the Organisation for Economic Cooperation and Development – to demonstrate the relevance of heterarchical processes to educational governance. Drawing suggestively across new ‘relational’ thinking around policy networks, and new spatialities associated with globalisation, the research shows how PISA for Schools helps constitute new spaces and relations of, and for, educational governance. Informed by policy documents and interviews conducted with 33 key actors across the PISA for Schools policy cycle, I show how PISA for Schools typifies contemporary educational policy-making and governance via the export of ‘statework’ to private actors and agencies, including intergovernmental organisations, philanthropic foundations and edu-businesses. I conclude by considering how treating PISA for Schools, and other similar education services, as a ‘product’ produces a potentially dangerous blurring of public and private benefits, with the potential that (private) profit might ultimately trump (public) education.     

FROM PISA TO EDUCATIONAL STANDARDS: THE IMPACT OF LARGE-SCALE ASSESSMENTS ON SCIENCE EDUCATION IN GERMANY

The German education system does not traditionally rely on standardized testing. However, when the Programme for International Student Assessment (PISA) study revealed an average performance of German students compared to other participating countries, a particular proportion of low-performing students, and remarkable disparities between the federal states, German policy makers decided for a major reform of the education system. A core piece of this reform was the introduction of National Education Standards. For science education, these standards were heavily influenced by the PISA results and its underlying framework. That is, with the standards, a paradigm shift took place from the German notion of Bildung towards the Anglo-American notion of literacy. With the introduction of these standards, a new field of empirical educational research was created: research on models of scientific literacy or competency models as a basis of benchmarking the standards. This article describes the German education system before PISA, summarizes the major findings from PISA, and describes how these findings informed the formulation of the performance standards for science education. It also details the measures undertaken to benchmark these standards. Finally, it provides insight into the issues with developing and benchmarking performance standards and points out future areas of research on evidence-based decision making in educational policy.     

THE RELEVANCE AND CONSEQUENCES OF PISA SCIENCE IN A DANISH CONTEXT

This article reports a case study that described and analyzed the changes in the Danish school culture induced and encouraged by the Programme for International Student Assessment (PISA) results. The educational policy and reforms that were temporally connected with the publication of the PISA 2000 results are outlined and the related socioeconomic and sociopolitical influences are explicated. Furthermore, we investigated to what degree the PISA science assessment framework and test system were in accordance with the Danish educational goals in science in order to discuss the relevance of PISA as a catalyst for the educational actions taken. The results of our inquiry revealed areas of good correspondence and fundamental differences related to values underlying the Danish school system and PISA, respectively (e.g., Bildung orientation versus cognitive skills/competency orientation, different learning/assessment paradigms). We argue that such differences are crucial when considering curricular relevance, validity, and the use of PISA as an agent of change on the national level.     

专业视野中的PISA

近来,国际学生评估项目(PISA)备受国内外关注。PISA项目缘于发达国家对基础教育质量的反思,是一项以改善教育政策为导向的跨国测试研究。本文解读了PISA测试独创的素养及其内涵结构,探究了PISA研究试图解释的学生素养形成与学生个人参与及学习策略间的关系,分析了PISA研究试图发现的学生素养与家庭影响、学校均衡、教育公平的关系。本文认为,公平而卓越正在成为发达国家和国际组织衡量基础教育质量的新尺度;多维度、多学科、重数据与国际参照已经成为教育研究与教育评价的重要发展方向;教育政策的制定应该建立在研究、数据和证据的基础上。     

Steering the National: Exploring the Education Policy Uses of PISA in Spain

This article presents findings from a recent study of the education policy uses and impact of international large-scale assessments, namely the Programme for International Student Assessment (PISA). The paper focuses on two overlapping dimensions of PISA’s education policy use in the context of Spain. These include political dimensions, such as the use of PISA to initiate and justify the 2012?2013 educational reforms; and technical dimensions, namely the use of PISA in the development of national-level indicators used to benchmark progress and guide education and curriculum reform. The study points to the growing dominance of PISA as a powerful policy tool. Findings from the paper add to the body of literature on the different ways in which international assessments are used to guide education policy within national spaces and the role of the OECD as an agent of transnational policy steering.     

Policy impact of PISA on mathematics education: the case of Norway

This article addresses the policy implications of participation in international large-scale assessments (ILSAs), particularly the Programme for International Student Assessment (PISA), and the ways in which such implications might influence mathematics education. Taking Norway as a special case, this discussion focuses on insights into teaching, learning and assessment practices that can be inferred from the PISA study, and how participation in ILSAs has contributed to educational policy and even changed policymakers’ perspectives on schools, teachers and students. Following publication of the PISA 2000 results, Norway experienced a ‘PISA shock’, leading to the implementation of a national quality assessment system and national tests. In addition, changes were made to the mathematics curriculum for compulsory school and to mathematics teacher education. More recently, public debate has focused less on rank and league tables, shifting instead to the high number of low-achieving students and the low number of high achievers. Moreover, there has been little uptake of policy advice provided by the Organisation for Economic Co-operation and Development (OECD), which focuses on strengthening accountability measures. Furthermore, although the Norwegian educational system in the past decade has undergone a decentralisation process, the educational system still follows the Nordic model, which focuses on equity and ‘education for all’. Analyses of the Norwegian case indicate that policymaking takes place in highly cultural contexts, and that international studies might be used merely to validate existing policy directions.     

FIRST CYCLE OF PISA (2000–2006)—INTERNATIONAL PERSPECTIVES ON SUCCESSES AND CHALLENGES: RESEARCH AND POLICY DIRECTIONS

This special edition of IJMSE focuses on the Programme of International Student Assessment (PISA) project now that it has completed a full cycle of administration—reading, mathematics, and science—to look at ways in which PISA has been used in participating countries and with what consequences, and to identify potential research and policy directions emanating from this initiative. Articles were invited to (a) reflect international perspectives on the uses and consequences of PISA to date and (b) speculate on future directions for research, curriculum, and policy using the PISA datasets. The introductory article provides a brief overview of common aspects of PISA: Evolving definitions of reading literacy, mathematics literacy, and science literacy; technical design of the instruments and data analysis procedures; the changing emphasis of administrations; and recent research using the datasets. PISA, unlike other international assessments in reading, mathematics, and science, has provided a fresh perspective on ‘what might be’ by decoupling the assessment from mandated curricula to focus on literacies needed for a 21st century economy. This unique feature of PISA brings with it possibilities and cautions for policy makers.     

Die PISA-Studien: Ihre Präsentation und Interpretation im Lichte der Evidenzbasierung

This paper investigates the presentation and interpretation of PISA results by educational researchers, whereby the topics Development of student performance in international comparison, Social background and competence acquisition, Acquisition of competencies of students with a migration background and Identification of a risk group were selected for analysis. The findings indicate that the presentation and interpretation of PISA results by educational researchers do not provide a solid basis for evidence-based educational policy: the researchers present different and partially contradictory interpretations, as well as various and partial incompatible recommendations. Moreover, particular results are not robust, and concerning some research areas the embedding into competing findings is not offered. In view of this analysis, the paper closes with a skeptical appraisal regarding the concept of evidence-based educational policy.     

基于PISA测试结果的教育政策调整分析

PISA测试结果是影响参加国或地区调整教育政策的重要因素之一。这些影响主要包括公众对教育质量的关注及其形成的舆论压力,表现优异的参加国或地区可借鉴的成功经验等。研究发现,基于PISA测试结果的教育政策调整类型主要有四种:政策深化型、政策变革型、政策调适型和政策效仿型。     

PISA and high-performing education systems: explaining Singapore’s education success

Singapore’s remarkable performance in Programme for International Student Assessment (PISA) has placed it among the world’s high-performing education systems (HPES). In the literature on HPES, its ‘secret formula’ for education success is explained in terms of teacher quality, school leadership, system characteristics and educational reform. This article offers an alternative explanation for the education success of Singapore and, in so doing, questions the basic assertions of the HPES literature and, in particularly, the use of PISA results as the prime indicator of the educational performance of a school system. The explanation is informed by a historical perspective on the development of the Singapore education system and based upon a body of empirical findings on the nature of pedagogical practice in classrooms, both of which are vital for understanding the educational performance of Singapore’s education system. The article concludes by addressing the implications of this analysis for educational policy borrowing.     

Counting and comparing school performance: an analysis of media coverage of PISA in Australia, 2000–2014

This paper empirically documents media portrayals of Australia’s performance on the Program for the International Student Assessment (PISA), 2000–2014. We analyse newspaper articles from two national and eight metropolitan newspapers. This analysis demonstrates increased media coverage of PISA over the period in question. Our research data were analysed using ‘framing theory’, documenting how the media frames stories about Australia’s performance on PISA. Three frames were identified: counts and comparisons; criticisms; and contexts. Most of the media coverage (41%) was concerned with the first frame, counts and comparisons, which analysed PISA data to provide ‘evidence’ that was then used to comparatively position Australia against other countries, reference societies, which do better, with particular emphasis on Finland and also Shanghai after the 2009 PISA. The other two frames dealt with criticisms and contextual issues. This paper only focuses on the first frame. The analysis demonstrates the ways in which media coverage of Australia’s PISA performance has had policy impact.     

Seeing education through the prism of PISA

International comparisons are never easy and they are not perfect. But PISA shows what is possible in education and it helps countries to see themselves in the mirror of student performance and educational possibilities in other countries. This article summarises key policy insights from PISA. It highlights how excellence and improving equity need not be conflicting policy objectives, but that they tend to be jointly achieved only when deliberate policies are in place that match resources with needs and when stratification and grade repetition are contained. The article also shows how a number of countries have been able to raise learning outcomes and moderate the impact of social background in the last decade and highlights some of the policies and practices that characterise these countries.     

How not to reason with PISA data: an ironic investigation

International comparative testing, such as the OECD Programme for International Student Assessment (PISA), has considerable impact on policy-makers, the media and the general public. A central assumption underlying PISA is that global variation in students’ academic performance is attributable to national educational structures and policies. The aim of this article is to demonstrate the problematic nature of this assumption. Rather than critiquing it from the outside, we turn the tools, data and presumptions of the current discourse of international comparisons upon themselves, showing that this assumption is refuted by analysis of immigrant student test scores. Data from the PISA 2009 tests show that Chinese immigrant students in New Zealand and Australia achieve math scores that are more similar to those of students in Shanghai than to their non-immigrant Australian and New Zealand peers. Thus, cultural background appears to be more consequential for the educational attainment of Chinese immigrant students than exposure to the educational systems of Australia or New Zealand. We discuss limitations of our analysis, using them as basis to discuss the shortcomings of PISA more generally.     

Chinese responses to Shanghai’s performance in PISA

This article analyses the public responses in China to Shanghai’s performance in the 2012 Programme for International Student Assessment (PISA). Based on data obtained from media accounts and other materials published between 2013 and 2016, the research findings show that the responses in China are generally reflective, measured and self-critical. Drawing upon Gadamer’s notions of ‘tradition’, ‘horizon’ and ‘prejudice’, this paper contends that the responses reflect the prevailing worldviews in China that perceive Shanghai’s education system to be academically rigorous but too exam-oriented and burdensome. It is further argued that Confucian knowledge traditions and structures in China shape the Chinese interpretations of the PISA assessment format, leading them to downplay Shanghai’s success. This study introduces the metaphor of ‘triadic eyes’ and highlights the mediating effects of the ‘local eyes’ of policy actors in an era of global educational governance.     

The OECD and the local: PISA-based Test for Schools in the USA

The Organization for Economic Cooperation and Development (OECD) has recently piloted the Program for International Student Assessment (PISA)-based Test for Schools in the USA. In this paper, I contend that by connecting directly with local school boards this new initiative has the potential to further promote the OECD's educational agenda in local policy debates. I begin to develop this argument by providing an overview of the OECD and its work. I then lay out a theoretical framework around global governance and knowledge production within the context of the OECD. Next, I provide a brief overview of the traditional PISA study and compare it to the new PISA-based Test for Schools initiative. This context provides the foundation for a discussion of the ways in which a school-based international assessment can operate as a governance tool, allowing international organizations to have greater influence in the formation and implementation of local educational policies.     

Fifteen years of research on PISA effects on education governance: A critical review

This article provides a literature review on the effects of the OECD's Programme for International Student Assessment (PISA) on education governance and policy process across participating countries. This review seemed necessary because there has been a growing body of literature on this topic since 2003, especially since 2010, because this literature is not always well‐known and because the discourse on the so‐called ‘PISA shock’ remains important, even if it is more of a metaphor than a concept and may be politically partial. The article exploits a dataset of 87 references which show that PISA introduced major changes in the governance of education worldwide. Driven by soft power strategies and new policy transfers, this governance is based on data and measurement tools which redefine the scales of education policies. It also shows that PISA has a strong influence on a variety of national reforms, as illustrated in many case studies. However, this influence strongly depends on domestic policy contexts that scholars intended to capture through different theoretical frameworks. Nonetheless, few propose overarching theorisations of the political meaning of PISA effects on education governance and policy processes. The article concludes by stressing three main challenges for the subsequent studies on these PISA effects: better conceptualising these effects, preserving an epistemology of uncertainty in order to avoid taken for granted views and normalising the research on PISA effects not to perpetually and artificially rediscover its so‐called novelty.     

PISA Data: Raising concerns with its use in policy settings

This article considers the role played by policy makers, government organisations, and research institutes (sometimes labelled “think tanks”) in the analysis, use and reporting of PISA data for the purposes of policy advice and advocacy. It draws on the ideas of Rizvi and Lingard (Globalizing Education Policy, 2010), Bogdandy and Goldmann (Governance by Indicators/ Global Power through Quantification and Rankings, 2012) and others to explore the ways in which such “agents of change” can interpret, manipulate and disseminate the results of data arising from large scale assessment survey programs such as PISA to influence and determine political and/or educational research agendas. This article illustrates this issue by highlighting the uncertainty surrounding the PISA data that have been used by a number of prominent, high profile agents of change to defend policy directions and advice. The final section of this paper highlights the need for policy makers and their advisors to become better informed of the technical limitations of using international achievement data if such data are to be used to inform policy development and educational reforms.     

The uses of the national PISA results by Finnish officials in central government

In this article we attempt to analyse how OECD knowledge production is integrated with the process in which Finnish education policy takes shape. This is done by analysing the uses of the OECD PISA Study by Finnish central government officials. The main question posed is: How do these officials interpret the PISA results so as to justify the decisions made in Finnish education policy in the past or to point out new areas of development concerning basic education? The analysis shows that the interpretations of the PISA results tend to favour those responsible for actions within the central government. In the texts analysed, the scientificity of the PISA programme is presented as beyond question, while the direct usefulness of the research results for the further development of national education is also proclaimed. As to the specific results of PISA, the excellent learning outcomes of Finnish students are claimed to be due to educational reforms conducted and decisions made by the central government, whereas shortcomings and areas in which the officials see a need for improvement are argued to be dependent on the actions of other agents. Thus, the analysis shows that the conclusions drawn from the PISA results in texts representing the views of central government are biased and serve to justify its policy agenda.     

PISA and policy-borrowing: A philosophical perspective on their interplay in mathematics education

Mathematics achievement in different education systems around the world is assessed periodically in the Programme for International Student Assessment (PISA). PISA is deemed to yield robust international comparisons of mathematical attainment that enable individual countries and regions to monitor the performance of their education systems relative to standards being achieved internationally, with a view to informing their mathematics education policy decisions. Initially, the role of PISA in instigating mathematics education policy borrowing is outlined using England as a case study, and some existing technical critiques of PISA are then reviewed. Following this, aspects of Ludwig Wittgenstein’s later philosophy of mind are used to reason that an over-reliance on the use of PISA to inform policy decisions in mathematics education may be problematic. It is suggested that, when PISA is viewed through a later Wittgensteinian lens, a potential deficiency in the underpinning psychometric model, pertaining to the inherent indeterminism in unmeasured mathematical abilities, may weaken PISA’s utility in guiding mathematics education policy decisions. It is concluded that, whilst PISA mathematics scores may give some indication of the mathematical proficiency of a nation’s students, caution is required before mathematics education policies are borrowed from other jurisdictions on the basis of PISA performance. Implications for the other PISA domains are also outlined.     

The existence and impact of floor effects for low-performing PISA participants

ABSTRACT

Modern international studies of educational achievement have grown in terms of participating educational systems. Accompanying this development is an increase in heterogeneity, as more and different kinds of educational systems take part. This growth has been particularly pronounced among low-performing, less economically developed systems. Although studies such as PISA have made modifications to account for increased diversity, the degree to which international assessments serve educational systems at the lower ends of the achievement continuum is understudied. We used modified Wright maps and PISA’s definition of proficiency to evaluate the fitness of PISA, especially among low performers. Our findings suggest that there is mismatch between some populations and PISA. Results from a simulation show that such disparities produced biased achievement estimates and correlations with policy relevant variables. Projected PISA growth and new instantiations of PISA, particularly geared toward developing educational systems, make these findings timely and especially relevant.