The use (and misuse) of PISA in guiding policy reform: the case of Spain

In 2013 Spain introduced a series of educational reforms explicitly inspired by the Programme for International Student Assessment (PISA) 2012 results. These reforms were mainly implemented in secondary education – based upon the assumption that this is where Spain's educational problems lie. This paper questions this assumption by attempting to identify the point where Spanish children fall behind young people in other developed countries. Specifically, by drawing data from multiple international assessments, we are able to explore how cross-national differences in reading skills change as children age. Consideration is given to both the average level of achievement and the evolution of educational inequalities. Our conclusion is that policy-makers have focused their efforts on the wrong part of the education system; educational achievement is low in Spain (and educational inequalities large) long before children enter secondary school. This study therefore serves as a note of caution against simplistic interpretation of the PISA rankings.     

Interpreting in mental health settings: issues and concerns

Sign language interpreters in mental health settings face extreme linguistic and cultural difficulties in interpreting basic, everyday language used in these settings. This is particularly true when deaf clients have limited English proficiency, which often requires interpreters to use expansion techniques in order to render messages successfully. To examine how diagnostics may be affected by interpretation, Brauer (1993), Montoya et al. (2001), and Steinberg, Lipton, Eckhardt, Goldstein, and Sullivan (1998) translated two widely used psychological screening instruments into American Sign Language (ASL). The Minnesota Multiphasic Personality Inventory (MMPI) and the Diagnostic Interview Schedule-IV (DIS-IV) were selected for translation, and data from the three studies are presented and discussed. Their implications in terms of the expectations and stresses placed on interpreters are described within a framework of demand and control theory. Finally, sections of the Code of Ethics of the Registry of Interpreters for the Deaf (RID) are examined relative to both the issue of confidentiality and what the interpreter's contribution should be in mental health settings.     

On the supranational spell of PISA in policy

Abstract

Background: PISA results appear to have a large impact upon government policy. The phenomenon is growing, with more countries taking part in PISA testing and politicians pointing to PISA results as reasons for their reforms.

Purpose: The aims of this research were to depict the policy reactions to PISA across a number of jurisdictions, to see whether they exhibited similar patterns and whether the same reforms were evident.

Sources of evidence: We investigated policy and media reactions to the 2009 and 2012 PISA results in six cases: Canada, China (Shanghai), England, France, Norway and Switzerland. Cases were selected to contrast high-performing jurisdictions (Canada, China) with average performers (England, France, Norway and Switzerland). Countries that had already been well reported on in the literature were excluded (Finland, Germany).

Design and methods: Policy documents, media reports and academic articles in English, French, Mandarin and Norwegian relating to each of the cases were critically evaluated.

Results: A policy reaction of ‘scandalisation’ was evident in four of the six cases; a technique used to motivate change. Five of the six cases showed ‘standards-based reforms’ and two had reforms in line with the ‘ideal-governance’ model. However, these are categorisations: the actual reforms had significant differences across countries. There are chronological problems with the notion that PISA results were causal with regard to policy in some instances. Countries with similar PISA results responded with different policies, reflecting their differing cultural and historical education system trajectories.

Conclusions: The connection between PISA results and policy is not always obvious. The supranational spell of PISA in policy is in the way that PISA results are used as a magic wand in political rhetoric, as though they conjure particular policy choices. This serves as a distraction from the ideological basis for reforms. The same PISA results could motivate a range of different policy solutions.     

Using TIMSS and PISA results to inform educational policy: a study of Russia and its neighbours

In this paper, we develop a multi-level comparative approach to analyse Trends in International Mathematics and Science Survey (TIMSS) and Programme of International Student Achievement (PISA) mathematics results for a country, Russia, where the two tests provide contradictory information about students’ relative performance. Russian students do relatively well on the TIMSS mathematics test but relatively poorly on the PISA. We compare the performance of Russian students with different levels of family academic resources over the past decade on these tests compared to students with similar family resources in Russia’s neighbours and to Russian students studying in Latvian and Estonian Russian-medium schools. These comparisons and interviews with educators in Latvia and Estonia help us understand why students in Russia may perform lower on the PISA and to draw education policy lessons for improving international test performance generally and Russian students’ PISA mathematics performance specifically.     

Educational Data Mining: Identification of factors associated with school effectiveness in PISA assessment

With the main goal of identifying the process factors associated with school effectiveness in secondary education, this work presents an innovative methodological proposal. Based on secondary data from the Spanish sample of PISA 2015, high- and low-effectiveness schools were selected by analysing the residuals of the school level (level 2) in multilevel models. Subsequently, decision trees were used to identify the process variables with a greater predictive power for the identification of high- and low-effectiveness schools. While the hierarchical linear models obtained show inter-school variance scores greater than 10%, decision trees achieve a precision greater than 90%. We conclude by analysing the suitability of using decision trees in data originating from large-scale assessments, and examining the obtained factors associated with school effectiveness.     

The danger of inadequate conceptualisation in PISA for education policy

Due to the broad acceptance of the Programme for International Student Assessment (PISA) and other comparative studies as instruments of policymaking, its accuracy is essential. This article attempts to demonstrate omissions in the conceptualisation, and consequently in calculation and interpretation, of one of the central points of PISA 2006 and 2009. The authors point to the danger of such omissions for possible evidence-based education policymaking. With regard to the reproduction of social inequalities through education, we focus on variance in student performance between schools and within schools on the science scale (PISA 2006) and in the mother tongue (PISA 2009). The thesis of the PISA study is that there are countries (Finland, Iceland and Norway) where between only 14 and 29% of the average OECD variance (33%) is attributed to between-school variance – which is good – and there are countries (Germany, Bulgaria, Austria, Hungary, Belgium, Slovenia, the Netherlands, etc.), where the variance between schools is at least ‘one-and-a-half times that of the OECD’ – which is bad. For Slovenia, we demonstrate a significantly different share of variance explained by between-school variance of the same or similar cohort if we move the point of research by only nine months. Our main argument is that such a difference in the share of variance is not the result of a substantial change in the results (grooving difference in the performance) but the result of formal organisational change – the transition of the cohort to upper secondary education. The difference in calculations also radically changes policy implications, which is crucial. While PISA data suggest the necessity of policy measures to reduce between-school variance, our calculations suggest the need for reconsideration of policy measures aimed at the reduction of within-school variance, gender differences, and differences in performance between native and immigrant students.     

Education policy in Poland: The impact of PISA (and other international studies)

The impact of the PISA study on Polish education policy has been significant, but probably different from any other country. Poland has not experienced the so‐called ‘PISA shock’, but its education system has been benefiting considerably from PISA. For experts and policy makers, it has been a useful and reliable instrument that has made it possible to measure the effects of consecutive reforms of the school education system. Moreover, PISA and other international studies have influenced the perception of education policy in Poland. The latter has shifted from an ideology‐driven, centralised policy to an evidence‐informed policy, developed with the involvement of multiple stakeholders, although this has mostly affected the thinking of experts and policy makers rather than the general public. The new government (in power from 2015), following public opinion polls, has reversed most of the previous education reforms, eliminating lower secondary schools introduced in 1999.     

大规模教育质量监测数据再分析中的若干原则及注意点——以PISA 2015数据为例

大规模教育质量监测数据有两个基本特征:采用抽样调查方式、使用多个试题版本。本文以PISA 2015数据为例,提出了利用大规模教育质量监测数据进行再分析的五个原则:优先使用官方数据、比较研究须选择具有可比性的群体、确保样本对目标总体的代表性、再分析时需确保有效样本量、警惕相关分析中选择效应的影响。同时还提出了三个解读数据分析结果时的注意点:重视系统性误差对统计分析结果精度的影响、全方位考察相关关系的成因以及避免机械地解读统计结果。     

The politics of PISA: The media,policy and public responses in Norway and England

Using the PISA 2015 releases in Norway and England, this article explores how PISA has been presented in the media and how the policy level has responded to the results. England will be used as an example for comparison. The article presents early media responses from the 20 most circulated daily newspapers in the two countries and discusses them in relation both to the national PISA reports in Norway and England, as well as the international report of the OECD. The media responses are further interpreted in light of previous research in both countries, with a particular focus upon Norway, where previous Ministers of Education have been interviewed about assessment policy and education reforms.     

PISA,policymaking and political pantomime: education policy referencing between England and Hong Kong

Since the mid-1980s, a number of East Asian societies have consistently performed well in international tests, and their education systems have emerged as models of ‘best practice’, including Hong Kong, which has been extensively referenced by politicians and their advisers in England. In parallel, local dissatisfaction with the education system in Hong Kong has prompted major education reforms. This mismatch between the perceptions of the Hong Kong education system of the two policy communities is explored using documentary analysis and interviews with policymakers and other key stakeholders. We analyse the ways in which features of Hong Kong’s education system are reconstructed and projected in policymaking in England and argue that the referencing to Hong Kong in England is akin to a form of political theatre, reminiscent of a pantomime, with stereotyped villains, heroes and fairy godmothers, narratives of good conquering evil, and comical set-pieces. We argue that these elements provide the means for both constructing and validating simple causal claims and their associated policy actions.     

PISA and its consequences: Shaping education policies through international comparisons

As the field of education has become a highly internationalised policy field in the last decade, international organisations such as the OECD play an ever more decisive role in the dissemination of knowledge, monitoring of outcomes, and research in education policy. Although the OECD lacks any binding governance instruments to put coercion on States or to provide material incentive, it has successively expanded its competences in this field. OECD advanced its status as an expert organisation in the field of education mainly by designing and conducting the international comparative PISA study. With PISA, the OECD was able to greatly influence national education systems. Basically, States were faced with external advice based on sound empirical data that challenged existing domestic policies, politics, and ideas. One prominent case for the impact of PISA is Germany. PISA was a decisive watershed in German education policy‐making. Almost instantly after the PISA results were publicised in late 2001, a comprehensive education reform agenda was put forward in Germany. The experienced reform dynamic was highly surprising because the traditional German education system and politics were characterised by deep‐rooted historical legacies, many involved stakeholders at different levels, and reform‐hampering institutions. Hence, a backlog of grand education reforms have prevailed in Germany since the 1970s. The external pressure exerted by PISA completely changed that situation.     

PISA as an ideational roadmap for policy change: exploring Germany and England in a comparative perspective

Education policy has undergone transformation in many countries over the last decade. In this article, we focus on the effects of the most significant international initiative in secondary education, which is the Organisation for Economic Co-operation and Development's (OECD) Programme for International Student Assessment (PISA). We analyse two countries that provide variation regarding the degree of change in their respective education policy-making due to this study; while Germany substantially reformed its education system in reaction to its mediocre PISA results, almost no change has been observed in England. As we show, alterations and shifts in ideas of education policy best account for such a change.     

Tracing assessment policy discourses in neoliberalised higher education settings

This article explores assessment policy in two European universities with different political, historical and social backgrounds: the University of Glasgow and Tallinn University. The University of Glasgow is a well-established Russell Group university in the UK; Tallinn University is a relatively new university in post-Soviet Estonia, shaped by very recent neoliberalisation processes. By applying a Foucauldian theorisation and Faircloughian methodology, this article approaches assessment policy as not only relating to institutional contexts but also national and global policy environments. The article argues that the assessment policy in Glasgow relates to globally dominant neoliberal discourses of accountability and excellence. These discourses have turned assessment into a complex technology of government that manages educational processes as well as academic and student subjectivities. While Tallinn University is shaped by neoliberalism at strategic levels, the policy documents in Tallinn still indicate a strong sense of local tradition where regulations have a modest impact on academic freedom and assessors’ disciplinary power over students.     

The narrative approach in research and its use for policy advice

ABSTRACT

Recent developments in Europe show that policymakers acknowledge the role of qualitative research in evidence building. A number of qualitative research studies, including narrative inquiry, have been contracted out to gather information into selected policy issues. The management of career transitions and lifelong learning has become a focus of research and policy interest in many countries. Based on the findings of two Cedefop studies, which included the collection of narrative interviews among mid-career adults in several European countries, this article argues in favour of narrative inquiry as a tool to inform policymakers and influences policies, for example in the field of career guidance and counselling and lifelong learning. A selection of narratives is provided as an example to show how unique situated knowledge within the context of mid-career adults’ life is influenced by existing policies or implies that policies need to be changed. By providing snapshots of the findings from these two studies and exemplify their informative value for policy, the article advocates for the usefulness of narrative inquiry. The presentation of qualitative data humanises the commonly found statistical representations of individuals’ life circumstances and enables imagination and meaning making.     

阅读参与度和学习策略对阅读成绩的影响——基于上海PISA2009数据的实证研究

学会学习是终身学习的关键能力,PISA2009从阅读参与度和学习策略两方面来研究学生在多大程度上学会了阅读领域的学习。本文首先介绍PISA2009的主要测评领域阅读中如何测量参与度和学习策略,接着用2009年PISA问卷数据说明中国上海学生在这方面所达到的水平,然后用二层回归模型分析上海学生阅读参与度和学习策略与上海学生阅读成绩的关系。研究发现,元认知策略和对阅读的喜爱程度对上海学生的阅读成绩影响最大,学校整体的参与水平和策略水平对上海学校阅读平均成绩有极显著的影响,提高阅读参与度和学习策略水平能有效克服不利社会经济背景的作用。     

Seeing education with one's own eyes and through PISA lenses: considerations of the reception of PISA in European countries

The paper addresses the Organization for Economic Cooperation and Development's Program for International Student Assessment (PISA) as a public policy instrument, whose worldwide circulation is mediated by processes of reinterpretation, negotiation, and re-contextualization, where national, local, and international agencies intertwine. It is focused on the active reception of PISA in six European spaces (Francophone Belgium, France, Hungary, Portugal, Romania, and Scotland) along its first three cycles. The paper identifies two contrasting developments: the Program's divergent uses and its attractiveness in different social worlds. The paper gives particular attention to what is called the ‘update of reference societies’ in the context of national receptions of PISA. These ‘updates’ are analyzed as part of a composite process that involves domestic reasons, either related to current agendas for education or to deep historical factors, and injunctions related to PISA's rationale and PISA objects.     

Looking East: Shanghai,PISA 2009 and the reconstitution of reference societies in the global education policy field

This paper examines the outstanding performance of Shanghai, China on PISA 2009 and its effects on other national systems and within the global education policy field. The OECD's PISA is helping to create this field by constituting the globe as a commensurate space of school system performance. The effects of Shanghai's success are considered in three other national contexts: the USA, England and Australia. We combine (a) analysis of data from more than 30 research interviews with senior policy actors at the OECD, the IEA and within Australia and England; and (b) document analysis of policy speeches, commissioned research reports and media coverage from the three national contexts. Shanghai's performance in PISA 2009 produced a global ‘PISA-shock’ that has repositioned this system as a significant new ‘reference society’, shifting the global gaze in education from Finland to the ‘East’ at the beginning of the so-called ‘Asian century’.