PISA 2012: how do results for the paper and computer tests compare?

The Programme for International Assessment (PISA) is an important cross-national study of 15-year olds academic achievement. Although it has traditionally been conducted using paper-and-pencil tests, the vast majority of countries will use computer-based assessment from 2015. In this paper, we consider how cross-country comparisons of children’s skills differ between paper and computer versions of the PISA mathematics test. Using data from PISA 2012, where more than 200,000 children from 32 economies completed both paper and computer versions of the mathematics assessment, we find important and interesting differences between the two sets of results. This includes a substantial drop of more than 50 PISA test points (half a standard deviation) in the average performance of children from Shanghai-China. Moreover, by considering children’s responses to particular test items, we show how differences are unlikely to be solely due to the interactive nature of certain computer test questions. The paper concludes with a discussion of what the findings imply for interpretation of PISA results in 2015 and beyond.     

PISA Country Rankings for Italy and Spain: Revised Results

This article examines whether the way that PISA models item outcomes in mathematics affects the validity of its country rankings. As an alternative to PISA methodology, a two-parameter logistic model is applied to PISA mathematics item data from Italy and Spain for the year 2009. In the estimation procedure, item difficulty and dispersion parameters were allowed to differ across the two countries and samples were restricted to respondents who actually answered items in a mathematics cluster. Different normalizations for identifying the distribution parameters were also considered. The choice of normalization is shown to be crucial in guaranteeing certain invariance properties required by item response models. The ability or proficiency scores obtained from the methods employed here are significantly higher for Spain, in sharp contrast to PISA results, which gave both countries virtually the same rank in mathematics (489 for Italy and 488 for Spain). These results raise serious questions about PISA methodology and the role that PISA results play in the formulation educational policy in member countries.     

Counting and comparing school performance: an analysis of media coverage of PISA in Australia, 2000–2014

This paper empirically documents media portrayals of Australia’s performance on the Program for the International Student Assessment (PISA), 2000–2014. We analyse newspaper articles from two national and eight metropolitan newspapers. This analysis demonstrates increased media coverage of PISA over the period in question. Our research data were analysed using ‘framing theory’, documenting how the media frames stories about Australia’s performance on PISA. Three frames were identified: counts and comparisons; criticisms; and contexts. Most of the media coverage (41%) was concerned with the first frame, counts and comparisons, which analysed PISA data to provide ‘evidence’ that was then used to comparatively position Australia against other countries, reference societies, which do better, with particular emphasis on Finland and also Shanghai after the 2009 PISA. The other two frames dealt with criticisms and contextual issues. This paper only focuses on the first frame. The analysis demonstrates the ways in which media coverage of Australia’s PISA performance has had policy impact.     

Is the United States losing its status as a reference point for educational policy in the age of global comparison? The case of South Korea

This paper seeks to contribute to recent comparative discussions about the shift of traditional referential points as a result of new global governance by the OECD through PISA. In doing so, the authors investigate whether the lower PISA rankings of the US have resulted in the shifting of its referential status in South Korea. For the purposes of analysis, media representation of US education in South Korea is analysed by using two disparate newspapers from two time periods: three years before the first PISA release in 2001 and three years after 2009. This paper uses media discourses as primary data, but it also considers other complementary data such as policy documents and government policy statements on education. Recently, global governance through comparative data has become more significant, but the results of this study suggest that one should be careful about oversimplification. The results of this study imply that the pattern of external referencing in media representation hinges on historical, political, and cultural experiences rather than purely evidence-based discourses, at least in the case of South Korea.     

Leaning too far? PISA,policy and Australia's ‘top five’ ambitions

Australia has declared its ambition to be within the ‘top five’ in the Programme for International Student Assessment (PISA) by 2025. So serious is it about this ambition, that the Australian Government has incorporated it into the Australian Education Act, 2013. Given this focus on PISA results and rankings, we go beyond average scores to take a close look at Australia's performance in PISA, examining rankings by different geographical units, by item content and by test completion. Based on this analysis and using data from interviews with measurement and policy experts, we show how uninformative and even misleading the ‘average performance scores’, on which the rankings are based, can be. We explore how a more nuanced understanding would point to quite different policy actions. After considering the PISA data and Australia's ‘top five’ ambition closely, we argue that neither the rankings nor such ambitions should be given much credence.     

Fifteen years looking at the mirror: On the presence of PISA in education policy processes (Portugal, 2000‐2016)

This article describes and discusses what happens when knowledge for policy generated within PISA is received by its target audience: what have the Portuguese policy actors been doing with PISA data and analysis when they consider, express and justify their choices? Drawing on previous and current studies, using interview materials and formal and informal policy documents, as well as texts published in the written press, the article analyses two main phenomena related to the reception of PISA and how this has evolved between 2001 and 2012 in Portugal: the consolidation of PISA's credibility as a source for policy processes and texts; the emergence of new actors and modes of intervention in the production of knowledge for national policy, drawing on PISA. Finally, it presents an analysis of the reception of PISA 2015 in the Portuguese media, focusing on the interventions by political actors in the Portuguese daily and weekly written press. Two main elements emerge from our content analysis as the main common elements of that reception: the consecration of PISA's credibility; and the practices of qualification and disqualification of educational policies and perspectives. The article concludes by emphasising the regulatory role of PISA in Portuguese policy processes and the relevant contribution played by the politics of reception in legitimising this role.     

Response to 'Cautions on OECD's Recent Educational Survey (PISA)'

In his recent paper, 'Cautions on OECD's recent educational survey (PISA)' ( Oxford Review of Education , 29, 2), S.J. Prais questioned the outcomes of the Organisation for Economic Cooperation and Development's PISA survey of the reading, mathematics and science attainments of 15-year-olds. Prais suggested that methodological flaws in PISA had resulted in an apparent improvement in the attainment of British students--particularly when compared to their Swiss and German counterparts. This paper responds to Prais's criticisms, noting that when Prais's conjectures are tested with empirical data they are not supported. Further it is noted that many of Prais's criticisms are due to an incomplete understanding and knowledge of the methodology of international studies, and of PISA in particular.     

‘Education is a life marathon rather than a hundred-meter race’: Chinese ‘folk’ comparative discourses on Finnish education

Over the past decade Finnish education has been praised worldwide for its students’ ‘amazing’ results in the OECD PISA studies. Thousands of pedagogical tourists – including policy-makers, researchers and educators – have visited the country to find out about the reasons behind the success and to borrow, often uncritically and un-reflexively, Finnish practices that can help them to become ‘good performers’ too. This has resulted in what we call ‘folk’ comparative discourses on Finland. China is no exception to the rule. In this article we examine a range of books about Finnish education published in the Middle Kingdom (China) for a general rather than narrowly specialist readership. We are interested in how these volumes construct certain images and myths about it and what these tell us about how the authors view Chinese education but also current societal discussions about it. Our approach is based on critical and reflexive interculturality.     

The role that mathematics plays in college- and career-readiness: evidence from PISA

Many studies have found a strong relationship between the mathematics students study in school and their performance on an academic or school mathematics assessment but not on an assessment of mathematics literacy (ML). With many countries, like the USA, placing emphasis on finishing secondary education being mathematically literate and prepared for college or career, this raises the question about the relationship between the mathematics studied in school and any ML students may have. The Programme for International Student Assessment (PISA) ML assessment is embedded in real-world contexts that provide an important window on how ready students are to tackle the situations and problems that await them whether they intend to pursue further education beyond high school or intend to go directly into the labour force. In this paper, we draw upon the PISA 2012 data to investigate the extent to which the cumulative exposure to rigorous mathematics content, such as that embedded in college- and career-ready standards, is associated with ML as assessed in PISA. Results reveal that both exposure to rigorous school mathematics and experiencing the instruction of this mathematics through real-world applications are significantly related to all the real-world contextualized PISA ML scores.     

PISA Country Rankings Valid? Results for Canada and Finland

This article examines whether the way that PISA models item outcomes in mathematics affects the validity of its country rankings. As an alternative to PISA methodology a two-parameter model is applied to PISA mathematics item data from Canada and Finland for the year 2012. In the estimation procedure item difficulty and dispersion parameters are allowed to differ across the two countries and samples are restricted to respondents who actually answered items in a mathematics cluster. Different normalizations for identifying the distribution parameters are also considered. The choice of normalization is shown to be crucial in guaranteeing certain invariance properties required by item response models. The ability scores obtained from the methods employed here are significantly higher for Finland, in sharp contrast to PISA results, which gave both countries very similar ranks in mathematics.     

The PISA calendar: Temporal governance and international large-scale assessments

Abstract

This article analyses international large-scale assessments in education from a temporal perspective. The article discusses and compares the different conceptions of time in the early international assessments conducted in the 1960s and 1970s by the IEA with the PISA studies conducted by the OECD from the year 2000 onwards. The paper argues that there has been a shift in the ways that the assessments structure time. The early IEA surveys were characterized by a relative slowness, lack of synchronization and lack of trend analyses. PISA, by contrast, is characterized by high pace, simultaneous publication of results around the world and regular and recurrent studies making the analysis of trends possible. The emergence of this new time regime, it is argued, has implications for how education is governed. At the transnational level, it strengthens the influence and importance of OECD as a significant policy actor. At the national level, as educational discourse and policy adapts to the temporalities of the PISA calendar, two kinds of effects can be distinguished. First, there is a tendency towards searching for “retrotopian” solutions for contemporary problems. Second, there is a tendency towards acceleration and short-term planning when it comes to educational reforms.     

Excellence through Special Education? Lessons from the Finnish School Reform

The present article focuses on connections between part-time special education and the good results of Finnish students in PISA studies. After a brief summary of the comprehensive school system and special education in Finland, PISA results are analysed. The analysis shows that the relative amount of special education targeted at language problems is highest in Finland among those countries from which comparative statistics are available. The writers argue that this preventive language-oriented part-time special education is an important factor behind the good PISA results.     

Explaining the difference between PISA 2009 reading scores in Finland and Estonia

The aim of the study was to explain the difference between the Programme for International Student Assessment (PISA) 2009 reading results for Finland and Estonia using characteristics of teaching and learning, and characteristics of the overall development of these countries. PISA data were collected via a reading test and student questionnaires from 4,729 students in Estonia and 5,810 students in Finland. Regression analysis made it possible to identify the speed of the rise in PISA scores in relation to the selected variables. The speed was multiplied by the value of the variable to calculate the effect of the variable. The effects of the joy of reading and the diversity of reading materials were greater in Finland, but the effects of metacognition and online reading activities were greater in Estonia. The countries had different values for several indices of development, and this was concordant with the difference in the PISA scores.     

Students’ test motivation in PISA: the case of Norway

Do students make their best effort in large-scale assessment studies such as the Programme for International Student Assessment (PISA)? Despite six cycles of PISA surveys from 2000 to 2015, empirical studies regarding students’ test motivation and experience of the tests are sparse. The present study examines students’ test motivation in PISA, and how closely students’ achievement correlates with students’ reported test motivation. A total of 40 students from eight schools that participated in the PISA study in 2006, 2009 and 2012 were interviewed. In addition, questionnaire data from a total of 9400 students who participated in PISA 2009 and 2012 were collected. The findings of this study indicate that students overall were motivated to do their best in the PISA study, despite the fact that these low-stakes tests have no impact on students’ grades or future school entrance. In contrast to claims in the media, where it has been suggested that Norwegian students are more relaxed towards the PISA test than students in other countries, students report that they are motivated to do their best, but girls report significantly higher test motivation than boys. Despite the policy influence of PISA and the large number of countries participating, very few studies have reported findings from students’ perspectives regarding their test motivation and experience of the test. The present study aims to fill that gap.     

Competitive comparison and PISA bragging rights: sub-national uses of the OECD's PISA in Canada and the USA

Participation in the Organization for Economic Co-operation and Development's (OECD) Program for International Student Assessment (PISA) has continuously expanded: from 43 systems in 2000 to 65 systems in the 2012 cycle, with 71 signed up for PISA 2015. There also has been a growth in sub-national participation, expanding PISA's reach beyond the nation-state. This paper explores sub-national PISA participation in Canada and the USA, asking how PISA is being used within sub-national education policy spaces. We draw on analysis of documents and data from interviews with officials at sub-national, national, and international levels. Findings illustrate some of the diverse motivations and uses of PISA, providing insights into the effects of PISA at the sub-national scale. As such, we argue that competitive comparison in education has deepened through the enhanced granularity of international large-scale assessment data to new scales beyond the nation-state.     

FIRST CYCLE OF PISA (2000–2006)—INTERNATIONAL PERSPECTIVES ON SUCCESSES AND CHALLENGES: RESEARCH AND POLICY DIRECTIONS

This special edition of IJMSE focuses on the Programme of International Student Assessment (PISA) project now that it has completed a full cycle of administration—reading, mathematics, and science—to look at ways in which PISA has been used in participating countries and with what consequences, and to identify potential research and policy directions emanating from this initiative. Articles were invited to (a) reflect international perspectives on the uses and consequences of PISA to date and (b) speculate on future directions for research, curriculum, and policy using the PISA datasets. The introductory article provides a brief overview of common aspects of PISA: Evolving definitions of reading literacy, mathematics literacy, and science literacy; technical design of the instruments and data analysis procedures; the changing emphasis of administrations; and recent research using the datasets. PISA, unlike other international assessments in reading, mathematics, and science, has provided a fresh perspective on ‘what might be’ by decoupling the assessment from mandated curricula to focus on literacies needed for a 21st century economy. This unique feature of PISA brings with it possibilities and cautions for policy makers.     

Fifteen years of research on PISA effects on education governance: A critical review

This article provides a literature review on the effects of the OECD's Programme for International Student Assessment (PISA) on education governance and policy process across participating countries. This review seemed necessary because there has been a growing body of literature on this topic since 2003, especially since 2010, because this literature is not always well‐known and because the discourse on the so‐called ‘PISA shock’ remains important, even if it is more of a metaphor than a concept and may be politically partial. The article exploits a dataset of 87 references which show that PISA introduced major changes in the governance of education worldwide. Driven by soft power strategies and new policy transfers, this governance is based on data and measurement tools which redefine the scales of education policies. It also shows that PISA has a strong influence on a variety of national reforms, as illustrated in many case studies. However, this influence strongly depends on domestic policy contexts that scholars intended to capture through different theoretical frameworks. Nonetheless, few propose overarching theorisations of the political meaning of PISA effects on education governance and policy processes. The article concludes by stressing three main challenges for the subsequent studies on these PISA effects: better conceptualising these effects, preserving an epistemology of uncertainty in order to avoid taken for granted views and normalising the research on PISA effects not to perpetually and artificially rediscover its so‐called novelty.     

International assessments of student achievement and public confidence in education: evidence from a cross-national study

One of the overarching goals of international large-scale assessments (ILSA) is to inform public discourse about the quality of education in different countries. To fulfil this function, the Organisation for Economic Co-operation and Development (OECD), for example, raises awareness of the Program for International Student Assessment (PISA) results through different forms of traditional and social media (e.g. press releases and other activities under the slogan PISA Day). Scholars have responded to the rapid growth of ILSA by examining public discourse through newspaper articles, policy documents, and other outlets. However, we know very little about whether and to what extent the general public is actually affected by PISA results. In order to address this gap, this study uses data regarding public trust in education from the 2011 wave of the International Social Survey Program (ISSP). Drawing on survey data from 30 countries and Hierarchical Linear Models (HLM), the study shows that PISA rankings have a significant effect on public perceptions. We find that in high performing countries the general public expresses higher levels of confidence in the education system. We discuss these patterns in the context of growing politicisation of education policy making and the use of ILSA as evidence.     

The contribution of PISA to the convergence of education policies in Europe

The international comparative studies on students’ outcomes have initiated analyses that have had a growing influence on national and sub‐national education policies in industrialised and developing countries. It is particularly the case of the OECD's Programme for International Student Assessment (PISA) which started in 2000 and has organised surveys every 3 years, so that the 2015 survey was the 6^th. Its influence has been particularly important for several reasons: 1) it assesses the basic competences in reading literacy, maths and science of 15 year‐olds students, i.e. around the end of compulsory education in many countries; 2) the assessment is based on a reliable methodology and the tests are completed by qualitative surveys and studies; 3) and the results lead to recommendations and are amplified by the media in most countries. However, it is not easy to evaluate the real impact of PISA because of the existence of other international studies such as IEA's TIMSS and, particularly in Europe, the influence of the recommendations and benchmarks of the EU that has been growing steadily in the last 25 years. Our analysis of the impact of PISA and EU policy focuses on the evolution of the education policy in France, but also studies its evolution in a few other European countries. Finally, we underline the limits of the influence of PISA and international standards in education towards a convergence of education systems because of the importance of their specific historic and cultural contexts.     

How Chinese students’ scientific competencies are influenced by their attitudes?

ABSTRACT

In recent years, research on students’ scientific competencies has generated much discussion. According to the PISA 2015 Assessment and Analytical Framework, both scientific competencies and attitudes towards science – interest in science, valuing scientific approaches to enquiry (or epistemological beliefs about science), and environmental awareness – are integral aspects of scientific literacy. However, few studies have focused on the relationships between them. This article examines how students’ scientific competencies are affected by their attitudes. Based on a theoretical analysis, the present study argues that the three attitudinal factors have a direct impact on scientific competencies, interest has an immediate influence on the other two attitudes, and epistemological beliefs are directly affected by environmental awareness. To test this hypothesis, a model was constructed to verify such relationships. PISA 2015 assessment instruments were used for data collection. The participants were 25,658 students from different parts of China, containing Beijing, Shanghai, Jiangsu, Guangdong, Taiwan, Hong Kong, and Macao. Structural Equation Modelling (SEM) was employed for data analysis. The results confirmed our hypothesis. The way in which Chinese students’ attitudes affected their scientific competencies did not differ (to a statistically significant level) by gender or district. The limitations and implications of this research are discussed below.