Using TIMSS and PISA results to inform educational policy: a study of Russia and its neighbours

In this paper, we develop a multi-level comparative approach to analyse Trends in International Mathematics and Science Survey (TIMSS) and Programme of International Student Achievement (PISA) mathematics results for a country, Russia, where the two tests provide contradictory information about students’ relative performance. Russian students do relatively well on the TIMSS mathematics test but relatively poorly on the PISA. We compare the performance of Russian students with different levels of family academic resources over the past decade on these tests compared to students with similar family resources in Russia’s neighbours and to Russian students studying in Latvian and Estonian Russian-medium schools. These comparisons and interviews with educators in Latvia and Estonia help us understand why students in Russia may perform lower on the PISA and to draw education policy lessons for improving international test performance generally and Russian students’ PISA mathematics performance specifically.     

On the supranational spell of PISA in policy

Abstract

Background: PISA results appear to have a large impact upon government policy. The phenomenon is growing, with more countries taking part in PISA testing and politicians pointing to PISA results as reasons for their reforms.

Purpose: The aims of this research were to depict the policy reactions to PISA across a number of jurisdictions, to see whether they exhibited similar patterns and whether the same reforms were evident.

Sources of evidence: We investigated policy and media reactions to the 2009 and 2012 PISA results in six cases: Canada, China (Shanghai), England, France, Norway and Switzerland. Cases were selected to contrast high-performing jurisdictions (Canada, China) with average performers (England, France, Norway and Switzerland). Countries that had already been well reported on in the literature were excluded (Finland, Germany).

Design and methods: Policy documents, media reports and academic articles in English, French, Mandarin and Norwegian relating to each of the cases were critically evaluated.

Results: A policy reaction of ‘scandalisation’ was evident in four of the six cases; a technique used to motivate change. Five of the six cases showed ‘standards-based reforms’ and two had reforms in line with the ‘ideal-governance’ model. However, these are categorisations: the actual reforms had significant differences across countries. There are chronological problems with the notion that PISA results were causal with regard to policy in some instances. Countries with similar PISA results responded with different policies, reflecting their differing cultural and historical education system trajectories.

Conclusions: The connection between PISA results and policy is not always obvious. The supranational spell of PISA in policy is in the way that PISA results are used as a magic wand in political rhetoric, as though they conjure particular policy choices. This serves as a distraction from the ideological basis for reforms. The same PISA results could motivate a range of different policy solutions.     

Comparing OECD educational models through the prism of PISA

The PISA survey influences educational policies through an international competitive process which is not wholly rationally-oriented. Firstly, PISA league tables act normatively upon the definition of formal educational aims while the survey tests cannot evaluate the educational systems’ relative strengths with regards to such aims. We argue that they measure a kind of academic potential of the students. Secondly, errors in interpreting national success or failure stem from the causal inferences developed from observing national cases. In order to give such comparisons a stronger basis, we distinguish five main educational models within the OECD countries and compare their PISA 2006 results to their population’s socio-cultural levels as well as their performance recorded in an academic-programme-based survey such as TIMSS. Our conclusions contrast heavily with the usual lessons inferred from PISA, which follow OECD-promoted guidelines, and explain French students’ weakening as well as Finnish students’ success.     

Constructing and educating 'problem children': the case of post-communist Estonia

Estonia is a post-communist Baltic state in which neoliberal market ideologies and still prevailing socialist norms exist side by side in educational policies and practices. It is no longer self-evident what is ‘normal’ and what is ‘problematic’. Special education class for students regarded as having problems is a new innovation in Estonia. The paper analyses the history and current changes and discusses how a ‘problem child’ is constructed in Estonian teachers' perceptions and in the pedagogical methods used with these children. The paper draws on teachers' writings and on an ethnographic study in a special education class. Some comparisons with Finnish educational policies and ethnographic research on secondary schools are made. The findings suggests that in Estonia, a ‘problem child’ is a student who does not conform to the norms of the school, and in focus is behaviour that disturbs the teacher: problems in relation to schools' regulations of time, voice, embodiment and equipment, lack of engagement and unruly behaviour. We reflect the Estonian case with some comparisons with Finland using the concept ‘professional pupil’ introduced in an ethnographic research.     

Language‐in‐education across the Baltic: policies,practices and challenges

Since their political reorientation that started in the late 1980s, the Baltic states have experienced profound social reforms to rebuild their democratizing societies. Education has featured as a key vehicle to this transformation process of overcoming the limitations of the Soviet past. National legislative frameworks were therefore soon provided to restructure educational systems and allow for ‘liberal’ approaches to schooling. The process of policy‐making for a new concept of education, however, entailed numerous challenges, and it has resulted in a diverse tapestry of educational provision across the Baltic. With specific focus on minority education, this paper presents a comparative analysis of education policies and practices in contemporary Estonia, Latvia and Lithuania in response to emerging language needs in these intensely multiethnic settings.     

Structural embeddedness of students’ professional expectations: the Czech Republic and Taiwan in comparative perspective

Abstract

We compare the role of gender, socioeconomic background and measured ability on students’ secondary school placement and professional expectations in Taiwan and the Czech Republic, two countries that have experienced substantial reforms in their educational systems in the context of political change and economic development. Using data from the international PISA 2006 survey of 15-year-olds, our analysis reveals that Czech academic upper-secondary schools are much more selective in terms of socioeconomic background and gender than comparable Taiwanese senior high schools. Controlling for measured ability and other factors, students’ professional expectations are also more closely linked to socioeconomic background and gender in the Czech Republic than in Taiwan.     

School size and student achievement: a longitudinal analysis

Numerous initiatives by private philanthropies and the US government have supported school size reduction policies as an educational reform intended to improve student outcomes. Empirical evidence to support these claims, however, is underdeveloped. In this article, we draw on information from a longitudinal dataset provided by the Northwest Evaluation Association covering more than 1 million students in 4 US states. Employing a student fixed effects strategy, we estimate how a student’s achievement changes as (s)he moves between schools of different sizes. We find evidence that students’ academic achievement in math and reading declines as school size increases. The negative effects of large schools appear to matter most in higher grades, which is also when schools tend to be the largest.     

Is exam hell the cause of high academic achievement in East Asia? The case of Japan and the case for transcending stereotypes

High student achievement across East Asia is often explained as an outcome of highly competitive, stress‐inducing college entrance exams across the region. This ‘exam hell’ drives students to study longer and harder than their peers worldwide, a race that leads—unsurprisingly—to higher marks in international comparisons such as the Programme for International Student Assessment (PISA). Yet, the race comes at a considerable cost: psychological pressure, forfeited childhoods, regimes of rote memorisation, lack of creativity and private outlays for cram schools. In this article, we seek to refute this outdated narrative by focusing on changes in Japanese education over the past three decades. We first analyse quantitative data for learning time based on three longitudinal surveys conducted by researchers in Japan to show that learning time for Japanese upper secondary students declined during 1980–2000 (Study 1). We then turn to analyse PISA 2015 data to show that now Japanese students study less than their peers in major Anglo‐American countries: the USA, the UK and Australia (Study 2). These findings run counter to well‐worn images frequently rehearsed in both the academic and popular literature. In helping to remove one of the most persistent stereotypes about East Asian education, we seek to help make visible other explanations for East Asian student achievement.     

The past,history, and education

If you mention ‘educational reform’ in a room full of educators in the US, prepare to leave the room with your ears burning. For most teachers and school administrators in the US, ‘educational reform’ currently means standards, curriculum frameworks, mandatory state tests for students, and sanctions for schools if test scores fail to rise. Although some educators see benefits in parts of this agenda, many find that their jobs have become more stressful and less rewarding, and they wonder how schools got into their current situation. Reading the two books reviewed in this essay provides some answers, including insights into what motivates educational reformers and into why reforms often play out differently from what their designers intended.     

The impact of PISA and the interrelation and development of assessment policy and assessment theory in Japan

PISA presents a new image for academic achievement, which has prompted Japanese education reforms over the past decade to innovate teaching and learning for ‘PISA-style literacy’. Supported by theoretical foundations, particularly with regard to the concept of ‘PISA literacy’ and ‘authentic assessment’, these reforms have accomplished progress in the focus on higher order competencies, such as application and the development of new assessment strategies. However, more recently, various critical discussions of ‘PISA literacy’ are underway in the Japanese academy. They interrogate it more critically and reveal the narrow emphasis on functional application and technical operation. Current assessment practices, which tend to fall into ‘criteria compliance’, are in urgent need of review. There is a need to extend the critical discussions in progress to the new assessment strategies. This paper responds to this, by considering the Japanese acceptance of ‘PISA literacy’ and its assessment, discussing the features and limitations.     

Special educational needs,disability and school accountability: an international perspective

Standards-based accountability approaches in education are increasingly being adopted in many industrialised countries. This tends to involve the scrutiny of student performance in national academic tests and then holding schools accountable for subsequent progress. International assessments such as Programme for International Student Assessment (PISA) are not accountability measures in themselves, but are often utilised in a similar way and national policies are judged against performance in these tests. Drawing upon the examples of PISA and national assessments in the USA, this paper explores how students with special educational needs and disability (SEND) are included in, and served by, these assessment policies. The analysis suggests that while these approaches are inclusive in their intent (most obviously through efforts to include students with SEND in testing) they may poorly represent the progress of these students. It is argued that more equitable inclusion of students with SEND must find a balance between accountability measures which measure performance in traditional attainment-focused tests and other relevant assessments which are meaningful to the given student and/or SEND sub-group.     

Education reforms and adult skills: Evidence from Estonia

This paper studies the impact of education reforms in Estonia in the 1990s on adult skills using the OECD PIAAC surveys. Estonia implemented extensive education reforms in the early 1990s throughout Estonian-speaking schools while Russian-speaking schools were exposed to less comprehensive reforms, which were implemented later. A large minority of Estonia’s population at the time was enrolled in Russian-speaking schools providing a unique opportunity to measure the impact of education reforms on literacy, numeracy and problem-solving skills among adults by comparing improvements in PIAAC performance among Estonian and Russian speakers. Difference-in-difference estimation suggests that the reforms led to an adult skill dividend of around 15 to 30 percent of a standard deviation. This translates to a wage (productivity) premium of around 5 to 12 percent.     

The relationship of school-based parental involvement with student achievement: a comparison of principal and parent survey reports from PISA 2012

This paper explores parental involvement using principal and parent survey reports to examine whether parents’ involvement in their children’s schools predicts academic achievement. Survey data from principals and parents of seven countries from the PISA 2012 database and hierarchical linear modelling were used to analyse between- and within- school variance in students’ math achievement. Factor analysis of both principal and parent responses revealed three dimensions of parental involvement with schools: parent-initiated involvement, teacher-initiated involvement and parent volunteerism. Principal reports of parent-initiated involvement positively predicted between-school differences in student achievement. Within schools, parent reports of teacher-initiated involvement negatively predicted student achievement. The paper shows the importance of understanding the source of information for survey measures. Information on parental involvement from the parent surveys of the PISA study is suitable for describing within-school variation in student achievement, whereas principal reports can be used to predict variation between schools.     

Inclusive education reform in Queensland: implications for policy and practice

In Queensland, Australia, the school system is being reformed to be more ‘inclusive’. However, the enthusiasm for ‘inclusive education’ in Queensland seems to be waning amongst practitioners, and the ‘confusion, frustration, guilt and exhaustion’ that has emerged with teachers and support practitioners in the UK is emerging amongst support practitioners and teachers in Queensland. This paper argues that this is happening because inclusive education reforms that intend to provide an equitable education for all students regardless of cultural, physical, social/emotional and behavioural differences are being introduced, but these policies, procedures and structures continue to label, isolate and segregate students within schools in the way in which segregated special education facilities did in the past. Also, new policies and structures are being introduced without practitioners having the time and support to examine critically the underlying assumptions about disability, difference and inclusion that underpin their practices. These reforms need to be reviewed in terms of their effectiveness in achieving their ‘inclusive’ goals, i.e. in terms of the impacts that these reforms are having on the students themselves, and on the educational practitioners who support the students.     

‘Better policies for better lives’?: constructive critique of the OECD’s (mis)measure of student well-being

ABSTRACT

Facing increasing critique that PISA focuses too narrowly on cognitive achievement and human/knowledge capital, the OECD has recently shifted some of its focus to student happiness. The 2017 Students’ Well-Being report distinguishes between ‘happy schools’ and ‘unhappy schools’, showing that among students who combined high performance and life satisfaction, northern European countries topped the charts. Meanwhile, students in East Asian countries including Japan, China, Taiwan, Hong Kong, and Korea registered the lowest ‘life satisfaction’ scores among all participating countries. This piece points out some of the problems inherent in the OECD’s recent turn to happiness, problematizing the OECD yardstick of life satisfaction. Attempting to keep the critique constructive, we suggest that the OECD may want to consider using alternative metrics, then briefly highlight one developed in East Asia from different first assumptions: the Interdependent Happiness Scale. In conclusion we flag, but cannot answer, some related educational questions concerning policy, pedagogy, and priorities for the future.     

Structural Communications: An Instructional Strategy for the Language Arts

The ranking of the United States in major international tests such as the Progress in International Reading Literacy Study (PIRLS), Trends in International Mathematics and Science Study (TIMSS), and Program for International Student Assessment (PISA) is used as the driving force and rationale for the current educational reforms in the United States. Through current reforms, the United States is preparing to “race to the top” of the international rankings. However, using international test results as the ultimate indicator of a country's educational achievement might not be an appropriate measure. Additionally, replicating some aspects of high-achieving countries’ education systems or standards may not guarantee success for U.S. public schools. In this article, four reasons are presented explaining why there is no guaranteed success in comparing the United States to other countries’ educational standards and to their achievement in international tests. In order to successfully implement reforms in U.S. public schools, educational leaders and reformers should carefully consider the four points discussed in this article and use tests to inform their reforms, rather than to create uniformity.     

Anti-school attitudes,school culture and friendship networks

This article explores the relationship between educational outcomes and anti-school attitudes at different levels of social organization in schools. Data were collected in St. Petersburg, Russia (104 schools, 7300 students) and analyzed using multi-level regression models that included three levels: individual, clique of friends and school. A clique is defined as a tight group of friends in a school class; we used social network analysis software Kliquefinder for clique identification. We demonstrate that friends’ attitudes are strongly related to the educational outcomes of a student (net of person’s individual attitudes and socio-demographic characteristics). In contrast, school-level effects disappear in the multi-level model when individual characteristics are included. The results of the study clearly demonstrate that the socio-economic and curricular differentiation of schools does not always lead to the polarization of ‘school academic cultures’. A school social environment is sufficiently heterogeneous, and different value systems in small peer groups may coexist.     

Assessing the quality and equality of Hong Kong basic education results from PISA 2000+ to PISA 2006

This paper seeks to examine the quality and equality of basic education of Hong Kong based on the first three cycles of the Program for International Student Assessment (PISA). Results from these three assessments suggested that the Hong Kong students have an outstanding performance in mathematics, science and reading. In particular, the performance of reading improves substantially in PISA 2006. As far as equality in education is concerned, the achievement gap of students from different socio-economic backgrounds in Hong Kong is relatively small compared with other countries. However, the academic performance variation between-schools suggest that, there is still academic segregation among secondary schools in Hong Kong although it has been reduced slightly in the PISA 2006.     

Fifteen years of research on PISA effects on education governance: A critical review

This article provides a literature review on the effects of the OECD's Programme for International Student Assessment (PISA) on education governance and policy process across participating countries. This review seemed necessary because there has been a growing body of literature on this topic since 2003, especially since 2010, because this literature is not always well‐known and because the discourse on the so‐called ‘PISA shock’ remains important, even if it is more of a metaphor than a concept and may be politically partial. The article exploits a dataset of 87 references which show that PISA introduced major changes in the governance of education worldwide. Driven by soft power strategies and new policy transfers, this governance is based on data and measurement tools which redefine the scales of education policies. It also shows that PISA has a strong influence on a variety of national reforms, as illustrated in many case studies. However, this influence strongly depends on domestic policy contexts that scholars intended to capture through different theoretical frameworks. Nonetheless, few propose overarching theorisations of the political meaning of PISA effects on education governance and policy processes. The article concludes by stressing three main challenges for the subsequent studies on these PISA effects: better conceptualising these effects, preserving an epistemology of uncertainty in order to avoid taken for granted views and normalising the research on PISA effects not to perpetually and artificially rediscover its so‐called novelty.