Problematizing global educational governance of OECD PISA: Student achievement,categorization, and social inclusion and exclusion

Abstract

This study problematizes the global educational governance of OECD PISA and its statistical data as a governing technology in contemporary discourses of education reforms. The study examines principles that order the discourses and practices of the international comparative assessment. The effort of analysing the impact of an education reform regime led by OECD PISA reveals how statistical reasoning defines problems in educational systems and forms social discourse surrounding educational reform to solve such problems. In doing so, this article focuses on standardization, classification, and normalization for measuring and comparing student achievement and national effectiveness. The study also offers an alternative way of considering the politics of inclusion and exclusion embedded in practices of education reforms propelled by the international comparative assessment.     

Do Teachers’ Instructional Practices Moderate Equity in Mathematical and Scientific Literacy?: an Investigation of the PISA 2012 and 2015

Many efforts have been made to reach educational equity, especially to reduce mathematics and science achievement gaps by students’ socioeconomic status. Across countries, educators strive to reform traditional teacher-centered instructional approaches to more student-centered/inquiry-based instruction to improve equity in education. In this context, this study examines whether relationships between socioeconomic status and scientific or mathematical literacy are moderated by student-centered instruction. Ten countries covering a wide range of achievement levels as well as equity in education are selected for an international comparison. A linear regression analysis is applied to student achievement, equity, and frequency of student-centered instruction data from the PISA 2012 and PISA 2015. We find mixed results: As student-centered instruction is offered more frequently, the gap in mathematical and scientific literacy between low and high socioeconomic status is generally narrowed or maintained. In most countries, students’ mathematical and scientific literacy scores are expected to decrease across all socioeconomic status as student-centered instruction is given more frequently. The findings necessitate further scrutiny of how teachers implement student-centered instruction in various educational systems. This further research need to consider the complexity of implementation related to sociological and pedagogical aspects.     

Globalisation and the politics of international tests: the case of Israel

Comparative international tests of students' achievements have become increasingly popular over the past decade. An outcome of this widespread practice is the tendency of various countries to evaluate their national status according to their pupils' international ranking in such achievement tests, partly due to the common belief that high ranking is an indication of the nation's future ability to compete in the global market. Using PISA 2000 and PISA 2006 data, the present study aims to show that the low achievements of Israeli pupils were to be expected, and that their average scores are a reflection of the economic and demographic characteristics of their country. Our findings suggest that the public debate in Israel on school achievements has been magnified out of all proportion. We also discuss the educational policy which derives proposals for educational reform from a conception of globalisation of education. This policy focuses on the setting of standards for educational achievements aimed at improving pupils' place on the international scale. Finally, we claim that the focus on the international grading of educational achievements deflects public attention from an understanding of the significance of inequalities within the country.     

国际教育测评视野下的芬兰教育成功特点探析

芬兰在国际教育测评中,特别是在“国际学生评价项目”中的成功引起了世界各国教育研究者及政策制定者的关注。通过对芬兰在IEA和OECD等组织的测评项目中所取得的成绩情况进行梳理,认为芬兰教育在国际测评中取得了以成绩卓越、均衡公平和持续发展为特性的成功。通过考察芬兰教育成功的特点,以期深化对芬兰教育和教育成功的理解,为我国基础教育提供有益启示。     

The contribution of PISA to the convergence of education policies in Europe

The international comparative studies on students’ outcomes have initiated analyses that have had a growing influence on national and sub‐national education policies in industrialised and developing countries. It is particularly the case of the OECD's Programme for International Student Assessment (PISA) which started in 2000 and has organised surveys every 3 years, so that the 2015 survey was the 6^th. Its influence has been particularly important for several reasons: 1) it assesses the basic competences in reading literacy, maths and science of 15 year‐olds students, i.e. around the end of compulsory education in many countries; 2) the assessment is based on a reliable methodology and the tests are completed by qualitative surveys and studies; 3) and the results lead to recommendations and are amplified by the media in most countries. However, it is not easy to evaluate the real impact of PISA because of the existence of other international studies such as IEA's TIMSS and, particularly in Europe, the influence of the recommendations and benchmarks of the EU that has been growing steadily in the last 25 years. Our analysis of the impact of PISA and EU policy focuses on the evolution of the education policy in France, but also studies its evolution in a few other European countries. Finally, we underline the limits of the influence of PISA and international standards in education towards a convergence of education systems because of the importance of their specific historic and cultural contexts.     

Investigating the relationship between instructional practices and science achievement in an inquiry-based learning environment

ABSTRACT

This study investigates the discrete effects of inquiry-based instructional practices that described the PISA 2015 construct ‘inquiry-based instruction’ and how each practice, and the frequency of each practice, is related to science achievement across 69 countries. The data for this study were drawn from the PISA 2015 database and analysed using hierarchical linear modelling (HLM). HLMs were estimated to test the contribution of each item to students’ science achievement scores. Some inquiry practices demonstrated a significant, linear, positive relationship to science achievement (particularly items involving contextualising science learning). Two of the negatively associated items (explaining their ideas and doing experiments) were found to have a curvilinear relationship to science achievement. All nine items were dummy coded by the reported frequency of use and an optimum frequency was determined using the categorical model and by calculating the inflection point of the curvilinear associations in the previous model e.g. students that carry out experiments in the lab in some lessons have higher achievement scores than students who perform experiments in all lessons. These findings, accompanied by detailed analyses of the items and their relationships to science outcomes, give stakeholders clear guidance regarding the effective use of inquiry-based approaches in the classroom.     

Comparability of educational achievement and learning attitudes across nations

We estimated the invariance of educational achievement (EA) and learning attitudes (LA) measures across nations. A multi-group confirmatory factor analysis was used to estimate the invariance of educational achievement and learning attitudes across 55 nations (Programme for International Student Assessment [PISA] 2006 data, N?=?354,203). The constructs had the same meaning (factor loadings) but different scales (intercepts). Our conclusion is that comparisons of the relationships between educational achievement and learning attitudes across countries need to take into consideration two sources of variability: individual differences of students and group differences of educational systems. The lack of scalar invariance in EA and LA measures means that the relationships between EA and LA may have a different meaning at the level of nations and at the student level within countries. In other words, as PISA measures are not invariant in scalar sense, the comparisons across countries with nationally aggregated scores are not justified.     

Alleviating or exacerbating disadvantage: does school attended mediate the association between family background and educational attainment?

ABSTRACT

Opportunities for social mobility are generated by education systems designed to alleviate the effects of social origin by providing equality of opportunities and resources. The persistence of the strong association between socioeconomic status (SES) and child’s educational achievement and attainment suggests that social origin continues to play an integral role in the educational outcomes of successive generations of Australians. Sociologists draw on a range of theoretical perspectives to explain this association including Bourdieu’s cultural and social capital theories. Using data collected by the Longitudinal Survey of Australian Youth 2009 (LSAY09) project, I examine the associations between student SES, school SES and two outcome variables: Programme for International Student Assessment (PISA) score and university enrolment. The results indicate that low SES students attending high SES schools perform better on PISA tests than low SES students attending low SES schools. After controlling for PISA score, low SES students were less likely than their high SES peers to enrol at university. Furthermore, students attending low SES schools were less likely than their peers attending high SES schools to enrol at university, net of their individual SES and their PISA scores.     

PISA Data: Raising concerns with its use in policy settings

This article considers the role played by policy makers, government organisations, and research institutes (sometimes labelled “think tanks”) in the analysis, use and reporting of PISA data for the purposes of policy advice and advocacy. It draws on the ideas of Rizvi and Lingard (Globalizing Education Policy, 2010), Bogdandy and Goldmann (Governance by Indicators/ Global Power through Quantification and Rankings, 2012) and others to explore the ways in which such “agents of change” can interpret, manipulate and disseminate the results of data arising from large scale assessment survey programs such as PISA to influence and determine political and/or educational research agendas. This article illustrates this issue by highlighting the uncertainty surrounding the PISA data that have been used by a number of prominent, high profile agents of change to defend policy directions and advice. The final section of this paper highlights the need for policy makers and their advisors to become better informed of the technical limitations of using international achievement data if such data are to be used to inform policy development and educational reforms.     

A new perspective on memorization practices among East Asian students based on PISA 2012

Abstract

This study examined learning strategy use in mathematics among East Asian students in East Asian educational systems. By employing latent class analysis on the Programme for International Student Assessment (PISA) 2012 data, we found four classes of learning strategy types, namely memorization with metacognitive strategies (17.49%), metacognitive strategies with memorization (50.70%), elaboration only (10.33%), and metacognitive strategies with elaboration (16.47%). The results showed that the majority of the students in all seven East Asian educational systems belonged to the ‘metacognitive strategies with memorization’ class, and most students adopted more than one type of learning strategy when learning mathematics. Additionally, students who reported the use of metacognitive strategies along with either memorization or elaboration showed higher mathematics achievement. We conclude that the cognitive processes employed by students of East Asian backgrounds are more complex and nuanced than the previous perception that they relied heavily on memorization.     

The (mis)use of the Finnish teacher education model: ‘policy-based evidence-making’?

Abstract

Background: International achievement studies such as the Programme for International Student Assessment (PISA) have an increasing influence on education policy worldwide. The use of such data can provide a basis for evidence-based policy-making to initiate educational reform. Finland, a high performer in PISA, is often cited as an example of both efficient and equitable education. Finland’s teachers and teacher education have not only garnered much attention for their role in the country’s PISA successes, but have also influenced education policy change in England.

Main argument: This article argues that the Finnish model of teacher education has been borrowed uncritically by UK policy-makers. Finnish and English philosophies of teacher preparation differ greatly, and the borrowing of the Finnish teacher education model does not fit within the teacher training viewpoint of England. The borrowed policies, thus, were decontextualised from the wider values and underpinnings of Finnish education. This piecemeal, ‘pick “n” mix’ approach to education policy reform ignores the fact that educational policies and ‘practices exist in ecological relationships with one another and in whole ecosystems of interrelated practices’. Thus, these borrowed teacher preparation policies will not necessarily lead to the outcomes outlined by policy-makers in the reforms.

Sources of evidence: Two teacher preparation reforms in England, the University Training Schools (outlined in the UK Government’s 2010 Schools White Paper, The Importance of Teaching) and the Master’s in Teaching and Learning (MTL), are used to illustrate the problematic nature of uncritical policy borrowing. This article juxtaposes these policies with the Finnish model of teacher education, a research-based programme where all candidates are required to complete a Master’s degree. The contradictions exposed from this analysis further highlight the divergent practices of teacher preparation in England and Finland, or the disparate ‘ecosystems’. Evidence of educational policy borrowing in other settings is also considered.

Conclusions: Both the MTL and the White Paper reforms overlook the ‘ecosystem’ surrounding Finnish teacher education. The school-based MTL contrasts with the research-based Finnish teachers’ MA. Similarly, the University Training Schools scheme, based on Finnish university-affiliated, teaching practice schools, contrasts heavily with the rest of the White Paper reforms, which contradict the philosophies and ethos behind Finnish teacher education by proposing the move of English teacher preparation away from the universities. The analysis highlights the uncritical eye through which politicians may view international survey results, looking for ‘quick fix’ options instead of utilising academic evidence for investigation on education and education reform.     

大规模学生评估的影响力与重要性：德国视角 大规模学生评估的影响力与重要性： 现状与趋势

二十年来,大规模学生评估对教育研究、学校体系和教育政策产生了深远影响。“国际中学生评估项目”（PISA）、“国际学生数学与科学能力动态项目”（TIMSS）和“国际学生阅读能力进步研究项目”（PIRLS）使各个国家学生成绩值具有一定可比性,由此,人们能更细致地从学校内部来观察不同国家学校工作的差异度。大规模学生评估还为学校发展、教育领导以及学生成绩的改进提供必要数据。本文以PIRLS为例,旨在从德国视角为中国今后开展大规模国际学生评估提供借鉴。     

Reading comprehension of immigrant students in Germany: research evidence on determinants and target points for intervention

International studies, such as the Programme for International Student Assessment (PISA), have shown that, in most participating countries, students who do not typically speak the test language at home reach lower levels of reading comprehension than students using the test language at home (Stanat & Christensen, 2006). Results from PISA indicate that Germany is among the countries with the most pronounced differences in reading comprehension between immigrant students and students from native families. The present article summarizes these findings and shows that the reading achievement gap persists even when the socioeconomic and educational background of students?? families are controlled. Furthermore, although controlling for background factors reduces the effect of the language spoken at home on reading, it continues to be substantial. Using these findings as a starting point, the article addresses the question of what should be done to close this gap. It summarizes research findings indicating that oral proficiency presents an important determinant of reading comprehension in a second language. This suggests that effective general approaches to second-language teaching are needed to promote reading literacy in a L2. The article closes with a discussion of the available evidence on the effectiveness of such approaches and outlines the need for further research.     

The PISA calendar: Temporal governance and international large-scale assessments

Abstract

This article analyses international large-scale assessments in education from a temporal perspective. The article discusses and compares the different conceptions of time in the early international assessments conducted in the 1960s and 1970s by the IEA with the PISA studies conducted by the OECD from the year 2000 onwards. The paper argues that there has been a shift in the ways that the assessments structure time. The early IEA surveys were characterized by a relative slowness, lack of synchronization and lack of trend analyses. PISA, by contrast, is characterized by high pace, simultaneous publication of results around the world and regular and recurrent studies making the analysis of trends possible. The emergence of this new time regime, it is argued, has implications for how education is governed. At the transnational level, it strengthens the influence and importance of OECD as a significant policy actor. At the national level, as educational discourse and policy adapts to the temporalities of the PISA calendar, two kinds of effects can be distinguished. First, there is a tendency towards searching for “retrotopian” solutions for contemporary problems. Second, there is a tendency towards acceleration and short-term planning when it comes to educational reforms.     

Who makes it into PISA? Understanding the impact of PISA sample eligibility using Turkey as a case study (PISA 2003–PISA 2012)

ABSTRACT

New evidence presented in this paper shows that existing accounts of Programme for International Student Assessment (PISA) in developing countries underestimate both progress and inequality because they do not take into account the large proportion of 15–16-year olds who are ineligible for the PISA sampling frame, primarily due to dropout or delay. In such countries, who makes it into PISA (sample eligibility) heavily influences representivity and therefore the interpretation of PISA results. We use PISA in Turkey 2003–2012 as a case study to show how combining measures of access (from the Turkish Demographic and Health Surveys) and learning outcomes (from PISA) can account for changes in the underlying population. We find the percentage of 15–16-year olds reaching Level 2 in PISA is up to twice as large as that reflected in official PISA reports. These findings have strong implications for inter-country and inter-temporal comparisons using PISA, particularly for developing countries.     

‘Becoming European’? Respatialising the European Schools System through PISA for Schools

ABSTRACT

This paper examines the Organisation for Economic Cooperation and Development’s (OECD) PISA for Schools, a local variant of the more-renowned ‘main PISA’ test that measures and compares individual school performance on reading, mathematics and science against international schooling systems. Here, I address the governance implications of how PISA for Schools data has been taken up by the transnational European Schools System (ESS). Drawing suggestively across new ‘relational’ thinking around data-driven modes of governance, as well as interviews with key policy actors within the ESS, I show how PISA for Schools reflects contradictory logics within the ESS, in which the inherently context-based goal of ‘becoming European’ is juxtaposed with the desire to employ decontextualised international evidence. I conclude by exploring how the perceived need for such data, associated with the global authority of the OECD, can produce a problematic focus on data-driven practices.     

What is the influence of cultural capital on student reading achievement in Confucian as compared to non-Confucian heritage societies?

Abstract

The present study compared the contribution of familial cultural capital to the reading achievement of 116,508 15-year-old students who participated in PISA 2012 in six Confucian heritage cultures (CHCs) and nine non-CHCs with comparable educational and economic development. The different states of cultural capital examined comprised institutionalised (maternal, paternal education) and objectified (educational, cultural resources) indicators. Results showed that: (1) cultural capital levels were lower in CHCs (vis-à-vis non-CHCs); (2) cultural capital was generally positively related to student achievement in CHCs and non-CHCs; (3) the relationships between all cultural capital indicators, except educational resources, and achievement were weaker in CHCs than non-CHCs; and (4) objectified (vis-à-vis institutionalised) cultural capital was more strongly associated with achievement in CHCs. These results suggest that the stronger sociocultural emphasis on education in CHCs may have moderated cultural capital effects on student achievement and enabled more students to succeed academically.     

Structural Communications: An Instructional Strategy for the Language Arts

The ranking of the United States in major international tests such as the Progress in International Reading Literacy Study (PIRLS), Trends in International Mathematics and Science Study (TIMSS), and Program for International Student Assessment (PISA) is used as the driving force and rationale for the current educational reforms in the United States. Through current reforms, the United States is preparing to “race to the top” of the international rankings. However, using international test results as the ultimate indicator of a country's educational achievement might not be an appropriate measure. Additionally, replicating some aspects of high-achieving countries’ education systems or standards may not guarantee success for U.S. public schools. In this article, four reasons are presented explaining why there is no guaranteed success in comparing the United States to other countries’ educational standards and to their achievement in international tests. In order to successfully implement reforms in U.S. public schools, educational leaders and reformers should carefully consider the four points discussed in this article and use tests to inform their reforms, rather than to create uniformity.     

Heterogeneous Subgroup Identification in Observational Studies

Abstract

Despite the overwhelming focus on the overall average treatment effect in the methodological and statistical literature, in many cases the efficacy of an educational program or intervention might vary based on unit background characteristics. The identification of subgroups for which an educational intervention is particularly effective or, on the other hand, has no effect or is possibly harmful, may have important practical implications, especially in terms of allocation of resources. We propose a five-step approach using propensity score matching and regression trees to identify subgroups with heterogeneous treatment effects in observational studies. Results of two Monte Carlo simulation studies demonstrate that the proposed approach can accurately identify heterogeneous subgroups while maintaining Type I error rate. In a case study with Early Childhood Longitudinal Study-Kindergarten cohort data, we find that the effect of exposure to special education services on fifth-grade mathematics achievement varies based on kindergarten mathematics achievement and student gender.