Linking errors in international large-scale assessments: calculation of standard errors for trend estimation

Abstract

One major aim of international large-scale assessments (ILSAs) is to monitor changes in student performance over time. To accomplish this task, a set of common items is repeatedly administered in each assessment and linking methods are used to align the results from the different assessments on a common scale. The present article introduces a framework for discussing linking errors in ILSAs, in which different components of linking errors are distinguished (country-by-item interaction, assessment-by-item interaction and country-by-assessment-by-item interaction). Furthermore, the different components of linking errors are used to analytically derive standard errors for national trend estimates. In a simulation study, the proposed standard error formula outperforms the method that is used in PISA. In addition, the PISA 2006 and 2009 reading data are used to illustrate how the interpretation of national trend estimates can change when different procedures are applied to calculate standard errors.     

Golden relics & historical standards: how the OECD is expanding global education governance through PISA for Development

ABSTRACT

Setting this paper against the backdrop of scholarly research on recent changes in the OECD’s approach and workings in education, I analyse how the OECD has reinforced its infrastructural and epistemological global governance through the Programme for International Student Assessment (PISA) for Development (PISA-D). Drawing on qualitative data, this paper makes three arguments. First, there has been a reinforced effort at the OECD to align national and international large-scale assessments; second, the OECD-ensured PISA-D was enhanced only in so far as it remained comparable with PISA, with a view to joining up lower- and middle-income data infrastructures with the global PISA infrastructure; and third, the OECD has bound together the aims of PISA, PISA-D, the Education and Skills Directorate, the Organisation’s Strategy on Development and the global education agenda (Sustainable Development Goals), thus strengthening its global education governance potential. With a note of concern, I suggest these recent changes in the OECD’s work in education may be spreading a very narrow framework of educational values, which does not sufficiently recognise the complexity of learning and teaching.     

PISA and its consequences: Shaping education policies through international comparisons

As the field of education has become a highly internationalised policy field in the last decade, international organisations such as the OECD play an ever more decisive role in the dissemination of knowledge, monitoring of outcomes, and research in education policy. Although the OECD lacks any binding governance instruments to put coercion on States or to provide material incentive, it has successively expanded its competences in this field. OECD advanced its status as an expert organisation in the field of education mainly by designing and conducting the international comparative PISA study. With PISA, the OECD was able to greatly influence national education systems. Basically, States were faced with external advice based on sound empirical data that challenged existing domestic policies, politics, and ideas. One prominent case for the impact of PISA is Germany. PISA was a decisive watershed in German education policy‐making. Almost instantly after the PISA results were publicised in late 2001, a comprehensive education reform agenda was put forward in Germany. The experienced reform dynamic was highly surprising because the traditional German education system and politics were characterised by deep‐rooted historical legacies, many involved stakeholders at different levels, and reform‐hampering institutions. Hence, a backlog of grand education reforms have prevailed in Germany since the 1970s. The external pressure exerted by PISA completely changed that situation.     

Examining the influence of international large-scale assessments on national education policies

This paper examines whether, to what extent, and how international large-scale assessments (ILSAs) have influenced education policy-making at the national level. Based on an exploratory review of the research and policy literature on ILSAs and two surveys administered to educational policy experts, researchers, policymakers, and educators, our research found that ILSAs, with their multiple and ambiguous uses, increasingly function as solutions in search for the right problem – that is, they appear to be used as tools to legitimize educational reforms. The survey results pointed to a growing perception among stakeholders that ILSAs are having an effect on national educational policies, with 38% of respondents stating that ILSAs were generally misused in national policy contexts. However, while the ILSA literature indicates that these assessments are having some influence, there is little evidence that any positive or negative causal relationship exists between ILSA participation and the implementation of education reforms. Perhaps the most significant change associated with the use of ILSAs in the literature reviewed is the way in which new conditions for educational comparison have been made possible at the national, regional, and global levels.     

Using Large-scale Assessment Datasets for Research in Science and Mathematics Education: Programme for International Student Assessment (PISA)

Large-scale assessments of student achievement provide a window into the broadly defined concepts of literacy and generate information about levels and types of student achievement in relation to some of the correlates of learning, such as student background, attitudes, and perceptions, and perhaps school and home characteristics. This paper provides an overview and outlines potential research opportunities of one such assessment—the Programme for International Student Assessment (PISA). In order to provide examples of the work that can be accomplished with these data, we describe and discuss the results generated from PISA 2000 and PISA 2003 in terms of international comparisons of achievement and the models of relational patterns of student, home, and school characteristics. We provide insight from the recent pilot testing conducted in Taiwan for PISA 2006, which has a focus on scientific literacy. This is followed by a discussion of the implications and potentials of the 2000 and 2003 datasets to facilitate research on scientific and mathematical literacy. The paper concludes with a look ahead to PISA 2006 and what researchers should be attending to in the research reports generated from the OECD and the research interests that they could follow given access to the datasets generated.

Unpacking equity. Educational equity in secondary analyses of international large-scale assessments: A systematic review

The discourse of educational equity has captured public imagination and become an entrenched policy value. In the attendant struggle to improve educational equity, governments draw inspiration from analyses based on data from highly standardized, international large-scale assessments. As these analyses are used as a lever for policy formulation, it is important to understand how this concept of equity is appropriated. Lacking a synthesis of how equity is conceptualized and operationalized in this body of research, the current paper seeks to fill that gap by presenting a systematic review. Our analysis incorporates an identification of operational patterns and an exploration regarding the linguistic elements for defining equity. We chart the theoretical and methodological diversity among interpretations of equity, as clustered into five major research approaches. In light of these results, benefits and limitations of the concept's complexity are discussed and implications for research are forwarded.     

PISA in Spain: Expectations,impact and debate

PISA, which was launched by OECD, is one of the most significant and successful initiatives on which education systems have recently collectively embarked. However, although it is a well‐coordinated international programme, its reception differs according to country. There is therefore a need to analyse specific national circumstances in order to gain a deeper understanding of the undertaking as a whole. This article specifically considers Spain's participation in PISA and focuses on a number of aspects: a) the expectations created when it joined the programme, in parallel to the implementation of its own national education evaluation system; b) the impact PISA has had, both in the media and in political and discursive spheres; and c) the technical and scientific debates generated in Spanish academic media. Finally, it is argued that, in the last few years, PISA has met with a certain disenchantment among specialists and the public opinion because of its limitations as a ranking tool, the difficulty in explaining its findings, and its inability to prescribe education policies that are suitable for very different contexts.     

Understanding business interests in international large-scale student assessments: a media analysis of The Economist,Financial Times,and Wall Street Journal

The media analysis is situated in the larger body of studies that explore the varied reasons why different policy actors advocate for international large-scale student assessments (ILSAs) and adds to the research on the fast advance of the global education industry. The analysis of The Economist, Financial Times, and Wall Street Journal covers publications on ‘PISA’, ‘TIMSS’, and related search items over the period 1996–2016. The three media outlets vary in terms of ILSA reporting. The Economist and Financial Times tend to focus on PISA, whereas the Wall Street Journal pays greater attention to TIMSS than PISA. The content analysis of 59 articles yields interesting results about how the business-oriented readership of the three media outlets frames public education and why it sees education as a profitable business opportunity. The three most common narratives, reflecting the business logic, are the following: (i) public education is in crisis; (ii) there is no correlation between spending and education outcome; and (iii) school accountability, teacher performance, and decentralisation represent the most effective policies to improve the quality of education. Drawing on these three common narratives, the financial media outlets present a particular vision of how to improve education; a vision in which the private sector is supposed to play a major role.     

PISA for Development: how the OECD and World Bank shaped education governance post-2015

ABSTRACT

The year 2015 was significant for the arena of international development, as UNESCO’s Education for All agenda was replaced by Education 2030, which would identify minimum standards of education quality. The OECD had been working on extending its existing PISA assessment into low- and middle-income countries through PISA for Development (PISA-D) and positioned the new assessment as a means of tracking progress on the post-2015 goals. The organisation maintains that PISA-D was introduced primarily in response to the demands of the international community, especially low- and middle-income countries, and that the assessment was developed in partnership with them. This paper investigates those claims through an analysis of the arrival of PISA-D in Cambodia, situating the analysis within UNESCO’s shifting agenda and the strategic visions of the OECD and World Bank that first emerged in the 1990s. The result is a very different picture to the portrayals of local agency, demand, consensus and partnership that adorn the official websites and pamphlets of global agencies and much academic research, raising serious doubts about education governance post-2015.     

Fifteen years of research on PISA effects on education governance: A critical review

This article provides a literature review on the effects of the OECD's Programme for International Student Assessment (PISA) on education governance and policy process across participating countries. This review seemed necessary because there has been a growing body of literature on this topic since 2003, especially since 2010, because this literature is not always well‐known and because the discourse on the so‐called ‘PISA shock’ remains important, even if it is more of a metaphor than a concept and may be politically partial. The article exploits a dataset of 87 references which show that PISA introduced major changes in the governance of education worldwide. Driven by soft power strategies and new policy transfers, this governance is based on data and measurement tools which redefine the scales of education policies. It also shows that PISA has a strong influence on a variety of national reforms, as illustrated in many case studies. However, this influence strongly depends on domestic policy contexts that scholars intended to capture through different theoretical frameworks. Nonetheless, few propose overarching theorisations of the political meaning of PISA effects on education governance and policy processes. The article concludes by stressing three main challenges for the subsequent studies on these PISA effects: better conceptualising these effects, preserving an epistemology of uncertainty in order to avoid taken for granted views and normalising the research on PISA effects not to perpetually and artificially rediscover its so‐called novelty.     

Student composition in the PISA assessments: Evidence from Brazil

PISA is one of the most widely used instruments to assess student performance in the world. This paper investigates how changes in testing dates and students’ dates of birth (eligibility) between 2000 and 2018 have affected results in Brazil. Controlling for changes in students’ grade distribution over time, we show that changes in grade composition are correlated with increases in average score. We estimate that about 60 % of improvements in math are unrelated to student improvement. Since changes in testing date and eligibility are unrelated to the school system, performance increases over time might not be a result of a country’s educational policy.     

Language effects in international testing: the case of PISA 2006 science items

We investigate the extent to which language versions (English, French and Arabic) of the same science test are comparable in terms of item difficulty and demands. We argue that language is an inextricable part of the scientific literacy construct, be it intended or not by the examiner. This argument has considerable implications on methodologies used to address the equivalence of multiple language versions of the same assessment, including in the context of international assessment where cross-cultural fairness is a concern. We also argue that none of the available statistical or qualitative techniques are capable of teasing out the language variable and neutralising its potential effects on item difficulty and demands. Exploring the use of automated text analysis tools at the quality control stage may be successful in addressing some of these challenges.     

Problematizing global educational governance of OECD PISA: Student achievement,categorization, and social inclusion and exclusion

Abstract

This study problematizes the global educational governance of OECD PISA and its statistical data as a governing technology in contemporary discourses of education reforms. The study examines principles that order the discourses and practices of the international comparative assessment. The effort of analysing the impact of an education reform regime led by OECD PISA reveals how statistical reasoning defines problems in educational systems and forms social discourse surrounding educational reform to solve such problems. In doing so, this article focuses on standardization, classification, and normalization for measuring and comparing student achievement and national effectiveness. The study also offers an alternative way of considering the politics of inclusion and exclusion embedded in practices of education reforms propelled by the international comparative assessment.     

PISA as an ideational roadmap for policy change: exploring Germany and England in a comparative perspective

Education policy has undergone transformation in many countries over the last decade. In this article, we focus on the effects of the most significant international initiative in secondary education, which is the Organisation for Economic Co-operation and Development's (OECD) Programme for International Student Assessment (PISA). We analyse two countries that provide variation regarding the degree of change in their respective education policy-making due to this study; while Germany substantially reformed its education system in reaction to its mediocre PISA results, almost no change has been observed in England. As we show, alterations and shifts in ideas of education policy best account for such a change.     

Development of School Achievement in the Nordic Countries During Half a Century

The aim is to describe the development of achievement in compulsory school in the Nordic countries from the 1960s. The study relies on published results concerning literacy and numeracy from the international large-scale assessments between 1964 and 2012. Among others, the following conclusions are drawn: (1) for most countries, a small but consistent increase in the level of achievement was observed from the mid-1970s to around 1990 for both literacy and numeracy; (2) Finland improved literacy performance dramatically between the mid-1980s and the mid-1990s, which is hypothesized to be associated with the introduction of part-time special education; (3) for all countries performance declined from the late-1990s/early-2000s, which is hypothesized to be due to different factors in different countries.     

Education policy in Poland: The impact of PISA (and other international studies)

The impact of the PISA study on Polish education policy has been significant, but probably different from any other country. Poland has not experienced the so‐called ‘PISA shock’, but its education system has been benefiting considerably from PISA. For experts and policy makers, it has been a useful and reliable instrument that has made it possible to measure the effects of consecutive reforms of the school education system. Moreover, PISA and other international studies have influenced the perception of education policy in Poland. The latter has shifted from an ideology‐driven, centralised policy to an evidence‐informed policy, developed with the involvement of multiple stakeholders, although this has mostly affected the thinking of experts and policy makers rather than the general public. The new government (in power from 2015), following public opinion polls, has reversed most of the previous education reforms, eliminating lower secondary schools introduced in 1999.     

The Contribution of International Large-Scale Assessments to Educational Research: Combining Individual and Institutional Data Sources

The present paper aims to discuss how data from international large-scale assessments (ILSAs) can be utilized and combined, even with other existing data sources, in order to monitor educational outcomes and study the effectiveness of educational systems. We consider different purposes of linking data, namely, extending outcomes measures, analyzing differences over time or across cohorts, and supplementing context information. These linking strategies are illustrated by a non-exhaustive selection of studies that exploited ILSAs to investigate a wide range of educational topics. We conclude that the main contribution of ILSA to educational research lies in the ways they facilitate analyses of educational policy and policy-related issues at the institutional level by means of cross-country analyses. However, the scope of these studies also covers high-quality data on lower levels of the educational system.     

When Assessment Validation Neglects Any Strand of Validity Evidence: An Instructive Example from PISA

The Standards for Educational and Psychological Testing identify several strands of validity evidence that may be needed as support for particular interpretations and uses of assessments. Yet assessment validation often does not seem guided by these Standards, with validations lacking a particular strand even when it appears relevant to an assessment. Consequently, the degree to which validity evidence supports the proposed interpretation and use of the assessment may be compromised. Guided by the Standards, this article presents an independent validation of OECD's PISA assessment of mathematical self-efficacy (MSE) as an instructive example of this issue. OECD identifies MSE as one of a number of “factors” explaining student performance in mathematics, thereby serving the “policy orientation” of PISA. However, this independent validation identifies significant shortcomings in the strands of validity evidence available to support this interpretation and use of the assessment. The article therefore demonstrates how the Standards can guide the planning of a validation to ensure it generates the validity evidence relevant to an interpretive argument, particularly for an international large-scale assessment such as PISA. The implication is that assessment validation could yet benefit from the Standards as what Zumbo calls “a global force for testing”.     

Finland and Singapore in PISA 2009: similarities and differences in achievements and school management

In PISA 2009, Finland and Singapore were both ranked high among the participating nations and have caught much attention internationally. However, a secondary analysis of the means for Reading achievement show that the differences are rather small and are attributable to spurious precision. Hence, the two nations should be considered as being on par with each other in achievements and be assigned the same rank. Spurious precision as a problem of interpreting and reporting research findings has caught the attention of researchers in several other disciplines, though not in the field of education, and this needs to be rectified. In spite of the finding of no differences in PISA Reading achievement, principals in Finland and Singapore differ somewhat in school management and involvement in school matters. It is suggested that some intervening variables (e.g., teachers’ quality and instruction) are needed to explain the correlation (or the lack of it) between principals’ management styles and student Reading achievement. It is also suggested that school principals’ management styles might have been influenced by the cultural milieus of the countries and have influenced students’ social-emotional development, which is not measured by PISA.