International large-scale assessments: what uses,what consequences?

Abstract

Background: International large-scale assessments (ILSAs) are a much-debated phenomenon in education. Increasingly, their outcomes attract considerable media attention and influence educational policies in many jurisdictions worldwide. The relevance, uses and consequences of these assessments are often the focus of research scrutiny. Whilst some argue that the assessment outcomes provide an effective basis for informed policy-making, critics claim that the use of international assessment data can result in a range of unintended consequences, such as the shaping and governing of school systems ‘by numbers’.

Purpose: This article explores and analyses the arguments about the uses and consequences of ILSAs. In particular, the discourse about the assessments’ consequential validity will be discussed and evaluated.

Sources of evidence: Literature relating to the uses and consequences of large-scale assessment was analysed, with a focus on research on the consequential aspects of validity.

Main argument: Much research suggests that ILSAs have unintended consequences that affect and influence educational policy. However, the influences on educational policy are complex and interwoven: for example, it is not clear-cut whether effects such as converging curricular are, necessarily, direct consequences of large-scale assessments. Further, it is suggested that a beneficial consequence of large-scale assessment is the infrastructure they provide for studies in the social sciences, although caution must be applied to causal claims, in particular because of the cross-sectional design of the assessments.

Conclusions: The considerable literature discussing the uses and consequences of large-scale assessments tends to point out potential negative aspects of the studies. However, it is also apparent that large-scale international assessments can be a valuable resource for studying global trends and evolving systems in education. Despite the extensive debates around large-scale assessment outcomes both in the media and in educational policy arenas, empirical educational research all too often appears underused in the discussion.     

Examining the influence of international large-scale assessments on national education policies

This paper examines whether, to what extent, and how international large-scale assessments (ILSAs) have influenced education policy-making at the national level. Based on an exploratory review of the research and policy literature on ILSAs and two surveys administered to educational policy experts, researchers, policymakers, and educators, our research found that ILSAs, with their multiple and ambiguous uses, increasingly function as solutions in search for the right problem – that is, they appear to be used as tools to legitimize educational reforms. The survey results pointed to a growing perception among stakeholders that ILSAs are having an effect on national educational policies, with 38% of respondents stating that ILSAs were generally misused in national policy contexts. However, while the ILSA literature indicates that these assessments are having some influence, there is little evidence that any positive or negative causal relationship exists between ILSA participation and the implementation of education reforms. Perhaps the most significant change associated with the use of ILSAs in the literature reviewed is the way in which new conditions for educational comparison have been made possible at the national, regional, and global levels.     

Linking errors in international large-scale assessments: calculation of standard errors for trend estimation

Abstract

One major aim of international large-scale assessments (ILSAs) is to monitor changes in student performance over time. To accomplish this task, a set of common items is repeatedly administered in each assessment and linking methods are used to align the results from the different assessments on a common scale. The present article introduces a framework for discussing linking errors in ILSAs, in which different components of linking errors are distinguished (country-by-item interaction, assessment-by-item interaction and country-by-assessment-by-item interaction). Furthermore, the different components of linking errors are used to analytically derive standard errors for national trend estimates. In a simulation study, the proposed standard error formula outperforms the method that is used in PISA. In addition, the PISA 2006 and 2009 reading data are used to illustrate how the interpretation of national trend estimates can change when different procedures are applied to calculate standard errors.     

How International Large-Scale Skills Assessments engage with national actors: mobilising networks through policy,media and public knowledge

ABSTRACT

This paper examines how international, large-scale skills assessments (ILSAs) engage with the broader societies they seek to serve and improve. It looks particularly at the discursive work that is done by different interest groups and the media through which the findings become part of public conversations and are translated into usable form in policy arenas. The paper discusses how individual countries are mobilised to participate in international surveys, how the public release of findings is managed and what is known from current research about how the findings are reported and interpreted in the media. Research in this area shows that international and national actors engage actively and strategically with ILSAs, to influence the interpretation of findings and subsequent policy outcomes. However, these efforts are indeterminate and this paper argues that it is at the more profound level of the public imagination of education outcomes and of the evidence needed to know about these that ILSAs achieve their most totalising effects.     

Human Capital Index and the hidden penalty for non-participation in ILSAs

The World Bank’s Human Capital Index (HCI) aims to provide new information regarding future productivity of each country’s workforce, by synchronizing available International Large-Scale Assessment (ILSA) and regional test program results. Linking the literature on ILSA participation, this study questions the problematic nature of this approach and revisits the comparability issue of ILSA results. We find that education systems are imposed upon a score penalty depending on which ILSA or regional test program they choose to partake in. In particular, our results show that (i) test-overlap systems used in the score synchronization procedure are systematically different from systems that only choose to participate in one ILSA or exclusively in regional tests, (ii) inter-test score exchange rate is volatile due to sampling design and cohort effects, (iii) test participation type alone accounts for about 57.8 percent of the variation in synchronized scores, and the score penalty is especially salient for systems that exclusively participate in regional test programs; the majority of which are low-income and lower-middle income countries. Findings in this study show how various intra- and extrapolations to compensate for missing data in effect introduce large score penalties for systems that either did not participate or only partially participated in ILSAs. Finally, this study contributes to research on reasons for participation in ILSAs and the global rise of test-based accountability reform, under which the World Bank’s new HCI may be seen as a tool to incentivize participation in ILSAs by penalizing those governments that have chosen alternative, non-standardized paths for measuring learning outcomes of students.     

Policy impact of PISA on mathematics education: the case of Norway

This article addresses the policy implications of participation in international large-scale assessments (ILSAs), particularly the Programme for International Student Assessment (PISA), and the ways in which such implications might influence mathematics education. Taking Norway as a special case, this discussion focuses on insights into teaching, learning and assessment practices that can be inferred from the PISA study, and how participation in ILSAs has contributed to educational policy and even changed policymakers’ perspectives on schools, teachers and students. Following publication of the PISA 2000 results, Norway experienced a ‘PISA shock’, leading to the implementation of a national quality assessment system and national tests. In addition, changes were made to the mathematics curriculum for compulsory school and to mathematics teacher education. More recently, public debate has focused less on rank and league tables, shifting instead to the high number of low-achieving students and the low number of high achievers. Moreover, there has been little uptake of policy advice provided by the Organisation for Economic Co-operation and Development (OECD), which focuses on strengthening accountability measures. Furthermore, although the Norwegian educational system in the past decade has undergone a decentralisation process, the educational system still follows the Nordic model, which focuses on equity and ‘education for all’. Analyses of the Norwegian case indicate that policymaking takes place in highly cultural contexts, and that international studies might be used merely to validate existing policy directions.     

League tables in educational evidence-based policy-making: can we stop the horse race,please?

International large-scale student assessments (ILSAs) in education represent a valuable source of information for policy-makers, not only on student achievements, but also on their relationship with different contextual factors. The results are partly described in the official studies’ reports; more can be derived from the publicly released data sets. However, league tables are often the only evidence used in policy debates and decisions on education. Indeed, the comparison of student achievement across the participating educational systems is a legitimate proxy for estimating countries’ development and productivity, but the use of league tables more often turns into ‘horse-ranking’, ignoring the contexts of teaching and learning. This is often supported by the media, turning the use of results into their abuse. The purpose of this paper is to discuss the use and misuse of league tables in reporting ILSA results, vs. the use of data for in-depth analysis in order to make informed decisions.     

Causal Mediation in Educational Research: An Illustration Using International Assessment Data

Abstract

This paper applies the causal mediation framework proposed by Kosuke Imai and colleagues (Imai, Keele, & Tingley, 2010) to educational research by examining the causal mediating role of early literacy activities in parental education influences on reading performance. The paper argues that the study of causal mediation is particularly relevant in educational settings because learning outcomes result from complex interactions involving multiple actors. The analyses use retrospective longitudinal data from the Progress in International Reading Literacy Study (PIRLS) 2011. The value of international assessment data for producing evidence of causation is discussed. The paper aims to contribute to the study of causation with observational data from educational studies.     

Seeking sustainability competence and capability in the ESD and HESD literature: an international philosophical hermeneutic analysis

Education for sustainable development (ESD) and higher education for sustainable development (HESD) are complex, multidisciplinary fields of enquiry, drawing on concepts and terms from different disciplines and languages. Although the fields are advancing in their acceptability within educational systems worldwide, they are currently struggling to achieve sought-after graduate and societal outcomes such as environmentally-responsible or sustainability-focussed-citizenship. The research described in this article explores the possibility that miscommunication or misunderstanding of basic concepts within these fields is contributing to slow progress towards their objectives. We used a philosophical hermeneutic analysis to explore how the terms ‘competence’ and ‘capability’ are used within selected ESD/HESD papers. We identify substantial internal contradictions and inconsistencies with respect to differences between learners’ abilities and their willingness to perform these abilities, and to the educational context in which these outcomes are sought. We emphasise the importance of linking educational objectives with pedagogical approaches to teaching and assessment.     

Refuting the OECD-World Bank development narrative: was East Asia’s ‘Economic Miracle’ primarily driven by education quality and cognitive skills?*

Founded on several highly influential quantitative studies, the past decade has witnessed the OECD and World Bank increasingly converge on the view that cognitive levels of students and education quality, as proxied by international large-scale assessments (ILSAs), are the primary determinant of national economic growth worldwide. More recent OECD and World Bank pronouncements have further suggested these dynamics are clearly illustrated in East Asia’s education and ‘Economic Miracle’, in particular the cases of South Korea and Singapore. Herein we utilise the OECD’s own data to examine this new development narrative, finding little evidence in support of these claims.     

Research Findings from the Tennessee Value-Added Assessment System (TVAAS) Database: Implications for Educational Evaluation and Research

The Tennessee Value-Added Assessment System determines the effectiveness of school systems, schools, and teachers based on student academic growth over time. An integral part of TVAAS is a massive, longitudinally merged database linking students and student outcomes to the schools and systems in which they are enrolled and to the teachers to whom they are assigned as they transition from grade to grade. Research conducted utilizing data from the TVAAS database has shown that race, socioeconomic level, class size, and classroom heterogeneity are poor predictors of student academic growth. Rather, the effectiveness of the teacher is the major determinant of student academic progress. Teacher effects on student achievement have been found to be both additive and cumulative with little evidence that subsequent effective teachers can offset the effects of ineffective ones. For these reasons, a component linking teacher effectiveness to student outcomes is a necessary part of any effective educational evaluation system.     

Does money matter for schools?

There is considerable disagreement in the academic literature about whether raising school expenditure improves educational outcomes. Yet changing the level of resources is one of the key policy levers open to governments. In England, school expenditure has increased by about 40% since 2000. Thus assessing whether such spending has had an impact on educational outcomes is of paramount importance. We address this issue using data of better quality than what are often available in similar studies and test our identification assumption by use of a falsification test. We find that the increase in school expenditure over recent years has had a consistently positive effect on outcomes at the end of primary school. Back-of-envelope calculations suggest that the investment may well be cost-effective. There is also some evidence of heterogeneity in the effect of expenditure, with higher effects for students who come from economically disadvantaged backgrounds.     

A systematic review on the deployment and effectiveness of data analytics in higher education to improve student outcomes

Abstract

This is a systematic review conducted of primary research literature published between 2007 and 2018 on the deployment and effectiveness of data analytics in higher education to improve student outcomes. We took a methodological approach to searching databases; appraising and synthesising results against predefined criteria. We reviewed research on the effectiveness of three differentiated forms of data analytics: learning, academic and learner analytics. Student outcomes are defined as retention, academic performance and engagement. Our results find that three quarters of studies report the use of educational data analytics to be effective in improving student outcomes but their relationship with student outcomes requires further and more robust investigation and assessment. We argue that research must interpret and communicate effectiveness qualitatively, as well as quantitatively, by including the student voice in assessments of impact.     

Equality and Opportunity in Education: Evidence from the 1958 and 1970 birth cohort studies

There is controversy about whether inequalities and educational outcomes are increasing or decreasing. Using longitudinal data collected in two birth cohort studies started in 1970 and 1958 respectively, the paper examines the evidence in relation to two outcomes, probability of leaving school at 16 and highest qualification achieved. Multi-variate analysis (logistic and OLS regression) was used to model the relationships of these educational outcomes to family social class, taking account of a wide range of early life variables, including living in an urban as opposed to rural location. It is concluded that the impact of social class on educational achievement has not changed across the 12 years covered by the two studies, a result that applies in both rural and urban areas of Britain.     

Horizontal inequalities in children’s educational outcomes in Ethiopia

We go beyond previous studies of vertical inequality in Ethiopia, by investigating ethnic group-based, or horizontal, educational inequality. Currently, there are no known studies of differential schooling patterns across cultural groups in Ethiopia. The research draws on the Young Lives prospective longitudinal study of 929 children in later childhood. We investigate the extent to which ethnic group differences in schooling participation and progression can be explained by individual differences in socio-economic status. The findings show that poor educational participation and progression remain challenges for certain minority ethnic groups. Household wealth helps to explain the effect of ethnicity on educational outcomes. However, it cannot fully account for the ethnic gap. We conclude that conventional research, which emphasizes addressing individual differences in educational outcomes, or vertical inequalities, may downplay the role of cultural group-based inequalities in children's opportunities to learn.     

Do extra-curricular activities in schools improve educational outcomes? A critical review and meta-analysis of the literature

Secondary schools tend to sponsor a large number of extra-curricular activities (ECA) yet little is known about their contribution to students’ educational outcomes. This meta-analysis aims to determine what it is about ECA participation that supports positive educational outcomes. Furthermore, this study challenges the theoretical assumptions about the benefits of participation in ECA. 29 studies (all except for one based on data collected in the United States) met the search criteria for inclusion in the analysis. Most effect sizes on academic achievements yielded from non-specific ECA, academic clubs and journalism were small, as were participation in performing arts, sports and leadership activities on a range of educational outcomes. Although the results show associations between participation in ECA and educational outcomes, causal effects could not be confirmed. It is concluded that the lack of evidence supporting the causal effects, and thus the common theoretical assumptions about the effects of ECA on educational outcomes, is due to methodology limitations in these studies.     

Science in the schoolhouse: an uninvited guest

Science and scientific thinking have not made a substantial impact on educational practice. In this discussion, we examine the relationship between science and education and delineate four reasons for characterizing science as an uninvited guest in schools: (a) Science is not highly regarded in society; (b) good science and bad science are often mistaken for one another; (c) the amount of current data is overwhelming; and (d) science is not easy for those who practice it (researchers), those who translate it (teacher educators), or those who consume it (teachers). We suggest several strategies to improve this relationship, including promoting standards of educational practice, emphasizing the role of teacher educators as translators of the research base into classroom practice, and linking student outcomes with the use of effective instructional practices.