PISA Country Rankings Valid? Results for Canada and Finland

This article examines whether the way that PISA models item outcomes in mathematics affects the validity of its country rankings. As an alternative to PISA methodology a two-parameter model is applied to PISA mathematics item data from Canada and Finland for the year 2012. In the estimation procedure item difficulty and dispersion parameters are allowed to differ across the two countries and samples are restricted to respondents who actually answered items in a mathematics cluster. Different normalizations for identifying the distribution parameters are also considered. The choice of normalization is shown to be crucial in guaranteeing certain invariance properties required by item response models. The ability scores obtained from the methods employed here are significantly higher for Finland, in sharp contrast to PISA results, which gave both countries very similar ranks in mathematics.     

Italian students’ results in the PISA mathematics test: does reading competence matter?

In Italy, from the 2003 reports to the present, the National Institute for the Educational Evaluation of Instruction and Training (INVALSI) has conducted research on Programme for International Student Assessment (PISA) results in order to understand Italian students’ low achievement in mathematics. In the present paper, data from a representative sample of 15-year-old Italian students who participated in PISA 2012 were analysed. This study’s primary aim is to verify how students’ linguistic competences are associated with their performance in mathematics. For the evaluation of the impact of item reading demand on students’ performance, we selected 24 mathematics items with a high reading demand and 31 mathematics items with a low reading demand, as classified by Italian language and methodology experts. Repeated measure variance analyses were conducted. The results showed differences in function of gender: females are advantaged in mathematics items with a high reading demand, independent of their level of reading literacy. In contrast, males are advantaged in mathematics items with a low reading demand, independent of their level of reading literacy. Possible policy implications are discussed.     

Testing measurement invariance of mathematics self-concept and self-efficacy in PISA using MGCFA and the alignment method

The aim of the study is to investigate the measurement invariance of mathematics self-concept and self-efficacy across 40 countries that participated in the Programme for International Student Assessment (PISA) 2003 and 2012 cycles. The sample of the study consists of 271,760 students in PISA 2003 and 333,804 students in PISA 2012. Firstly, the traditional measurement invariance testing was applied in the multiple-group confirmatory factor analysis (MGCFA). Then, the alignment analyses were performed, allowing non-invariance to a minimum to estimate all of the parameters. Results from MGCFA indicate that mathematics self-concept and self-efficacy hold metric invariance across the 80 groups (cycle by country). The alignment method results suggest that a large proportion of non-invariance exists in both mathematics self-concept and self-efficacy factors, and the factor means cannot be compared across all participating countries. Results of the Monte Carlo simulation show that the alignment results are trustworthy. Implications and limitations are discussed, and some recommendations for future research are proposed.

     

Evaluating item response theory linking and model fit for data from PISA 2000–2012

ABSTRACT

Based on concerns about the item response theory (IRT) linking approach used in the Programme for International Student Assessment (PISA) until 2012 as well as the desire to include new, more complex, interactive items with the introduction of computer-based assessments, alternative IRT linking methods were implemented in the 2015 PISA round. The new linking method represents a concurrent calibration using all available data, enabling us to find item parameters that maximize fit across all groups and allowing us to investigate measurement invariance across groups. Apart from the Rasch model that historically has been used in PISA operational analyses, we compared our method against more general IRT models that can incorporate item-by-country interactions. The results suggest that our proposed method holds promise not only to provide a strong linkage across countries and cycles but also to serve as a tool for investigating measurement invariance.     

Benchmarking the performance of Bhutanese students with the performance of the students from the OECD's PISA countries

Setting international benchmarks for education systems of the Organisation for Economic Co-operation and Development (OECD) countries is one of the goals of the OECD's Programme for International Student Assessment (PISA). However, some countries are not able to participate in PISA, despite their desire to set international benchmarks for their education systems. This article presents a method of setting international benchmarks for a country's school education system, without necessarily participating in PISA, by designing a test using the test items released by PISA for public consumption. The method has been implemented in a study that involved 1,500 Grade 10 students across 60 schools in Bhutan. The students were administered a mathematics test constructed from the PISA Mathematical Literacy test items. The study showed that the performance of Bhutanese students was comparable with the performance of the students from the countries that participated in PISA 2003 and that Bhutan could learn from both high- and low-performing school education systems of those countries.     

A comparison of PISA and TIMSS 2003 achievement results in mathematics

This study compares the Programme for International Student Assessment (PISA) 2003 Mathematics results with the Trends in International Mathematics and Science Study (TIMSS) 2003 Grade 8 mathematics results, using country mean scores for 22 participants of both studies. It is found that Western countries generally performed better in PISA than in TIMSS, and Eastern European and Asian countries generally performed better in TIMSS than in PISA. Furthermore, two factors, content balance and years of schooling, can account for 93% of the variation between the differential performance of countries in PISA and TIMSS. Consequently, the rankings of countries in the two studies can be reconciled to a reasonable degree of accuracy.     

PISA and policy-borrowing: A philosophical perspective on their interplay in mathematics education

Mathematics achievement in different education systems around the world is assessed periodically in the Programme for International Student Assessment (PISA). PISA is deemed to yield robust international comparisons of mathematical attainment that enable individual countries and regions to monitor the performance of their education systems relative to standards being achieved internationally, with a view to informing their mathematics education policy decisions. Initially, the role of PISA in instigating mathematics education policy borrowing is outlined using England as a case study, and some existing technical critiques of PISA are then reviewed. Following this, aspects of Ludwig Wittgenstein’s later philosophy of mind are used to reason that an over-reliance on the use of PISA to inform policy decisions in mathematics education may be problematic. It is suggested that, when PISA is viewed through a later Wittgensteinian lens, a potential deficiency in the underpinning psychometric model, pertaining to the inherent indeterminism in unmeasured mathematical abilities, may weaken PISA’s utility in guiding mathematics education policy decisions. It is concluded that, whilst PISA mathematics scores may give some indication of the mathematical proficiency of a nation’s students, caution is required before mathematics education policies are borrowed from other jurisdictions on the basis of PISA performance. Implications for the other PISA domains are also outlined.     

PISA 2012: how do results for the paper and computer tests compare?

The Programme for International Assessment (PISA) is an important cross-national study of 15-year olds academic achievement. Although it has traditionally been conducted using paper-and-pencil tests, the vast majority of countries will use computer-based assessment from 2015. In this paper, we consider how cross-country comparisons of children’s skills differ between paper and computer versions of the PISA mathematics test. Using data from PISA 2012, where more than 200,000 children from 32 economies completed both paper and computer versions of the mathematics assessment, we find important and interesting differences between the two sets of results. This includes a substantial drop of more than 50 PISA test points (half a standard deviation) in the average performance of children from Shanghai-China. Moreover, by considering children’s responses to particular test items, we show how differences are unlikely to be solely due to the interactive nature of certain computer test questions. The paper concludes with a discussion of what the findings imply for interpretation of PISA results in 2015 and beyond.     

The PISA Tasks: Unveiling Prospective Elementary Mathematics Teachers’ Difficulties with Contextual,Conceptual, and Procedural Knowledge

The aim of this mixed methods study was to investigate the difficulties prospective elementary mathematics teachers have in solving the Programme for International Student Assessment (PISA) 2012 released items. A mathematics test consisting of 26 PISA items was administered, followed by interviews. Multiple data were utilized to provide rich insights into the types of mathematical knowledge that a particular item requires and prospective teachers’ difficulties in using these knowledge types. A sample of 52 prospective teachers worked the mathematics test, and 12 of them were interviewed afterwards. The data-sets were complementary: the quantitative data showed that PISA items could be categorized under contextual, conceptual, and procedural knowledge and indicated the most frequent difficulties in the combined contextual, conceptual, and procedural knowledge items. The qualitative data revealed that few prospective teachers could give mathematical explanations for conceptual knowledge items, and that their contextual knowledge was fragmented. Educational implications were discussed.     

FIRST CYCLE OF PISA (2000–2006)—INTERNATIONAL PERSPECTIVES ON SUCCESSES AND CHALLENGES: RESEARCH AND POLICY DIRECTIONS

This special edition of IJMSE focuses on the Programme of International Student Assessment (PISA) project now that it has completed a full cycle of administration—reading, mathematics, and science—to look at ways in which PISA has been used in participating countries and with what consequences, and to identify potential research and policy directions emanating from this initiative. Articles were invited to (a) reflect international perspectives on the uses and consequences of PISA to date and (b) speculate on future directions for research, curriculum, and policy using the PISA datasets. The introductory article provides a brief overview of common aspects of PISA: Evolving definitions of reading literacy, mathematics literacy, and science literacy; technical design of the instruments and data analysis procedures; the changing emphasis of administrations; and recent research using the datasets. PISA, unlike other international assessments in reading, mathematics, and science, has provided a fresh perspective on ‘what might be’ by decoupling the assessment from mandated curricula to focus on literacies needed for a 21st century economy. This unique feature of PISA brings with it possibilities and cautions for policy makers.     

The role that mathematics plays in college- and career-readiness: evidence from PISA

Many studies have found a strong relationship between the mathematics students study in school and their performance on an academic or school mathematics assessment but not on an assessment of mathematics literacy (ML). With many countries, like the USA, placing emphasis on finishing secondary education being mathematically literate and prepared for college or career, this raises the question about the relationship between the mathematics studied in school and any ML students may have. The Programme for International Student Assessment (PISA) ML assessment is embedded in real-world contexts that provide an important window on how ready students are to tackle the situations and problems that await them whether they intend to pursue further education beyond high school or intend to go directly into the labour force. In this paper, we draw upon the PISA 2012 data to investigate the extent to which the cumulative exposure to rigorous mathematics content, such as that embedded in college- and career-ready standards, is associated with ML as assessed in PISA. Results reveal that both exposure to rigorous school mathematics and experiencing the instruction of this mathematics through real-world applications are significantly related to all the real-world contextualized PISA ML scores.     

The causal effect of an extra year of schooling on skills and knowledge in Latin America. Evidence from PISA

Abstract

In this paper, we estimate the causal effect of an extra year of schooling on mathematics performance for seven Latin American countries based on PISA 2012. To that end we exploit exogenous variation in students’ birthdates around the school entry cut-off date using both sharp and fuzzy Regression Discontinuity designs. We find strong effects of an extra year of schooling in most countries, which amount to a 30% increase in PISA test scores in Brazil, 18% in Uruguay, 7% in Argentina and 6% in Costa Rica. These effects differ from the typical estimates obtained from simple regressions or multilevel models and are large enough to allow 15-year-old students to reach higher proficiency levels, suggesting significant potential gains of reducing dropout rates in the region. Finally, we stress the importance of taking into account the effects of school entry cut-off dates on PISA samples to avoid making unfair international comparisons.     

The use (and misuse) of PISA in guiding policy reform: the case of Spain

In 2013 Spain introduced a series of educational reforms explicitly inspired by the Programme for International Student Assessment (PISA) 2012 results. These reforms were mainly implemented in secondary education – based upon the assumption that this is where Spain's educational problems lie. This paper questions this assumption by attempting to identify the point where Spanish children fall behind young people in other developed countries. Specifically, by drawing data from multiple international assessments, we are able to explore how cross-national differences in reading skills change as children age. Consideration is given to both the average level of achievement and the evolution of educational inequalities. Our conclusion is that policy-makers have focused their efforts on the wrong part of the education system; educational achievement is low in Spain (and educational inequalities large) long before children enter secondary school. This study therefore serves as a note of caution against simplistic interpretation of the PISA rankings.     

The relationship between homework time and achievement is not universal: evidence from multilevel analyses in 40 countries

Although homework is generally believed to be an important supplement to in-school learning, research has not yet fully clarified the relationship between homework and achievement. This cross-cultural study analyzed the relationship between homework time and mathematics achievement drawing on data from 231,759 students in 9,791 schools and 40 countries who participated in the Programme for International Student Assessment (PISA) 2003. Multilevel analyses found a positive association between school-average homework time and mathematics achievement in almost all countries, but the size of the association decreased considerably once socioeconomic background and school track were controlled. At the student level, no clear-cut relationship was established between homework time and achievement across the 40 countries. The results highlight the need to use multilevel analyses and to control for confounding variables in homework research.     

Testing measurement invariance of PISA 2015 mathematics,science, and ICT scales using the alignment method

This study investigates measurement invariance of the mathematics, science, and ICT scales across the 47 countries that participated in the PISA 2015 ICT Familiarity Questionnaire. Knowing whether the same constructs and measurements can be reliably compared across countries constitutes an important goal. The Alignment method is employed to test the measurement invariance of the three scales. The results show that mathematics and science scores are highly invariant and can be used to compare countries, whereas the ICT scale is mostly non-invariant and cannot be used to reliably compare ICT means across all participating countries. Implications and limitations are discussed.     

多维项目反应理论在数学素养测验中的应用

学生的数学素养具有多维结构,素养导向的数学学业成就测评需要提供被试在各维度上的表现信息,而不仅是一个单一的总分。以PISA数学素养结构为理论模型,以多维项目反应理论(MIRT)为测量模型,利用R语言的MIRT程序包处理和分析某地区8年级数学素养测评题目数据,研究数学素养的多维测量方法。结果表明:MIRT兼具单维项目反应理论和因子分析的优点,利用其可对测试的结构效度和测试题目质量进行分析,以及对被试进行多维能力认知诊断。     

Identifying national cultures of mathematics education: Analysis of cognitive demands and differential item functioning in TIMSS

Large-scale assessments of student competencies address rather broad constructs and use parsimonious, unidimensional measurement models. Differential item functioning (DIF) in certain subpopulations usually has been interpreted as error or bias. Recent work in educational measurement, however, assumes that DIF reflects the multidimensionality that is inherent in broad competency constructs and leads to differential achievement profiles. Thus, DIF parameters can be used to identify the relative strengths and weaknesses of certain student subpopulations. The present paper explores profiles of mathematical competencies in upper secondary students from six countries (Austria, France, Germany, Sweden, Switzerland, the US). DIF analyses are combined with analyses of the cognitive demands of test items based on psychological conceptualisations of mathematical problem solving. Experts judged the cognitive demands of TIMSS test items, and these demand ratings were correlated with DIF parameters. We expected that cultural framings and instructional traditions would lead to specific aspects of mathematical problem solving being fostered in classroom instruction, which should be reflected in differential item functioning in international comparative assessments. Results for the TIMSS mathematics test were in line with expectations about cultural and instructional traditions in mathematics education of the six countries.     

PISA 2015: how big is the ‘mode effect’ and what has been done about it?

The Programme for International Student Assessment (PISA) is an important cross-national study of 15-year-olds’ academic knowledge and skills. Educationalists and public policymakers eagerly await the tri-annual results, with particular interest in whether their country has moved up or slid down the international rankings, as compared to earlier rounds. In 2015 a major change was implemented in PISA, with the introduction of computer-based assessment. This has the potential to reduce comparability of PISA test scores across countries and over time. We investigate this issue using PISA 2015 field trial data for three countries: Germany, Sweden, and Ireland. We show how, if left unaccounted for, the change to computer-based testing could limit the comparability of PISA test scores. We then describe the methodology the study organisers have used to account for such mode effects. Our key conclusion is that although the adjustment made is unlikely to overcome all the potential challenges of switching to computer-based tests, it represents an improvement over the alternative of making no adjustment at all.