PISA 2015: how big is the ‘mode effect’ and what has been done about it?

The Programme for International Student Assessment (PISA) is an important cross-national study of 15-year-olds’ academic knowledge and skills. Educationalists and public policymakers eagerly await the tri-annual results, with particular interest in whether their country has moved up or slid down the international rankings, as compared to earlier rounds. In 2015 a major change was implemented in PISA, with the introduction of computer-based assessment. This has the potential to reduce comparability of PISA test scores across countries and over time. We investigate this issue using PISA 2015 field trial data for three countries: Germany, Sweden, and Ireland. We show how, if left unaccounted for, the change to computer-based testing could limit the comparability of PISA test scores. We then describe the methodology the study organisers have used to account for such mode effects. Our key conclusion is that although the adjustment made is unlikely to overcome all the potential challenges of switching to computer-based tests, it represents an improvement over the alternative of making no adjustment at all.     

Can competencies assessed by PISA be considered the fundamental school knowledge 15-year-olds should possess?

The OECD Programme for International Student Assessment (PISA), launched by governments of the Organisation for Economic Co-operation and Development (OECD) in 1997, aims at assessing some of the key competencies that contribute to the success of 15-year-old individuals, on a regular basis and within an internationally accepted framework. PISA seeks to provide a basis for policy dialogue and for collaboration in defining and implementing educational goals, in innovative ways that reflect judgements about the skills that are relevant to adult life. PISA defines competence as the ability to successfully meet complex demands in varied contexts through the mobilisation of psychosocial resources, including knowledge and skills, motivation, attitudes, emotions, and other social and behavioural components. Measuring and comparing competencies across languages and cultures is a difficult challenge and is being pursued by PISA progressively. PISA focused its first three assessments on literacy skills, defined as the capacity of young adults to access, manage, integrate and evaluate information, to think imaginatively, to hypothesise and discover, and to communicate their thoughts and ideas effectively. The reasoning behind shifting the emphasis from assessing whether students can reproduce what they have learned towards whether they can extrapolate from what they have learned and apply their competencies in novel situations, derives from the nature of knowledge and skills required in modern life. For example, the tasks that can be solved through simple memorisation or with pre-set algorithms are those that are also easiest to digitise, automatise and offshore, and will thus be less relevant in a modern knowledge society. Since there is no overarching cross-national and cross-cultural agreement on what fundamental competencies 15-year-olds should possess, an international assessment such as PISA can only capture a selection of competencies. Moreover, since various methodological constraints limit the nature of competencies that are currently amenable to large scale assessment, PISA cannot capture the entirety of competencies that will make young people successful. However, the findings presented in this article suggest that the competencies that PISA does assess are highly predictive for the future success of students. In addition, PISA provides policy makers and practitioners with useful tools to improve quality, equity and efficiency in education, by revealing some common characteristics of students, schools and education systems that do well. In a modern world, comparative assessments are an essential tool for educational improvement and research shows that the existence of standardised assessments and examinations is one of the most powerful predictors for the success of an education system. That is not hard to understand, because without such assessments, all students, schools and education systems look the same, it is impossible for teachers and school administrators to detect institutional and systemic strengths and weaknesses, and to support and intervene where expectations are not met. Without reliable and comparable information on learning outcomes, teachers and governments alike rely on input-based incentives and policies that are all too often mirrored in large quality variation between schools as well as a strong dependency between learning success and the socio-economic context of students and schools. Last but not least, it is important to keep in mind that the absence of the reading, mathematical and scientific competencies measured by PISA does not automatically imply the presence of all those important competencies that have not been measured.     

Construct Equivalence of PISA Reading Comprehension Measured With Paper‐Based and Computer‐Based Assessments

For many years, reading comprehension in the Programme for International Student Assessment (PISA) was measured via paper‐based assessment (PBA). In the 2015 cycle, computer‐based assessment (CBA) was introduced, raising the question of whether central equivalence criteria required for a valid interpretation of the results are fulfilled. As an extension of the PISA 2012 main study in Germany, a random subsample of two intact PISA reading clusters, either computerized or paper‐based, was assessed using a random group design with an additional within‐subject variation. The results are in line with the hypothesis of construct equivalence. That is, the latent cross‐mode correlation of PISA reading comprehension was not significantly different from the expected correlation between the two clusters. Significant mode effects on item difficulties were observed for a small number of items only. Interindividual differences found in mode effects were negatively correlated with reading comprehension, but were not predicted by basic computer skills or gender. Further differences between modes were found with respect to the number of missing values.     

THE RELEVANCE AND CONSEQUENCES OF PISA SCIENCE IN A DANISH CONTEXT

This article reports a case study that described and analyzed the changes in the Danish school culture induced and encouraged by the Programme for International Student Assessment (PISA) results. The educational policy and reforms that were temporally connected with the publication of the PISA 2000 results are outlined and the related socioeconomic and sociopolitical influences are explicated. Furthermore, we investigated to what degree the PISA science assessment framework and test system were in accordance with the Danish educational goals in science in order to discuss the relevance of PISA as a catalyst for the educational actions taken. The results of our inquiry revealed areas of good correspondence and fundamental differences related to values underlying the Danish school system and PISA, respectively (e.g., Bildung orientation versus cognitive skills/competency orientation, different learning/assessment paradigms). We argue that such differences are crucial when considering curricular relevance, validity, and the use of PISA as an agent of change on the national level.     

Sources of difficulty in assessment: example of PISA science items

ABSTRACT

The understanding of what makes a question difficult is a crucial concern in assessment. To study the difficulty of test questions, we focus on the case of PISA, which assesses to what degree 15-year-old students have acquired knowledge and skills essential for full participation in society. Our research question is to identify PISA science item characteristics that could influence the item’s proficiency level. It is based on an a-priori item analysis and a statistical analysis. Results show that only the cognitive complexity and the format out of the different characteristics of PISA science items determined in our a-priori analysis have an explanatory power on an item’s proficiency levels. The proficiency level cannot be explained by the dependence/independence of the information provided in the unit and/or item introduction and the competence. We conclude that in PISA, it appears possible to anticipate a high proficiency level, that is, students’ low scores for items displaying a high cognitive complexity. In the case of a middle or low cognitive complexity level item, the cognitive complexity level is not sufficient to predict item difficulty. Other characteristics play a crucial role in item difficulty. We discuss anticipating the difficulties in assessment in a broader perspective.     

Students with special educational needs within PISA

Abstract

Since 2003, the Programme for International Student Assessment (PISA) has included students with special educational needs (SEN), identified as those with functional disabilities, those with cognitive/behavioural/emotional disabilities and those with limited test language proficiency. While the number of countries and included students has increased with each test administration, the percentage of students with SEN remains extremely low. The inclusion of these students is not an intentional PISA design parameter but rather a response to the interaction between the need to maintain strict sampling criteria and country-level educational mandates to include SEN students in standardised testing. Based on the analysis of student participation and performance across four cycles of PISA (2003–2012), this paper examines the challenges that exist in current PISA procedures related to: student sampling, eligibility and identification; assessment methodology; and reporting results. PISA practices, their limitations for scientific inferences and recommendations for design improvements are given.     

PISA 2012: how do results for the paper and computer tests compare?

The Programme for International Assessment (PISA) is an important cross-national study of 15-year olds academic achievement. Although it has traditionally been conducted using paper-and-pencil tests, the vast majority of countries will use computer-based assessment from 2015. In this paper, we consider how cross-country comparisons of children’s skills differ between paper and computer versions of the PISA mathematics test. Using data from PISA 2012, where more than 200,000 children from 32 economies completed both paper and computer versions of the mathematics assessment, we find important and interesting differences between the two sets of results. This includes a substantial drop of more than 50 PISA test points (half a standard deviation) in the average performance of children from Shanghai-China. Moreover, by considering children’s responses to particular test items, we show how differences are unlikely to be solely due to the interactive nature of certain computer test questions. The paper concludes with a discussion of what the findings imply for interpretation of PISA results in 2015 and beyond.     

The contribution of PISA to the convergence of education policies in Europe

The international comparative studies on students’ outcomes have initiated analyses that have had a growing influence on national and sub‐national education policies in industrialised and developing countries. It is particularly the case of the OECD's Programme for International Student Assessment (PISA) which started in 2000 and has organised surveys every 3 years, so that the 2015 survey was the 6^th. Its influence has been particularly important for several reasons: 1) it assesses the basic competences in reading literacy, maths and science of 15 year‐olds students, i.e. around the end of compulsory education in many countries; 2) the assessment is based on a reliable methodology and the tests are completed by qualitative surveys and studies; 3) and the results lead to recommendations and are amplified by the media in most countries. However, it is not easy to evaluate the real impact of PISA because of the existence of other international studies such as IEA's TIMSS and, particularly in Europe, the influence of the recommendations and benchmarks of the EU that has been growing steadily in the last 25 years. Our analysis of the impact of PISA and EU policy focuses on the evolution of the education policy in France, but also studies its evolution in a few other European countries. Finally, we underline the limits of the influence of PISA and international standards in education towards a convergence of education systems because of the importance of their specific historic and cultural contexts.     

PISA的教育测量技术在高考中的应用前景初探

PISA作为三年一轮针对15岁学生群体的国际评价项目已为中国教育界熟悉,其工具构造和数据分析体现了目前国际上教育测量理论和技术的最高水平。本文从教育测量专业角度归纳了PISA通过试卷矩阵设计保证考查内容覆盖广泛、利用Rasch模型打造客观等距量尺、结合考生背景解释和分析测试结果等主要技术特征,并类比分析了我国高考在相应环节的技术缺陷,展望了将这些技术移植到高考中,以达到创新考试形式,防范高考结果被滥用和误用的可能。在某省高考中试点后的结果表明,上述技术的应用使高考不只用于选拔分流,还能对评价教育质量、改进教育管理、促进教学改革发挥重要的作用。     

学生基础能力国际研究项目（PISA）的教育启示

PISA评价作为大规模的学生基础能力国际比较研究,是目前世界上最有影响力的国际学生评价项目之一。PISA应用了一系列目前世界上先进的教育测量理论和成熟的操作模式来评价15岁在校生进入未来社会所必需的知识、技能的获得情况,其评价结果具有高度的可比性、可信性和有效性,受到了各国政府的重视,被视为检验各国教育体制和未来人才竞争的重要指标。PISA项目为教育改革与发展提供了有益的启示,被誉为“教育界的世界杯”,是国际比较教育研究的重要成果。     

评估关键能力 培育生活智慧——OECD／PISA问题解决能力之构想、设计与评估

培养学生的问题解决能力是学校教育的重点内容。“国际学生评价项目”在2003年增加了对学生问题解决能力的测评。该评估项目旨在考察学生综合运用学科领域的知识,识别问题关键特征及其内在关系,能够明确界定问题,合理表征问题和有效解决问题,并能够对问题解决方案进行真实性评估、判断与交流。     

PISA Country Rankings for Italy and Spain: Revised Results

This article examines whether the way that PISA models item outcomes in mathematics affects the validity of its country rankings. As an alternative to PISA methodology, a two-parameter logistic model is applied to PISA mathematics item data from Italy and Spain for the year 2009. In the estimation procedure, item difficulty and dispersion parameters were allowed to differ across the two countries and samples were restricted to respondents who actually answered items in a mathematics cluster. Different normalizations for identifying the distribution parameters were also considered. The choice of normalization is shown to be crucial in guaranteeing certain invariance properties required by item response models. The ability or proficiency scores obtained from the methods employed here are significantly higher for Spain, in sharp contrast to PISA results, which gave both countries virtually the same rank in mathematics (489 for Italy and 488 for Spain). These results raise serious questions about PISA methodology and the role that PISA results play in the formulation educational policy in member countries.     

A CRITICAL EXAMINATION OF PISA’S ASSESSMENT ON SCIENTIFIC LITERACY

The OECD “Programme for International Student Assessment” or (PISA) is one of the largest-scale international efforts that have been launched to assess students’ scientific literacy. Such an international assessment would likely exert a profound impact on the science education policies of the participating countries/regions, including Hong Kong. This paper sets out to examine critically how scientific literacy has been assessed by PISA through analyzing its assessment frameworks and released sample items. It was found that the PISA 2000 and 2003 assessments of science have used a narrower definition of scientific literacy, as compared to that of PISA 2006 and what scientific literacy was construed for science education. However, even PISA 2006 appears to be more valid in its assessment framework, its validity was also called into question when the sample items for the trial study were examined. Knowledge about science was found largely about the processes of science, rather than the nature of science as described in the assessment framework. Besides, it intertwined with knowledge of science in a hidden manner. The application of knowledge of science in novel, real-life situations was also jeopardized because of the issue of curricular relevance. Besides these major problems, the article has discussed the problems with the concept of scientifically investigable questions and identifying research question of an investigation. Overall, the findings raised concern over what the PISA’s measure of scientific literacy actually means.     

指向创造性思维:PISA 2021的启示与借鉴

创造性思维是人类发展所需的必要能力,可以帮助人们适应不断变化的世界和应对充满挑战的未来。经济合作与发展组织确定在PISA 2021中增加对创造性思维能力的评估,其发布的《PISA 2021创造性思维评估框架草案(第三版)》明确阐述了创造性思维的内涵、表现形式和促成因素,以系统的通用框架、科学简易的"三维度四领域"能力模型向公众提供了一个操作性强的评估系统。通过此次评估,各参与国家和地区可获得学生创造性思维能力的可比数据,为未来教育政策的制定和教育实践的改进提供支持。基于PISA的经验,为了更好地评估和培养学生的创造性思维,我国可借鉴创造性思维能力模型,细化学科核心素养的考查;构建创造性课堂,加强学校创新氛围的建设;在课堂教学中以真实情境和实际问题为载体,培育和评价学生的创造性思维。     

PISA对大规模教育质量的评价解读

大规模教育质量的评价在很大程度上影响着国家和地区教育发展的走向。本文以PISA2006结果报告为中心,解析PISA对大规模教育质量和相关影响因素评价的理念和方法,为研究和开展我国大规模教育质量评价提供借鉴。     

Computer-based collaborative problem-solving assessment in Taiwan

Abstract

The current study explores students’ collaboration and problem solving (CPS) abilities using a human-to-agent (H-A) computer-based collaborative problem solving assessment. Five CPS assessment units with 76 conversation-based items were constructed using the PISA 2015 CPS framework. In the experiment, 53,855 ninth and tenth graders in Taiwan were recruited, and a multidimensional item response analysis was used to develop CPS scales and represent the students’ collaboration and problem solving performance. The results show that the developed H-A approach is feasible for measuring students’ CPS skills, and the CPS scales are also shown to be reliable. In addition, the students’ CPS performance scores are further explored and discussed under the PISA CPS framework.     

The role that mathematics plays in college- and career-readiness: evidence from PISA

Many studies have found a strong relationship between the mathematics students study in school and their performance on an academic or school mathematics assessment but not on an assessment of mathematics literacy (ML). With many countries, like the USA, placing emphasis on finishing secondary education being mathematically literate and prepared for college or career, this raises the question about the relationship between the mathematics studied in school and any ML students may have. The Programme for International Student Assessment (PISA) ML assessment is embedded in real-world contexts that provide an important window on how ready students are to tackle the situations and problems that await them whether they intend to pursue further education beyond high school or intend to go directly into the labour force. In this paper, we draw upon the PISA 2012 data to investigate the extent to which the cumulative exposure to rigorous mathematics content, such as that embedded in college- and career-ready standards, is associated with ML as assessed in PISA. Results reveal that both exposure to rigorous school mathematics and experiencing the instruction of this mathematics through real-world applications are significantly related to all the real-world contextualized PISA ML scores.     

Response to 'Cautions on OECD's Recent Educational Survey (PISA)'

In his recent paper, 'Cautions on OECD's recent educational survey (PISA)' ( Oxford Review of Education , 29, 2), S.J. Prais questioned the outcomes of the Organisation for Economic Cooperation and Development's PISA survey of the reading, mathematics and science attainments of 15-year-olds. Prais suggested that methodological flaws in PISA had resulted in an apparent improvement in the attainment of British students--particularly when compared to their Swiss and German counterparts. This paper responds to Prais's criticisms, noting that when Prais's conjectures are tested with empirical data they are not supported. Further it is noted that many of Prais's criticisms are due to an incomplete understanding and knowledge of the methodology of international studies, and of PISA in particular.     

When Assessment Validation Neglects Any Strand of Validity Evidence: An Instructive Example from PISA

The Standards for Educational and Psychological Testing identify several strands of validity evidence that may be needed as support for particular interpretations and uses of assessments. Yet assessment validation often does not seem guided by these Standards, with validations lacking a particular strand even when it appears relevant to an assessment. Consequently, the degree to which validity evidence supports the proposed interpretation and use of the assessment may be compromised. Guided by the Standards, this article presents an independent validation of OECD's PISA assessment of mathematical self-efficacy (MSE) as an instructive example of this issue. OECD identifies MSE as one of a number of “factors” explaining student performance in mathematics, thereby serving the “policy orientation” of PISA. However, this independent validation identifies significant shortcomings in the strands of validity evidence available to support this interpretation and use of the assessment. The article therefore demonstrates how the Standards can guide the planning of a validation to ensure it generates the validity evidence relevant to an interpretive argument, particularly for an international large-scale assessment such as PISA. The implication is that assessment validation could yet benefit from the Standards as what Zumbo calls “a global force for testing”.     

Fifteen years of research on PISA effects on education governance: A critical review

This article provides a literature review on the effects of the OECD's Programme for International Student Assessment (PISA) on education governance and policy process across participating countries. This review seemed necessary because there has been a growing body of literature on this topic since 2003, especially since 2010, because this literature is not always well‐known and because the discourse on the so‐called ‘PISA shock’ remains important, even if it is more of a metaphor than a concept and may be politically partial. The article exploits a dataset of 87 references which show that PISA introduced major changes in the governance of education worldwide. Driven by soft power strategies and new policy transfers, this governance is based on data and measurement tools which redefine the scales of education policies. It also shows that PISA has a strong influence on a variety of national reforms, as illustrated in many case studies. However, this influence strongly depends on domestic policy contexts that scholars intended to capture through different theoretical frameworks. Nonetheless, few propose overarching theorisations of the political meaning of PISA effects on education governance and policy processes. The article concludes by stressing three main challenges for the subsequent studies on these PISA effects: better conceptualising these effects, preserving an epistemology of uncertainty in order to avoid taken for granted views and normalising the research on PISA effects not to perpetually and artificially rediscover its so‐called novelty.