Can competencies assessed by PISA be considered the fundamental school knowledge 15-year-olds should possess?

The OECD Programme for International Student Assessment (PISA), launched by governments of the Organisation for Economic Co-operation and Development (OECD) in 1997, aims at assessing some of the key competencies that contribute to the success of 15-year-old individuals, on a regular basis and within an internationally accepted framework. PISA seeks to provide a basis for policy dialogue and for collaboration in defining and implementing educational goals, in innovative ways that reflect judgements about the skills that are relevant to adult life. PISA defines competence as the ability to successfully meet complex demands in varied contexts through the mobilisation of psychosocial resources, including knowledge and skills, motivation, attitudes, emotions, and other social and behavioural components. Measuring and comparing competencies across languages and cultures is a difficult challenge and is being pursued by PISA progressively. PISA focused its first three assessments on literacy skills, defined as the capacity of young adults to access, manage, integrate and evaluate information, to think imaginatively, to hypothesise and discover, and to communicate their thoughts and ideas effectively. The reasoning behind shifting the emphasis from assessing whether students can reproduce what they have learned towards whether they can extrapolate from what they have learned and apply their competencies in novel situations, derives from the nature of knowledge and skills required in modern life. For example, the tasks that can be solved through simple memorisation or with pre-set algorithms are those that are also easiest to digitise, automatise and offshore, and will thus be less relevant in a modern knowledge society. Since there is no overarching cross-national and cross-cultural agreement on what fundamental competencies 15-year-olds should possess, an international assessment such as PISA can only capture a selection of competencies. Moreover, since various methodological constraints limit the nature of competencies that are currently amenable to large scale assessment, PISA cannot capture the entirety of competencies that will make young people successful. However, the findings presented in this article suggest that the competencies that PISA does assess are highly predictive for the future success of students. In addition, PISA provides policy makers and practitioners with useful tools to improve quality, equity and efficiency in education, by revealing some common characteristics of students, schools and education systems that do well. In a modern world, comparative assessments are an essential tool for educational improvement and research shows that the existence of standardised assessments and examinations is one of the most powerful predictors for the success of an education system. That is not hard to understand, because without such assessments, all students, schools and education systems look the same, it is impossible for teachers and school administrators to detect institutional and systemic strengths and weaknesses, and to support and intervene where expectations are not met. Without reliable and comparable information on learning outcomes, teachers and governments alike rely on input-based incentives and policies that are all too often mirrored in large quality variation between schools as well as a strong dependency between learning success and the socio-economic context of students and schools. Last but not least, it is important to keep in mind that the absence of the reading, mathematical and scientific competencies measured by PISA does not automatically imply the presence of all those important competencies that have not been measured.     

The PISA Tasks: Unveiling Prospective Elementary Mathematics Teachers’ Difficulties with Contextual,Conceptual, and Procedural Knowledge

The aim of this mixed methods study was to investigate the difficulties prospective elementary mathematics teachers have in solving the Programme for International Student Assessment (PISA) 2012 released items. A mathematics test consisting of 26 PISA items was administered, followed by interviews. Multiple data were utilized to provide rich insights into the types of mathematical knowledge that a particular item requires and prospective teachers’ difficulties in using these knowledge types. A sample of 52 prospective teachers worked the mathematics test, and 12 of them were interviewed afterwards. The data-sets were complementary: the quantitative data showed that PISA items could be categorized under contextual, conceptual, and procedural knowledge and indicated the most frequent difficulties in the combined contextual, conceptual, and procedural knowledge items. The qualitative data revealed that few prospective teachers could give mathematical explanations for conceptual knowledge items, and that their contextual knowledge was fragmented. Educational implications were discussed.     

Using Large-scale Assessment Datasets for Research in Science and Mathematics Education: Programme for International Student Assessment (PISA)

Large-scale assessments of student achievement provide a window into the broadly defined concepts of literacy and generate information about levels and types of student achievement in relation to some of the correlates of learning, such as student background, attitudes, and perceptions, and perhaps school and home characteristics. This paper provides an overview and outlines potential research opportunities of one such assessment—the Programme for International Student Assessment (PISA). In order to provide examples of the work that can be accomplished with these data, we describe and discuss the results generated from PISA 2000 and PISA 2003 in terms of international comparisons of achievement and the models of relational patterns of student, home, and school characteristics. We provide insight from the recent pilot testing conducted in Taiwan for PISA 2006, which has a focus on scientific literacy. This is followed by a discussion of the implications and potentials of the 2000 and 2003 datasets to facilitate research on scientific and mathematical literacy. The paper concludes with a look ahead to PISA 2006 and what researchers should be attending to in the research reports generated from the OECD and the research interests that they could follow given access to the datasets generated.

Assessment drives learning: The effect of central exit exams on curricular knowledge and mathematical literacy

In this paper, we use data from the German PISA 2003 sample to study the effects of central exit examinations on student performance and student attitudes. Unlike earlier studies we use (i) a value-added measure to pin down the effect of central exit exams on learning in the last year before the exam and (ii) separate test scores for mathematical literacy and curriculum-based knowledge. The findings indicate that central exit exams improve curriculum-based knowledge but do not affect mathematical literacy. Students, although showing a better performance, are less intrinsically motivated in school.     

PISA as a Political Tool in Spain: Assessment Instrument,Academic Discourse and Political Reform

This study examines to what extent there is a sort of “political appropiation” by political parties when they seek to set a discourse about the Spanish PISA outcomes. We have consistently found that programs for assessing the competencies of students, especially PISA, have become tools of rationalization and the legitimization of education policies in both the national and territorial contexts. Furthermore, each regional government focuses its discourse on what it feels to be its strong points, that is, a clear “self-justifying reading” of PISA outcomes.     

How Chinese students’ scientific competencies are influenced by their attitudes?

ABSTRACT

In recent years, research on students’ scientific competencies has generated much discussion. According to the PISA 2015 Assessment and Analytical Framework, both scientific competencies and attitudes towards science – interest in science, valuing scientific approaches to enquiry (or epistemological beliefs about science), and environmental awareness – are integral aspects of scientific literacy. However, few studies have focused on the relationships between them. This article examines how students’ scientific competencies are affected by their attitudes. Based on a theoretical analysis, the present study argues that the three attitudinal factors have a direct impact on scientific competencies, interest has an immediate influence on the other two attitudes, and epistemological beliefs are directly affected by environmental awareness. To test this hypothesis, a model was constructed to verify such relationships. PISA 2015 assessment instruments were used for data collection. The participants were 25,658 students from different parts of China, containing Beijing, Shanghai, Jiangsu, Guangdong, Taiwan, Hong Kong, and Macao. Structural Equation Modelling (SEM) was employed for data analysis. The results confirmed our hypothesis. The way in which Chinese students’ attitudes affected their scientific competencies did not differ (to a statistically significant level) by gender or district. The limitations and implications of this research are discussed below.     

A CRITICAL EXAMINATION OF PISA’S ASSESSMENT ON SCIENTIFIC LITERACY

The OECD “Programme for International Student Assessment” or (PISA) is one of the largest-scale international efforts that have been launched to assess students’ scientific literacy. Such an international assessment would likely exert a profound impact on the science education policies of the participating countries/regions, including Hong Kong. This paper sets out to examine critically how scientific literacy has been assessed by PISA through analyzing its assessment frameworks and released sample items. It was found that the PISA 2000 and 2003 assessments of science have used a narrower definition of scientific literacy, as compared to that of PISA 2006 and what scientific literacy was construed for science education. However, even PISA 2006 appears to be more valid in its assessment framework, its validity was also called into question when the sample items for the trial study were examined. Knowledge about science was found largely about the processes of science, rather than the nature of science as described in the assessment framework. Besides, it intertwined with knowledge of science in a hidden manner. The application of knowledge of science in novel, real-life situations was also jeopardized because of the issue of curricular relevance. Besides these major problems, the article has discussed the problems with the concept of scientifically investigable questions and identifying research question of an investigation. Overall, the findings raised concern over what the PISA’s measure of scientific literacy actually means.     

The impact of PISA and the interrelation and development of assessment policy and assessment theory in Japan

PISA presents a new image for academic achievement, which has prompted Japanese education reforms over the past decade to innovate teaching and learning for ‘PISA-style literacy’. Supported by theoretical foundations, particularly with regard to the concept of ‘PISA literacy’ and ‘authentic assessment’, these reforms have accomplished progress in the focus on higher order competencies, such as application and the development of new assessment strategies. However, more recently, various critical discussions of ‘PISA literacy’ are underway in the Japanese academy. They interrogate it more critically and reveal the narrow emphasis on functional application and technical operation. Current assessment practices, which tend to fall into ‘criteria compliance’, are in urgent need of review. There is a need to extend the critical discussions in progress to the new assessment strategies. This paper responds to this, by considering the Japanese acceptance of ‘PISA literacy’ and its assessment, discussing the features and limitations.     

PISA2006科学试题的设计和开发及其启示

学业成就评价是当前新课程改革研究的热点之一。如何科学地设计和开发试题,对深化新课改、进行基础教育质量监控有着重要意义。PISA是一项权威的国际学生评价项目,具有较高的可比性、可信性和有效性。PISA2006科学评估框架包含情境、知识、态度和能力等相互联系的四个方面,其试题设计和开发技术采用了"双位编码"评分设计,增加了态度评估试题,保证了试题与标准的匹配。     

评估关键能力 培育生活智慧——OECD／PISA问题解决能力之构想、设计与评估

培养学生的问题解决能力是学校教育的重点内容。“国际学生评价项目”在2003年增加了对学生问题解决能力的测评。该评估项目旨在考察学生综合运用学科领域的知识,识别问题关键特征及其内在关系,能够明确界定问题,合理表征问题和有效解决问题,并能够对问题解决方案进行真实性评估、判断与交流。     

PISA Test Items and School-Based Examinations in Greece: Exploring the relationship between global and local assessment discourses

The paper explores the relationship of the global and the local assessment discourses as expressed by Programme for International Student Assessment (PISA) test items and school-based examinations, respectively. To this end, the paper compares PISA test items related to living systems and the context of life, health, and environment, with Greek school–based biology examinations' test items in terms of the nature of their textual construction. This nature is determined by the interplay of the notions of classification (content specialisation) and formality (code specialisation) modulated by both the linguistic and the visual expressive modes. The results of the analysis reveal disparities between assessment discourses promoted at the global and the local level. In particular, while PISA test items convey their scientific message (specialised content and code) principally through their visual mode, the specialised scientific meaning of school-based examinations test is mainly conveyed through their linguistic mode. On the other hand, the linguistic mode of PISA test items is mainly compatible with textual practices of the public domain (non-specialised content and code). Such a mismatch between assessment discourses at local and global level is expected to place Greek students at different discursive positions, promoting different types of knowledge. The expected shift from the epistemic positioning promoted in Greece to the one promoted by PISA could significantly restrict Greek students' ability to infer the PISA discursive context and produce appropriate responses. This factor could provide a meaningful contribution in the discussion of the relatively low achievement of Greek students in PISA scientific literacy assessment.     

评估科学素养 培育关键能力——OECD／PISA科学素养之构想、设计与评估

科学素养被认为是21世纪公民生活和工作所必须具备的关键能力的核心,而培养学生的科学素养就成了教育的重要使命之一,如何测评学生的科学素养也因此成了研究的重点。本文试图通过介绍欧洲经济合作与发展组织的“国际学生评价项目”对科学素养的界定和评估框架,为评价学生的综合能力提供借鉴。     

The OECD and the local: PISA-based Test for Schools in the USA

The Organization for Economic Cooperation and Development (OECD) has recently piloted the Program for International Student Assessment (PISA)-based Test for Schools in the USA. In this paper, I contend that by connecting directly with local school boards this new initiative has the potential to further promote the OECD's educational agenda in local policy debates. I begin to develop this argument by providing an overview of the OECD and its work. I then lay out a theoretical framework around global governance and knowledge production within the context of the OECD. Next, I provide a brief overview of the traditional PISA study and compare it to the new PISA-based Test for Schools initiative. This context provides the foundation for a discussion of the ways in which a school-based international assessment can operate as a governance tool, allowing international organizations to have greater influence in the formation and implementation of local educational policies.     

Effect of School Decentralization and School Climate on Student Mathematics Performance: The Case of Hong Kong

For more than a decade, the Government of Hong Kong has instituted a policy of school decentralization aimed at devolving authority to all stakeholders, including the representatives of School Sponsoring Bodies, principals, teachers, parents, and community members. This study examines the relative contribution of two dimensions of school decentralization – teacher participation and school autonomy – to students' mathematics performance, and examines the role of school climate as a mediating variable between decentralization and performance. Data for the present study come from the second cycle of the Programme for International Student Assessment (PISA 2003), which is a large international assessment measuring 15-year-old students' literacy performance across over 40 countries and regions around the world. Using multi-level analysis, the study finds that teacher participation is more important than school autonomy for student mathematics performance in Hong Kong. In addition, the effect of teacher participation on students' performance is mediated by four major school climatic factors – sense of belonging, disciplinary climate, students' morale and student behaviour – in Hong Kong's secondary schools.     

基于数学核心素养评价框架的试卷测评研究——以2019年高考全国卷为例

数学核心素养的测评研究是反馈学生核心素养培养成效的重要手段,高考试题是检测学生数学素养的重要载体,因此开展以高考试卷为样本的数学核心素养测评研究对中国新一轮高考改革有重要意义。以2019年全国卷为样本,基于喻平提出的数学核心素养评价框架,给出了数学核心素养教育评价表,利用多元统计分析方法研究了全国卷的数学核心素养考查特点,并提出了高考命题建议,为基于试卷的核心素养测评研究提供一定的参考。     

Gender equity in mathematical achievement: the case of China

Gender differences in achievement in mathematics, a traditionally male-stereotyped subject, have long been a concern for many educators around the world. Gender differences in mathematical achievement have decreased in recent decades, especially in Western countries, and become small or insignificant in large-scale tests, such as the Programme for International Student Assessment (PISA). The situation in China has not yet been studied. The recent PISA report lists China B-S-J-G (representing Beijing–Shanghai–Jiangsu–Guangdong) as an educational system with no significant gender difference in mathematical achievement. Based on a secondary analysis of PISA 2015 mathematics data from China B-S-J-G, this study more deeply scrutinized gender differences in Chinese students’ mathematical performance, emphasizing societal factors, namely students’ socioeconomic status, school level, school type, school location, and socioeconomic status at school level. This analysis revealed important differences within the overall picture. Most importantly, significantly more boys than girls scored in the top tier of mathematics achievement. At the lower- and upper-secondary school levels, boys performed significantly better than girls, with the achievement difference increasing at the upper-secondary level. Furthermore, this study found that, on average, Chinese (B-S-J-G) girls achieved significantly lower average scores on the PISA 2015 mathematics test than boys in the same school. Overall, students’ individual characteristics and school characteristics need to be separated and both taken into account to examine the role of gender in mathematical achievement, which has not been thoroughly investigated in the past.     

The role that mathematics plays in college- and career-readiness: evidence from PISA

Many studies have found a strong relationship between the mathematics students study in school and their performance on an academic or school mathematics assessment but not on an assessment of mathematics literacy (ML). With many countries, like the USA, placing emphasis on finishing secondary education being mathematically literate and prepared for college or career, this raises the question about the relationship between the mathematics studied in school and any ML students may have. The Programme for International Student Assessment (PISA) ML assessment is embedded in real-world contexts that provide an important window on how ready students are to tackle the situations and problems that await them whether they intend to pursue further education beyond high school or intend to go directly into the labour force. In this paper, we draw upon the PISA 2012 data to investigate the extent to which the cumulative exposure to rigorous mathematics content, such as that embedded in college- and career-ready standards, is associated with ML as assessed in PISA. Results reveal that both exposure to rigorous school mathematics and experiencing the instruction of this mathematics through real-world applications are significantly related to all the real-world contextualized PISA ML scores.     

Good Practice in Science Teaching: What Research has to Say (2nd edn)

The scientific competencies advocated by the Programme for International Student Assessment (PISA) focus on the abilities needed in students' adult lives. This study investigated how such scientific competencies could be improved by using online argumentation. One hundred and thirty-eight 8th grade high school students took part in the study, with 69 in the experimental group and 69 in the control group. A quasi-experimental design was adopted and qualitative and quantitative analyses were used. An online argumentation system served as an aid for argumentation instruction and activities among experimental group students during the experiment. The results showed that using online argumentation could improve the students' scores for the PISA scientific competencies. The experimental group students outperformed their counterparts in terms of overall mean scores for the scientific competencies. On the one hand, the individual competencies of ‘using scientific evidence’ and ‘identifying scientific issues’ of the experimental group were higher than those of the control group. On the other hand, the experimental group students did not outperform their counterparts in terms of competency in ‘explaining phenomena scientifically’. Using an online environment to complement argumentation instruction and organizing argumentation activities focused on related topics may be a potential direction to consider for improving students’ PISA scientific competencies.     

Counting and comparing school performance: an analysis of media coverage of PISA in Australia, 2000–2014

This paper empirically documents media portrayals of Australia’s performance on the Program for the International Student Assessment (PISA), 2000–2014. We analyse newspaper articles from two national and eight metropolitan newspapers. This analysis demonstrates increased media coverage of PISA over the period in question. Our research data were analysed using ‘framing theory’, documenting how the media frames stories about Australia’s performance on PISA. Three frames were identified: counts and comparisons; criticisms; and contexts. Most of the media coverage (41%) was concerned with the first frame, counts and comparisons, which analysed PISA data to provide ‘evidence’ that was then used to comparatively position Australia against other countries, reference societies, which do better, with particular emphasis on Finland and also Shanghai after the 2009 PISA. The other two frames dealt with criticisms and contextual issues. This paper only focuses on the first frame. The analysis demonstrates the ways in which media coverage of Australia’s PISA performance has had policy impact.