Early tracking and different types of inequalities in achievement: difference-in-differences evidence from 20 years of large-scale assessments

Educational Assessment, Evaluation and Accountability - Research to date on the effects of between-school tracking on inequalities in achievement and on performance has been inconclusive. A...     

School accountability and teacher stress: international evidence from the OECD TALIS study

Educational Assessment, Evaluation and Accountability - Accountability—the monitoring and use of student performance data to make judgements about school and teacher effectiveness—is...     

Linking errors in international large-scale assessments: calculation of standard errors for trend estimation

Abstract

One major aim of international large-scale assessments (ILSAs) is to monitor changes in student performance over time. To accomplish this task, a set of common items is repeatedly administered in each assessment and linking methods are used to align the results from the different assessments on a common scale. The present article introduces a framework for discussing linking errors in ILSAs, in which different components of linking errors are distinguished (country-by-item interaction, assessment-by-item interaction and country-by-assessment-by-item interaction). Furthermore, the different components of linking errors are used to analytically derive standard errors for national trend estimates. In a simulation study, the proposed standard error formula outperforms the method that is used in PISA. In addition, the PISA 2006 and 2009 reading data are used to illustrate how the interpretation of national trend estimates can change when different procedures are applied to calculate standard errors.     

Unpacking equity. Educational equity in secondary analyses of international large-scale assessments: A systematic review

The discourse of educational equity has captured public imagination and become an entrenched policy value. In the attendant struggle to improve educational equity, governments draw inspiration from analyses based on data from highly standardized, international large-scale assessments. As these analyses are used as a lever for policy formulation, it is important to understand how this concept of equity is appropriated. Lacking a synthesis of how equity is conceptualized and operationalized in this body of research, the current paper seeks to fill that gap by presenting a systematic review. Our analysis incorporates an identification of operational patterns and an exploration regarding the linguistic elements for defining equity. We chart the theoretical and methodological diversity among interpretations of equity, as clustered into five major research approaches. In light of these results, benefits and limitations of the concept's complexity are discussed and implications for research are forwarded.     

Studying the expertise reversal of the multimedia signaling effect at a process level: evidence from eye tracking

Instructional Science - The purpose of this study was to shed light on the cognitive processes underlying the expertise reversal effect related to multimedia signaling. Multimedia signals highlight...     

Self-regulated learning processes vary as a function of epistemic beliefs and contexts: Mixed method evidence from eye tracking and concurrent and retrospective reports

The objective of the current studies was to investigate how epistemic cognition related to specific phases and components of self-regulated learning and its adaptation to learning conditions of varying quality. In a multi-study, mixed method design, we presented university students with science content that relayed conceptual discrepancies and collected quantitative and qualitative data to study how students responded to discrepancies. In Study 1 (n = 42), we collected eye tracking patterns, study times, and metacognitive ratings and found that participants adapted their behavioral processing as a function of their epistemic cognition and discrepancy type. In Study 2 (n = 20), we collected concurrent think-aloud protocols and retrospective interviews to further explore why discrepancies were noticed (or not) and how they were resolved. Results revealed that prior knowledge and epistemic self-efficacy in oneself as an evaluator of knowledge emerged as important themes to detecting and efficiently resolving discrepancies. We conclude with a discussion of theoretical and methodological implications.     

Making sense out of measurement non-invariance: how to explore differences among educational systems in international large-scale assessments

International large-scale assessment in education aims to compare educational achievement across many countries. Differences between countries in language, culture, and education give rise to differential item functioning (DIF). For many decades, DIF has been regarded as a nuisance and a threat to validity. In this paper, we take a different stance and argue that DIF holds essential information about the differences between countries. To uncover this information, we explore the use of multivariate analysis techniques as ways to analyze DIF emphasizing visualization. PISA 2012 data are used for illustration.

     

Middle leadership in international schools: evidence from Malaysia

This paper examines middle leadership of the heads of English, maths and science departments in four international secondary schools in Malaysia. It focuses on their roles, responsibilities, role relationships, instructional engagement and leadership involvement within the theoretical framework of instructional, distributed and teacher leadership. The study is a qualitative multi-method case study, involving observation, documentary analysis, and semi-structured interviews with 12 heads of department, 36 teachers, and four principals. With respect to the middle leaders’ roles and responsibilities, the findings illustrate cross-school and in-school differential developments, with contextual factors contributing to uneasy role relationships. The results also show that, while broad-based leadership opportunities are limited for the middle leaders and teachers, the most powerful and common feature of all these international schools is the centrality of teaching and learning.     

Understanding the response to financial and non-financial incentives in education: Field experimental evidence using high-stakes assessments

We analyze the impact on high-stakes assessments of incentivizing students’ effort in a field experiment with over 10,000 high school students. We contribute to the literature by using our rich data and machine learning techniques to explore treatment heterogeneity; by comparing financial and non-financial rewards in rewarding effort rather than grades; and by using high-stakes outcomes. We find little average impact of incentives in the overall population, but we identify a “right tail” of highly responsive students: in the upper half of the responsiveness distribution, test scores improve by 0.1-0.2 SD, about half the attainment gap between poor and non-poor students.     

Prosody in skilled silent reading: evidence from eye movements

Recent eye movement experiments offer preliminary evidence that skilled readers activate word‐level prosodic information when silently reading sentences. This paper reviews the role of eye movements during reading as well as the preliminary evidence for prosodic processing. A new experiment examines whether prosodic processing differs for high and low frequency words. Readers' eye movements were monitored while reading target words presented in sentences preceded by parafoveal previews that either contained the exact initial syllable of the target (i.e. the congruent preview condition) or the initial syllable plus the next letter (i.e. the incongruent preview condition). Reading times on high frequency words did not differ in the congruent and incongruent preview conditions, but reading times on low frequency words were faster in the congruent condition. The implications of the present result and previous studies are discussed in terms of phonological hub theory, which is a production‐based theory of word recognition during skilled silent reading.     

How to improve the design of experimental studies in computing education: Evidence from the international assessments

Education and Information Technologies - Cluster randomized trials are frequently used in educational research for methodological reasons. This study aims to improve the efficiency of cluster...     

Promoting teaching,learning and informed decision-making through the lenses of international large-scale assessment: looking beyond numbers

Public discussion on the quality of education in different corners of the world very much relies on the data provided by the international large-scale assessment (ILSA) studies. While aware of different methodological keystones and technicalities embedded in these, the idea behind this special issue is to contribute to the understanding of how students solve ILSA items and the kinds of literacy skills they rely on. A shared element of reported studies is that they seek to document and analyse test-taking as a concrete practice using different theoretical and methodological perspectives. We find such scrutiny to be important and significant at this point in time as it offers some new lenses through which ILSA items could be inspected and further discussed.     

Exploring the lack of a disfluency effect: evidence from eye movements

An eye-tracking study with 60 native Swedish speakers (18–30 years) was conducted to investigate the positive effects on learning outcomes predicted by the disfluency effect. Subtle low-pass filtering was used as a disfluency manipulation and compared with a control condition using regular text. The text was presented on four separate text presentation screens (TPSs), and eye movements were recorded. A free recall task was given 25 min later, and working memory capacity (WMC) was assessed to test if it would moderate learning outcomes. The disfluency manipulation had no effect on learning outcomes, total reading times on words or lines, first- or second-pass reading on lines, or average fixation durations. Moreover, the disfluency effect was not moderated by students’ WMC or self-reported prior knowledge of the topic. However, an adaptation to the disfluency manipulation was found. Total reading times on both words and lines were shorter in TPS 1 and 2 in the disfluency condition compared with the control condition, whereas reading times were longer in TPS 3 and 4. It is discussed if failures to replicate the disfluency effect arise from material features, with positive adaptations to disfluency (i.e., higher effort investment) possibly requiring more comprehensive materials.     

Understanding business interests in international large-scale student assessments: a media analysis of The Economist,Financial Times,and Wall Street Journal

The media analysis is situated in the larger body of studies that explore the varied reasons why different policy actors advocate for international large-scale student assessments (ILSAs) and adds to the research on the fast advance of the global education industry. The analysis of The Economist, Financial Times, and Wall Street Journal covers publications on ‘PISA’, ‘TIMSS’, and related search items over the period 1996–2016. The three media outlets vary in terms of ILSA reporting. The Economist and Financial Times tend to focus on PISA, whereas the Wall Street Journal pays greater attention to TIMSS than PISA. The content analysis of 59 articles yields interesting results about how the business-oriented readership of the three media outlets frames public education and why it sees education as a profitable business opportunity. The three most common narratives, reflecting the business logic, are the following: (i) public education is in crisis; (ii) there is no correlation between spending and education outcome; and (iii) school accountability, teacher performance, and decentralisation represent the most effective policies to improve the quality of education. Drawing on these three common narratives, the financial media outlets present a particular vision of how to improve education; a vision in which the private sector is supposed to play a major role.     

Adult numeracy and the totally pedagogised society: PIAAC and other international surveys in the context of global educational policy on lifelong learning

This paper aims to discuss the emergence, form and likely effects of international surveys of adults’ skills by locating them in the global context of policies on education and Life Long Learning (LLL). It focuses on adults’ numeracy and discusses its conceptualisation and assessment in the Project for the International Assessment of Adult Competencies (PIAAC), which is the most recent survey. Drawing on critical theoretical resources about new forms of governance in education and transformations in the pedagogic discourse, the paper further substantiates existing critiques of global policy trends, namely that they are motivated by human capital approaches to education and LLL. In particular, we show that the apparently commonsensical appeal of evaluative instruments like PISA and PIAAC is based on a competency model of knowledge, which embodies a narrow notion of competence. Relatedly, the notional curricula promoted by such surveys potentially articulate a more radical idea of LLL, captured by Bernstein’s conception of trainability as the mode of socialisation into a Totally Pedagogised Society. The paper presents a dual approach to understanding international adult performance surveys in general—in that, besides deploying the theoretical resources already indicated, it also raises a number of methodological issues relevant to the valid interpretation of these studies’ results. Ultimately, it argues for the importance of mobilising resources from critical educational perspectives to support the development of potentially powerful knowledge like numeracy and to prevent its being reduced to a narrow competency.     

民众的腐败容忍度: 实证研究及启示——基于世界价值观调查数据的分析

要从根本上消除腐败,不仅需要纪检监察机关加强对政府及公职人员的监督,也需要社会公众的积极参与。降低腐败容忍度是提高公众反腐败参与意愿和实际行动的关键。基于全国314个地级行政区的数据分析发现,政府治理、经济发展和社会文化等三个维度对公众的腐败容忍度有显著影响。地区的政府科教事业支出越高,城乡收入差距越大,离首都的距离越远,则公众的腐败容忍度越高;地区的社会信任度越高,对中央政府的政治信任度、索贿指数越高,则公众的腐败容忍度越低;中西部地区的经济发展与腐败容忍度呈倒U型曲线关系。当前需要持续提高经济发展水平,缩小城乡收入差距,增强公众的政治信任和社会信任度,进一步加大反腐败力度,清除群众身边的腐败现象,从而切实降低公众的腐败容忍度,形成政府与社会共同反腐的合力,巩固发展反腐败斗争压倒性胜利。     

Teaching for equity: insights from international evidence with implications for a teacher education curriculum

Researchers, practitioners, and policy-makers in many countries are grappling with ways to address the persistent problem of inequitable educational outcomes between advantaged and disadvantaged students. This paper reports the results of a unique cross-country, cross-cultural analysis undertaken to provide insights into teaching practices that promote equity, drawing on programmes of empirical research or syntheses of major programmes of research that worked from a complex, non-linear view of teaching and its outcomes. We analysed international evidence about teaching practices that have a positive influence on diverse students’ learning outcomes and opportunities and then compared and contrasted the results of these analyses. From the commonalities we identified, we derived six interconnected facets of practice for equity, which are general principles of practice rather than specific teaching strategies or behaviours. Building on these facets, we developed a conceptual framework that can inform an equity-centred teacher education curriculum that specifically addresses the task of preparing teachers who can make a positive difference to the learning opportunities and outcomes of diverse students, particularly those historically disadvantaged by the education system.     

International assessments of student achievement and public confidence in education: evidence from a cross-national study

One of the overarching goals of international large-scale assessments (ILSA) is to inform public discourse about the quality of education in different countries. To fulfil this function, the Organisation for Economic Co-operation and Development (OECD), for example, raises awareness of the Program for International Student Assessment (PISA) results through different forms of traditional and social media (e.g. press releases and other activities under the slogan PISA Day). Scholars have responded to the rapid growth of ILSA by examining public discourse through newspaper articles, policy documents, and other outlets. However, we know very little about whether and to what extent the general public is actually affected by PISA results. In order to address this gap, this study uses data regarding public trust in education from the 2011 wave of the International Social Survey Program (ISSP). Drawing on survey data from 30 countries and Hierarchical Linear Models (HLM), the study shows that PISA rankings have a significant effect on public perceptions. We find that in high performing countries the general public expresses higher levels of confidence in the education system. We discuss these patterns in the context of growing politicisation of education policy making and the use of ILSA as evidence.     

Whose knowledge counts in international student assessments: examining the AHELO epistemic community of economics experts

International student assessments have become the ‘lifeblood’ of the accountability movement in educational policy contexts. Drawing upon Stuart Hall’s concept of representation, I critically examined who comprises epistemic communities responsible for developing the Organization for Economic Co-operation and Development’s Assessment of Higher Education Learning Outcomes economics strand. Through my analysis, I identify homogeneity among test-makers, missing voices, and western concepts as represented ideas that become fixed and circulated through policy channels. In an era of accountability and comparison, comparative international student assessments are becoming increasingly normative in educational settings. As scholars, policymakers, and educators use these test results, we must critically examine who is responsible for producing these tests.     

朗读条件下词切分方式对说明文阅读影响的眼动研究

在以采用默读方式研究词切分对阅读绩效影响研究的基础上,就朗读务件下词切分对说明文阅读绩效的影响进行了研究.实验一采用单因素4水平(呈现方式:无空格,词间空格,非词空格,字间空格)重复测量实验设计;实验二采用灰色突出显示词边界,确保四种呈现方式下材料的长度相同进行进一步探讨.结果发现:(1)在汉语语篇中插入词间空格会促进朗读;(2)在汉语语篇中插入非词间空格和字间空格会阻碍朗读.