Exploring gender gap and school differential effects in mathematics in Chilean primary schools

ABSTRACT

Despite the systematic recognition of large gender gaps in favour of boys in mathematics performance in Chile, the role of schools in explaining this gap has been comparatively underresearched. This paper analyses a longitudinal dataset to explore within- and between-school variation of Chilean primary schools in promoting attainment and progress of girls and boys. In order to reflect the hierarchical nature of the educational data, multilevel modelling is used, with 163,044 students, nested within 3,355 schools, within 310 municipalities to fit raw, contextualised attainment, and value-added models. The weak evidence indicative of gender differential school effect in progress in Chilean primary schools was interpreted as non-substantive. However, differences in effectiveness played a significant role, as girls progressed more than boys in less effective schools, but the opposite was true in more effective schools. Finally, the study concludes that the Chilean gender gap needs to be addressed mainly beyond schools.     

The ethics of competition: accountability policy enactment in Chilean schools’ everyday life

ABSTRACT

Public policies have a moral order, an ethical horizon. They offer a vocabulary of imagined micro-policies. Using the case of Chile, this paper examines the ways in which accountability policies are reworked within schools and how they affect actors’ subjectivities. It adds new findings to the existing body of research on school accountability policies, offering in-depth evidence based on the case of Chile, which has a high-stakes testing model and a widespread competitive voucher system. The research is based on case studies of ten public and private subsidised schools, framed by a sociological perspective of policy enactment theory. The research findings show the ways that accountability policies are recreated, expanded, and intensified at the local level, permeating an ethic of competition. The analysis focuses on three qualitative trends: school actors’ sense-making of test scores and labels; zones of safety and risk for teachers under an accountability regime; and the emergence of a sticky web of persuasion, surveillance, and coercion among school members in order to improve performance. The practices examined are not understood as ‘secondary effects’ or an ‘implementation problem’, as if they occur unconnected from the policy rationale. The outcomes are consistent with the policy itself in interaction with school life.     

Beyond traditional school value-added models: a multilevel analysis of complex school effects in Chile

School value-added studies have largely demonstrated the effects of socioeconomic and demographic characteristics of the schools and the pupils on performance in standardised tests. Traditionally, these studies have assessed the variation coming only from the schools and the pupils. However, recent studies have shown that the analysis of academic performance could significantly benefit from additional complexity in the model structure, incorporating non-hierarchical and unexplored levels of variation. Using data on secondary students from the Chilean National Pupil Database (2004–2006), this study shows how the traditional value-added models fall short in addressing the complex phenomenon of academic performance, because they largely overestimate school effects. A 4-level contextualised value-added model for progress in Mathematics was implemented and shown to avoid the masking of classroom and locality effects found in the traditional models. We also analyse the effects of important structural factors in Chile such as family income and school type.     

The importance of adjusting for pupil background in school value-added models: A study of Progress 8 and school accountability in England

In the UK, USA and elsewhere, school accountability systems increasingly compare schools using value-added measures of school performance derived from pupil scores in high-stakes standardised tests. Rather than naïvely comparing school average scores, which largely reflect school intake differences in prior attainment, these measures attempt to compare the average progress or improvement pupils make during a year or phase of schooling. Schools, however, also differ in terms of their pupil demographic and socioeconomic characteristics and these factors also predict why some schools subsequently score higher than others. Many therefore argue that value-added measures unadjusted for pupil background are biased in favour of schools with more ‘educationally advantaged’ intakes. But others worry that adjusting for pupil background entrenches socioeconomic inequities and excuses low-performing schools. In this article we explore these theoretical arguments and their practical importance in the context of the ‘Progress 8’ secondary school accountability system in England, which has chosen to ignore pupil background. We reveal how the reported low or high performance of many schools changes dramatically once adjustments are made for pupil background, and these changes also affect the reported differential performances of regions and of different school types. We conclude that accountability systems which choose to ignore pupil background are likely to reward and punish the wrong schools and this will likely have detrimental effects on pupil learning. These findings, especially when coupled with more general concerns surrounding high-stakes testing and school value-added models, raise serious doubts about their use in school accountability systems.     

The instrumentation of test-based accountability in the autonomous dutch system

ABSTRACT

Test-based accountability or ‘TBA,’ as a core element of the pervasive Global Education Reform Movement (GERM), has become a central characteristic of education systems around the world. TBA often comes in conjunction with greater school autonomy, enabling governments to assess ‘school quality’ (i.e. test results) from a distance. Often, quality improvement is further encouraged through the publication of these results. Research has investigated this phenomenon and its effects, much of it focusing on Anglo-Saxon cases. This paper, drawing on expert interviews and key policy documents, couples a policy borrowing with a policy instruments approach to critically examine how and why TBA has developed in the highly autonomous Dutch system. It finds that TBA evolved incrementally, advancing towards higher stakes for schools and boards. Further, it argues that school autonomy has been central to the development of TBA in two ways. Firstly, following a period of decentralisation that increased school(board) autonomy, the Dutch government saw a need to strengthen accountability to ensure education quality. This was influenced by international discourse and accelerated by a (politically exploited) national ‘quality crisis’ in education. Secondly, the traditionally autonomous Dutch system, shaped by ‘Freedom of Education’, has at times conflicted with TBA, and has played a significant role in (re)shaping global policy and in mitigating the GERM.     

Developing a Framework and Agenda for Students' Voices in the School System across Europe: from diametric to concentric relational spaces for early school leaving prevention

There are significant lacunae in the otherwise highly progressive documents on early school leaving prevention from the European Commission and Council in 2011, as part of the EU2020 headline target of 10% early school leavers across the EU. These documents offer no explicit account or analysis of the voices of children and young people, nor vision of systemic accountability of schools to students’ needs. This article argues that key problems of early school leaving can be interpreted as system level blockages in communication, including blockage in communication of children's voices. It seeks to develop a conceptual framework for understanding relational systems by reinterpreting a particular dimension of the structuralist anthropology of Lévi-Strauss’ cross-cultural examination of systems of relation, namely, the interplay between diametric oppositional and concentric relational spaces. The theoretical framework proposed in this article will also seek to translate structural features of system change into structural indicators for system scrutiny and accountability. Identification of key structural indicators is to facilitate change beyond blocked, diametric school space and towards concentric relational spaces in the school system — based on diverse accounts of students’ voices and needs in specific European contexts, as part of a potential strategy at European level to prevent early school leaving. Emerging issues highlighted as an agenda for reform include authoritarian teaching, alternatives to suspension, splits in communication, emotional supports, teacher conflict resolution skills and substantive structures and processes for active student voices in school.     

The impact of student composition on schools’ value-added performance: a comparison of seven empirical studies

In many countries, policy makers struggle with the development of value-added indicators of school performance for educational accountability purposes and in particular with the choice whether school context measured in the form of student composition variables should be included. This study investigates differences between 7 empirical studies that have looked at changes in value-added indicators through the inclusion of school composition variables. Differences were found in the strength of the association between Type A (student-level covariates) and Type B (student- and school-level covariates) school effect models in various regions, varying between 0.74 and 0.99 for secondary education. Important differences were found between the estimate of a school’s effectiveness by using a Type A or a Type B value-added model in secondary education in The Netherlands, Belgium, Chile, and Australia and in US and UK primary education. However, few differences were found in other cases.     

Tackling inequality? Teacher effects and the socioeconomic gap in educational achievement. Evidence from Chile

Although teacher quality is usually signalled to be the most relevant school-level factor impacting students’ learning, little is known about the relevance of teacher effects explaining educational inequality. By using a value-added model for a cohort of 10th graders, in this work we examine the socioeconomic gap in teacher effects across Chilean secondary schools, and its importance in explaining socioeconomic inequality in students’ achievement in math and language. We found an important proportion of highly effective teachers in low-socioeconomic-status (SES) schools, but also much bigger variation in teacher effects across those schools. Variability in teacher effects decreases when moving towards higher SES schools, where there is also a smaller proportion of low-performing teachers. All in all, teacher effects have a levelling impact for students in low-SES schools when compared to those in middle SES schools, but no significant impact when compared to students in high-SES schools.     

Reassessing school effectiveness: Multi-objective value-added measures (MOVAM) of academic and socioemotional learning

Under the Every Student Succeeds Act (ESSA), it is imperative that American public schools assess and improve not only academic but also non-academic measures of student learning such as socioemotional skills. The policy shift towards broad-based school accountability calls for reassessing school effectiveness from whole child development perspectives and addressing potential biases and limitations of conventional value-added measures (VAM). Through multivariate multilevel analyses of the Early Childhood Longitudinal Study-Kintergarten (ECLS-K): 2011 data, this study applies multi-objective value-added measures (MOVAM) approach to assess and improve school effectiveness for academic and socioemotional learning. The study results show different patterns of academic vs. socioemotional learning gains, and also weak correlations between school effects on the two types of learning outcomes. Nevertheless, the comparisons of academically and socioemotionally effective vs. ineffective schools imply that schools can and should improve both academic and socioemotional learning outcomes through synergistic improvement of key organizational and instructional conditions.     

Conceptual and empirical differences among various value-added models for accountability

Accountability systems in education generally include indicators of student performance. However, these indicators often differ considerably among the various systems. More and more countries try to include value-added measures, mainly because they do not want to hold schools accountable for differences in their initial intake of students. This study presents a conceptual framework of these value-added measures, resulting in an overview of 5 different types. Using data from Dutch secondary schools, we empirically provide estimates of these different measures. Our analyses show that the correlation between the different types of school effects estimated is rather high, but that the different models implicate different results for individual schools. Based on theoretical considerations, arguments are given to use the following indicators in the value-added accountability models: prior achievement, student-level background characteristics, and compositional characteristics of the student population.     

School autonomy,accountability and collaboration: a critical review

English education has recently experienced radical policy reform in the areas of school autonomy and accountability. The key focus of this paper is on how schools might best navigate through these policy moves. It highlights how these moves have constructed schools, teachers and students in problematic ways but also how they are offering possibilities for improving the quality of schools and schooling. The focus here is on the promise and scope of school collaboration. The difficulties of creating socially responsive and responsible collaboratives in the current ‘heterarchical’ and market-oriented policy environment are acknowledged. Guided by quality democratic governance, they are, nonetheless, presented as crucial in supporting schools to productively deal with the demands of this environment.     

The use of a school value-added model for educational improvement: a case study from the Portuguese primary education system

This article focuses on the use of a value-added approach for promoting school improvement. It presents yearly value-added estimates, analyses their stability over time, and discusses the contribution of this methodological approach for promoting school improvement programmes in the Portuguese system of evaluation. The value-added model is applied to longitudinal data gathered from a representative sample of students enrolled in primary school, using mathematics scores as outcome variable. The estimates are used to determine a systematic positive or negative school value-added effect. Results indicate that at least 41% of schools remain at the same level over 3 years, 15% present an instable pattern, and approximately 40% show a positive or negative trend. It is argued that these results, especially those that reveal systematically underperforming schools, could have relevant implications for educational policy geared towards school improvement.     

农村初级中学学校效能的增值性评价——基于我国西部五省调研数据的实证分析

随着学校效能研究的深入和统计技术的进步,学校效能增值评价日益成为学校效能评价的主流。以学生学业成绩和学生适应性为学校效能评价的指标,运用两水平线性模型对我国西部地区85所农村初中进行增值评价,发现:在提高学校效能方面,学校间存在明显差异;按原始分数获得的学校排名与按增值评价获得的排名存在较大差异;不同学科间的学校效能显著相关;学校效能在不同省份间存在较大差异;学生学习基础仍然是造成学生成绩差距的主要原因;学校基本条件对学生学业成绩作用明显等。据此,提出有利于提高学校效能的建议,如以增值评价为手段,建立增值评价指标体系;合理制定学校发展规划,走学校特色发展之路;教师应针对不同起点学生因材施教等。     

Diagnosing the learning crisis: What can value-added analysis contribute?

Advocates of teacher value-added modelling (VAM) argue that this technique can provide evidence on teacher effectiveness to inform teacher policies and broader education system reforms. Critics contend that value-added is a poor proxy for teacher quality and as such is of questionable utility, especially where teacher accountability is concerned. In low- and middle-income countries, and especially sub-Saharan Africa, where the challenge of the ‘learning crisis’ is most severe, a lack of longitudinal data has precluded extensive debate on the matter. In this paper we explore the potential of value-added analysis for diagnostic purposes in the context of Ethiopia. We make use of data from the Young Lives longitudinal study – specifically two rounds of school surveys conducted in Ethiopia between 2012 and 2017 when pupils were in grades 4–8. Learning levels in the Young Lives sites in Ethiopia are very considerably below curricular expectations. Like many countries in sub-Saharan Africa, Ethiopia faces a significant challenge in terms of a ‘learning crisis’ and in terms of the attendant need to develop policies to improve educational effectiveness within the confines of very limited resources. We discuss the background to VAM models and their use, including in relation to the context of Ethiopia. The paper shows that learning progress in primary schools varies widely between classrooms, and between pupils within the same classroom. Some schools and teachers are more successful in raising overall attainment by ‘raising the floor’ of learning and narrowing the dispersion. Others are more successful by ‘raising the roof’. Less effective teachers appear to be particularly ineffective for pupils with higher scores at the start of the year. In contrast, the most effective teachers showed high levels of ‘value-added’ for pupils at all levels of prior performance. Diagnostic analysis of teacher value-added has potential, we argue, to aid understanding of contributors to low levels of learning such as: (i) over-ambitious curricula; (ii) absence of ‘teaching at the right level’; (iii) within class heterogeneity and pupil grouping strategies; and (iv) teaching and learning strategies – such as ‘differentiation’ or ‘mastery’.     

Diversity and progress in school accountability systems in Australia

School education in Australia is a complex interplay between federal and state governments, and between government and non-government schools. This article explores the supervision of schools in Australia through school accountability systems. Utilising publicly available documents a systematic analysis of the state and territory systems for government schools is provided. It is a paper that attempts to document rather than critique school accountability, although a conceptual framework utilising contractual, moral and professional accountability is used to analyse the different accountability processes reported upon. Contractual and moral accountability is supported by most systems, whilst there is potential to foster professional accountability in only two systems. Fostering professional accountability is important because this is where the internal motivation of teachers helps to drive school improvement. When compared to leading-edge systems, Australian accountability systems are lacking in judgements on teaching practice in individual classrooms, and the use of sophisticated measures of learning and value-added analysis.     

School-to-school support within a competitive education system: views from the inside

ABSTRACT

This paper draws on evidence from a study carried out in England to explore how schools can support one another’s improvement within a policy context that emphasises competition. The findings offer some reasons to be optimistic, and are suggestive of the capacity and potential of the school system in England to “self-improve” through collaborative means. However, light is also thrown on a number of barriers that need to be overcome to make such an approach work. The paper argues that developing a greater understanding of the social complexities involved in school-to-school support requires research that takes account of the views of those involved. With this in mind, the paper reflects on the experiences of a group of school leaders in England, leading to lessons that are likely to be relevant to those in other national contexts where competition is seen as a driver for school improvement.     

The performative shift: middle leadership ‘in the line of fire’

ABSTRACT

State school inspection in Norway is currently changing with targeted schools becoming subject to more complex methods of inquiry. Not only school principals but middle leaders are exposed to this shifting system, for the latter are in the frontline of their schools’ everyday practices. The article examines how state school inspection is used as means of controlling legal compliance, as well as evaluating the formative assessment routines and practices of schools, middle leaders, and individual teachers. Drawing on the concepts of accountability and performativity, field observations of inspectors interrogating department heads in primary education are analysed. The empirical study demonstrates how use of standardised rubrics steer the inspection process in schools, aiming more towards completing on task, rather than supporting middle leaders in their struggle to comply with legal standards. During such interrogation, the department heads comply with the system, and are at the same time open towards the inspectors’ questioning concerning the school’s lack of fully implemented routines.     

Charter School Accountability

ABSTRACT

Stakeholder surveys conducted as part of the development of an accountability and assessment system for five charter schools in Miami-Dade County and Broward County, Florida, revealed high positive response regarding high expectations, school climate, basic skills instruction, and monitoring student progress. The lowest overall rating revealed dissatisfaction with charter school resources. Five researchers distributed questionnaires to stakeholders, defined as parents, pupils, teachers, administrators, special program teachers, and auxiliary personnel. Survey results were generally positive in assessing the schools, programs, teachers, administrators, and relationships between the various stakeholder groups. This study provided the quantitative data needed to form the framework for the development and implementation of an accountability system.     

Legal Regulation and Professional Discretion in Schools

ABSTRACT

The paper aims to analyse how pupils’ equal educational opportunities are warranted. We focus on how regulations of adapted education and the right to special needs education provide school leaders’ and teachers’ room for discretionary decision-making, how it is interpreted and how discretionary power is justified. The paper draws on findings from an interdisciplinary study (education and law) on the transformation of legal standards into professional actions in schools. This paper employs interview data from three compulsory schools (grades 1–10), legal documents, and public reports, and the analysis is guided by a conceptual distinction between structural and epistemic aspects of discretion. Based on our findings, we question whether the discretionary space regarding special needs education gives too many opportunities for action and, hence, weakens pupils’ legal rights. Under the guise of a common school, there is a risk that pupils’ special needs are made invisible.     

Curriculum policy reform in an era of technical accountability: ‘fixing’ curriculum,teachers and students in English schools

Abstract

Drawing on a Levinasian ethical perspective, the argument driving this paper is that the technical accountability movement currently dominating the educational system in England is less than adequate because it overlooks educators’ responsibility for ethical relations in responding to difference in respect of the other. Curriculum policy makes a significant contribution to the technical accountability culture through complicity in performativity, high-stakes testing and datafication, at the same time as constituting student and teacher subjectivities. I present two different conceptualizations of subjectivity and education, before engaging these in the analysis of data arising from an empirical study which investigated teachers’ and stakeholders’ experiences of curriculum policy reform in ‘disadvantaged’ English schools. The study’s findings demonstrate how a prescribed programme of technical curriculum regulation attempts to ‘fix’ or mend educational problems by ‘fixing’ or prescribing educational solutions. This not only denies ethical professional relations between students, teachers and parents, but also deflects responsibility for educational success from government to teachers and hastens the move from public to private educational provision. Complying with prescribed curriculum policy requirements shifts attention from broad philosophical and ethical questions about educational purpose as well as conferring a violence by assuming control over student and teacher subjectivities.