Value‐added is of little value

Published indicators of school ‘performance’, such as those shown annually in league tables in England, have been controversial since their inception. Raw‐score figures for school outcomes are heavily dependent on the prior attainment and family background of the students. Policy‐makers in Wales have reacted to this fundamental flaw by withdrawing the publication of school results. In England, on the other hand, they have reacted by asking for more information to be added to tables, in the form of student context such as the percentage with a special educational need, and ‘value‐added’ figures. In 2004, the Department for Education and Skills (DfES) value‐added figures for England were based on student progress from Key Stage 2 at the end of primary education to GCSE at the end of compulsory secondary education. For 2005, at time of writing, the DfES plan to use context information in their model as well. This paper re‐analyses the 2004 value‐added figures and shows that they contain the same flaw as the original raw‐score tables. The purported value‐added scores turn out to be a proxy for the overall level of attainment in the school, and almost entirely independent of any differential progress made by the students. The paper concludes by considering the implications of these findings, if accepted, for policies based on identifying schools that are clearly more or less effective, and for the field of school effectiveness and improvement research.     

Testing the validity of value‐added measures of educational progress with genetic data

Value‐added measures of educational progress have been used by education researchers and policy‐makers to assess the performance of teachers and schools, contributing to performance‐related pay and position in school league tables. They are designed to control for all underlying differences between pupils and should therefore provide unbiased measures of school and teacher influence on pupil progress, however, their effectiveness has been questioned. We exploit genetic data from a UK birth cohort to investigate how successfully value‐added measures control for genetic differences between pupils. We use raw value‐added, contextual value‐added (which additionally controls for background characteristics) and teacher‐reported value‐added measures built from data at ages 11, 14 and 16. Sample sizes for analyses range from 4,600 to 6,518. Our findings demonstrate that genetic differences between pupils explain little variation in raw value‐added measures but explain up to 20% of the variation in contextual value‐added measures (95% CI = 6.06% to 35.71%). Value‐added measures built from teacher‐rated ability have a greater proportion of variance explained by genetic differences between pupils, with 36.3% of their cross‐sectional variation being statistically accounted for by genetics (95% CI = 22.8% to 49.8%). By contrast, a far greater proportion of variance is explained by genetic differences for raw test scores at each age of at least 47.3% (95% CI: 35.9 to 58.7). These findings provide evidence that value‐added measures of educational progress can be influenced by genetic differences between pupils, and therefore may provide a biased measure of school and teacher performance. We include a glossary of genetic terms for educational researchers interested in the use of genetic data in educational research.     

Teaching science process skills: long‐term effects on science achievement

Abstract

In this follow‐up study to Brotherton and Preece (1996), the long‐term effects of teaching science with a special emphasis on process skills with Year 7, 8, and 9 classes were investigated. In a value‐added analysis, a positive effect of the intervention on Year 8 males was found on examining subsequent GCSE science results. (The GCSE‐‐General Certificate of Secondary Education‐‐is a public examination taken by pupils in England and Wales at the end of Year 11.) This finding was interpreted in terms of the particular ‘readiness’ of this group to enculturation into practices involved in teaching and assessment at the GCSE level.     

Public Accountability and the Academic Effectiveness of Grant‐Aided Catholic Schools

This paper is offered as a contribution to a debate about the public accountability of church‐affiliated schools grant‐aided by the state. The paper provides an account of two parallel secondary school systems in Northern Ireland, popularly described as ‘Catholic’ and ‘Protestant’, in effectiveness terms. The outcome measure is academic performance in public examinations taken by pupils at the end of the period of statutory schooling. The paper extends earlier work of the author by focusing on Catholic schools in the light of access to a second and more up‐to‐date data set and to recent Scottish findings. Having indicated ‘raw’ performance differences between the two sectors, a value‐added approach to performance measurement is presented. Doubt is cast on aspects of the received wisdom concerning the superior academic performance of Protestant‐educated pupils.     

The evolution of school league tables in England 1992–2016: ‘Contextual value‐added’, ‘expected progress’ and ‘progress 8’

Since 1992, the UK Government has published so‐called ‘school league tables’ summarising the average General Certificate of Secondary Education (GCSE) ‘attainment’ and ‘progress’ made by pupils in each state‐funded secondary school in England. While the headline measure of school attainment has remained the percentage of pupils achieving five or more good GCSEs, the headline measure of school progress has changed from ‘value‐added’ (2002–2005) to ‘contextual value‐added’ (2006–2010) to ‘expected progress’ (2011–2015) to ‘progress 8’ (2016–). This paper charts this evolution with a critical eye. First, we describe the headline measures of school progress. Second, we question the Government's justifications for scrapping contextual value‐added. Third, we argue that the current expected progress measure suffers from fundamental design flaws. Fourth, we examine the stability of school rankings across contextual value‐added and expected progress. Fifth, we discuss the extent to which progress 8 will address the weaknesses of expected progress. We conclude that all these progress measures and school league tables more generally should be viewed with far more scepticism and interpreted far more cautiously than they have often been to date.     

Analysis of Added Value of Subscores With Respect to Classification

Brennan noted that users of test scores often want (indeed, demand) that subscores be reported, along with total test scores, for diagnostic purposes. Haberman suggested a method based on classical test theory (CTT) to determine if subscores have added value over the total score. One way to interpret the method is that a subscore has added value only if it has a better agreement than the total score with the corresponding subscore on a parallel form. The focus of this article is on classification of the examinees into “pass” and “fail” (or master and nonmaster) categories based on subscores. A new CTT‐based method is suggested to assess whether classification based on a subscore is in better agreement, than classification based on the total score, with classification based on the corresponding subscore on a parallel form. The method can be considered as an assessment of the added value of subscores with respect to classification. The suggested method is applied to data from several operational tests. The added value of subscores with respect to classification is found to be very similar, except at extreme cutscores, to their added value from a value‐added analysis of Haberman.     

Modelling patterns of improvement over time: value added trends in English secondary school performance across ten cohorts

This paper looks at underlying patterns of school effectiveness through analysing a GCSE examination data‐set over a period of ten cohorts (1993–2002) in one very large English school district. Both value added and raw score approaches were explored by employing different statistical multilevel models to examine time trends of school and pupil performance from two perspectives: consistent (linear) and inconsistent (non‐linear) school improvement. Overall, levels of measured attainment for the vast majority of the schools increased over the decade and the results indicate that one in four schools had significantly higher value added improvement trajectories (linear) than would be expected over the decade—in comparison to the average school. Those schools with a lower value added starting point in 1993 were more likely to make significant improvement. However, underlying these linear improvement trajectories it appears that only one in 16 schools managed to improve continuously for more than four years at some point over the decade in terms of value added.     

Value added based on educational positions in Dutch secondary education

Estimating added value as an indicator of school effectiveness in the context of educational accountability often occurs using test or examination scores of students. This study investigates the possibilities for using scores of educational positions as an alternative indicator. A number of advantages of a value added indicator based on educational positions of students can be formulated, such as: (a) the societal significance of educational position as output measure; (b) the fact that a single indicator can be estimated for an entire school in a differentiated educational system, where not all schools provide education in all tracks; and (c) the expectation that value added based on educational positions leads to other incentives for schools than value added based on test scores. Empirical analysis of Dutch cohort data (VOCL'99) for secondary education showed considerable differences in effectiveness between schools in the positions of students. Furthermore, differential school effects were found for both socio‐economic status and prior achievement. The phenomena of differential school effects for socio‐economic status and prior achievement are linked to differences between schools in the kind of tracks in which the schools provide schooling.     

The attainment and progress of twins at the start of school

This paper considers cognitive differences between twins and other higher multiple births when they start school and their progress in reading and maths during the first year at school. The data came from the Performance Indicators in Primary Schools (PIPS) project which has developed an on‐entry assessment designed to provide a solid base against which the relative progress (value added) of pupils can be measured. Some of the findings were surprising, in that they appeared to be in conflict with some earlier work, but there was also some agreement. Additionally, teachers were asked to assess pupils on the Attention Deficit Hyperactivity Disorder (ADHD) scale at the end of their first year in school, and this information was used to check out an earlier rinding that twins were more prone to score highly on this scale.     

A comparative study of peer and teacher feedback and of various peer feedback forms in a secondary school writing curriculum

This study examines whether peer feedback can be a substitute for teacher feedback and which measures can be taken to improve its effectiveness. A pre‐test post‐test control group design examined the long‐term learning effects of individual peer feedback and of collective teacher feedback on writing assignments in secondary education. Moreover, it examined the added value of a priori question forms and a posteriori reply forms aimed at supporting the assessee's response to peer feedback. The study supports the ‘non‐inferiority’ hypothesis of there being no significant difference in students’ progress after plain substitutional peer feedback or teacher feedback. Both groups (plain peer feedback and teacher feedback), however, improved significantly less than the groups that worked with question or reply forms, confirming the added‐value of these forms. Almost half of the students found the received peer feedback helpful, but less than a quarter considered giving feedback an aid in their own learning process.     

A New Interpretation of Augmented Subscores and Their Added Value in Terms of Parallel Forms

The value‐added method of Haberman is arguably one of the most popular methods to evaluate the quality of subscores. The method is based on the classical test theory and deems a subscore to be of added value if the subscore predicts the corresponding true subscore better than does the total score. Sinharay provided an interpretation of the added value of subscores in terms of scores and subscores on parallel forms. This article extends the results of Sinharay and considers the prediction of a subscore on a parallel form from both the subscore and the total raw score on the original form. The resulting predictor essentially becomes the augmented subscore suggested by Haberman. The proportional reduction in mean squared error of the resulting predictor is interpreted as a squared multiple correlation coefficient. The practical usefulness of the derived results is demonstrated using an operational data set.     

Missing link: Integrated individual leadership development,employee engagement,and customer value‐added improvement

Long‐term care is a key public issue that affects all of us in some way at some time of our lives. Nowhere is performance improvement and quality management more imperative. Through an 8‐month field study and follow‐up case study, we discuss how using an integrated approach to individual leadership development, employee engagement, and customer value‐added improvement might provide the missing link toward sustaining and growing organizational culture change in a very demanding environment.     

Competency, capability, complexity and computers: exploring a new model for conceptualising end-user computer education

Notions of competency have dominated the computer education literature, and have underpinned Competency‐Based Training (CBT) in information technology at all levels of education and training. The emergence of counter‐narratives underpinned by the capability movement, have as yet had minimal impact on practice in computer education. New discourses in educational theory and practice which are founded on non‐linear approaches to learning and teaching provide added impetus to engage in the competency/capability debate, and re‐examine our approaches to computer education. This paper explores complexity theories and demonstrates how complexity's pedagogical implications can lead to new models for understanding computer learning and teaching. A new model for conceptualising end‐user computer education is presented that was derived from a three‐year action research initiative with pre‐service teachers.     

A Note on Assessing the Added Value of Subscores

Brennan ( 2012 ) noted that users of test scores often want (indeed, demand) that subscores be reported, along with total test scores, for diagnostic purposes. Haberman ( 2008 ) suggested a method based on classical test theory (CTT) to determine if subscores have added value over the total score. According to this method, a subscore has added value if the corresponding true subscore is predicted better by the subscore than by the total score. In this note, parallel‐forms scores are considered. It is proved that another way to interpret the method of Haberman is that a subscore has added value if it is in better agreement than the total score with the corresponding subscore on a parallel form. The suggested interpretation promises to make the method of Haberman more accessible because several practitioners find the concept of parallel forms more acceptable or easier to understand than that of a true score. Results are shown for data from two operational tests.     

我国加工贸易的现存问题与对策思考

改革开放以来,我国加工贸易取得了巨大成绩。但是,我国加工贸易仍然存在一些问题：加工贸易产业处于国际产业增值链条低端环节,加工贸易中高耗能和高污染产业所占比重较大,加工贸易中外资企业占据了主导的地位和分额。推进我国加工贸易持续健康发展的对策措施是：抓住国际产业转移机遇,正确处理好三个方面关系。不断完善监管制度机制。     

Learning to Performance at Larson‐Juhl: A Success Story

Often companies diagnose a performance gap, selecting training as the solution. Isolated training events that do not take a complete training approach produce less results, including value to the business, than a more comprehensive and integrated approach. What follows is a success story describing how Larson‐Juhl used a learning to performance approach to combine learning, leadership, and change management competencies to produce documented and sustainable results that added value.     

Running to stay still in the knowledge economy

To maintain their global positioning, some of the world's most prominent institutions are pursuing strategic transnational alliances. In this paper I examine one such transnational alliance – that between the Massachusetts Institute of Technology and the government of Singapore. Using governmentality as a framework of analysis, the paper locates the Singapore‐MIT Alliance within the broader policy architecture that underpins Singapore's knowledge economy aspirations. The Alliance demonstrates some of the practical complexities involved in ‘leap‐frogging’ into the ‘value‐added’ realms of knowledge and service‐related production. It highlights the resistances, tensions and contradictions arising from leveraging off foreign expertise to build an education hub. The paper concludes with a discussion of the changing regimes of value arising from aspiring knowledge economies.     

Building an evaluation framework for a competency‐based graduate program at the University of Southern Mississippi

This article describes an ongoing project to build an evaluation framework for a competency‐based graduate program at The University of Southern Mississippi. Many traditional methods of evaluating performance at academic institutions provide only a partial assessment of individual program performance. In an increasingly competitive global economy, program evaluations should also consider the perspectives of students, graduates, and employers in order to develop curricula that will address the critical skills sets needed for strategic and value‐added performance improvement work in the 21st century.     

A value‐added study of the impact of science curriculum reform using the National Pupil Database

This article compares the examination performance of students following General Certificate of Secondary Education Applied Science and Double Award science on a value‐added basis. It uses data from the National Pupil Database, which are analysed by a variety of methods, including multilevel modelling. It argues that the claims made within Office for Standards in Education reports on the impact of the new Applied Science course are inaccurate, and in some cases positively wrong. The authors' analysis, based on the second (2003–05) cohort of students following the course, shows that students of lower attainment at Key Stage 3 appear to perform better than would have been predicted from their Key Stage 3 attainment, but that higher attaining pupils perform less well. Schools offering Applied Science show greater diversity in the value‐added effects they produce than they do with Double Award science. The article argues that analyses of this kind are of value in seeking to understand the impact of curriculum reform, but that they are dangerous as a generalized guide for policy. They must be complemented by work in schools in which the sources of the effects observed and of school diversity are examined.