Test Score or Student Progress? A Value-Added Evaluation of School Effectiveness in Urban China

Outcome-oriented evaluation of school effectiveness is often based on student test scores in certain critical examinations. This study provides another method of evaluation—value-added—which is based on student achievement progress. This paper introduces the method of estimating the value-added score of schools in multi-level models. Based on longitudinal student achievement data, two measures of school effectiveness in one local education authority in China are compared. It is found that the between-school difference in both test-score and value-added is large comparable with that of Western countries. The results of the two measures of school effectiveness are highly different. The value-added measures lack consistency across different subject areas within schools while the test score measures are highly correlated between subjects. Teachers show their preference for value-added measures over test-score measures of education quality. It is suggested that value-added measures of school effectiveness should be used as a complement to rather than a substitute for test-score measures. The shortcomings of value-added approach are also discussed.     

Complexities in the Judgement of School Effectiveness∗

ABSTRACT

This article examines the aims and goals of school effectiveness research and definitions of effectiveness. Five issues in school effectiveness research which are relevant to the development of better methods of judging and measuring relative effectiveness, drawing on evidence from studies in different countries and contexts, are reviewed. These are (1) size and significance of effects; (2) consistency across outcomes; (3) stability over time; (4) the long term effects of schools; and (5) differential effects for different student groups.

It is concluded that effectiveness is best regarded as a relative concept dependent upon time period and age groups studied, and the choice of educational outcomes and intake measures

     

Modeling Educational Effectiveness in Classrooms: The Use of Multi-Level Structural Equations to Model Students’ Progress

Long-standing and enduring problems in quantitative studies of educational effectiveness relate to fitting models that adequately reflect the complex inter-relationships among multivariate, multilevel factors affecting students’ educational progress, particularly among those that operate within classrooms. This article illustrates one approach to solving such difficulties by combining the analytic approaches of multilevel analysis and structural equation modeling in a two-stage process. The data used are drawn from a longitudinal study of teacher and school effectiveness for three Grade-level cohorts of 4,558 students clustered within 334 class/teacher groups in 52 elementary schools. The article provides estimates of inter-relationships among achievement and social background factors including ‘critical events’ on students’ progress in mathematics, together with their affective schooling experiences and inattentive behaviors in the classroom. Findings are presented indicating the extent to which progress can be accounted for by the grouping effects of students within classes and schools, highlighting the importance of accounting for the interdependent effects among variables. Both the substantive and methodological implications of the findings are discussed. Résumé De longue date et de manière persistante, les problèmes posés dans les études quantitatives sur l’efficacité de l’enseignement tiennent à la capacité d’élaborer des modèles qui reflètent de manière adéquate la complexité des interrelations parmi les multiples variables, avec leurs multiples niveaux, qui interviennent dans la progression des étudiants, particulièrement en ce qui concerne celles qui opèrent à l’intérieur même des classes dans les écoles. Cet article présente une approche cherchant à résoudre de telles difficultés en combinant les techniques de l’analyse multivariée avec un modèle d'équation structurale impliquant un processus à deux niveaux. Les données utilisées sont tirées d’une étude longitudinale sur l’efficience des écoles et des enseignants portant sur des cohortes de trois degrés scolaires touchant 4558 élèves provenant de 334 groupes classe/enseignant dans 52 écoles élémentaires. L’article fournit des estimations concernant les interrelations entre la réussite scolaire et des facteurs relevant de l’environnement social incluant des “événements critiques” dans la progression des élèves en mathématiques, mis en rapport avec leur expérience scolaire affective et des comportements inattentifs en classe. Les conclusions présentées indiquent la mesure dans laquelle le progrès peut être expliqué par des effets de groupes d’élèves à l’intérieur des classes et des écoles. Elles éclairent l’importance de la prise en compte de l’interdépendance des éléments parmi les variables. La justification et les implications méthodologiques des conclusions sont discutées. (Traduction: Raymond Hutin, Dr ès sciences de l’éducation, Dardagny / Suisse)     

Methods in School Effectiveness Research∗

This paper discusses the methodological requirements for valid inferences from school effectiveness research studies. The requirements include long term longitudinal data and proper statistical modelling of hierarchical data structures. The paper outlines the appropriate multilevel statistical models and shows how these can model the complexities of school, class and student level data.

     

Changes in the Impact of Family Education on Student Educational Achievement in Sweden 1988–2014

The main aim of this study was to investigate the development of the correlation between family education and student achievement in Sweden, which previous research has found to be stable, in spite of increasing school segregation and widening differences in levels of achievement between schools. Based on register data for populations of graduates from compulsory school between 1988 and 2014, correlations between parental education and student grades were estimated. The correlation was found to increase by .04 units between the early-1990s and 2014. The main reasons why this has not been found in previous research are that too coarse a categorization of parental education has been used, a declining quality of measurement of grades, and demographic changes related to immigration.     

Tensions in Teacher Training for School Effectiveness: The Case of Pakistan∗

While research on school effectiveness is growing internationally, its impact on the training of teachers is not clear. Taking Pakistan as a case study, this paper reports an investigation of how far trainees, tutors and practising teachers felt that training matched the researched ‘factors’ linked to successful schools. Fewer than half the practising teachers felt well prepared; but there was not always agreement within or between the three groups about what elements of ‘effectiveness’ should be included, except that the training should be more ‘practical’. The hidden curriculum of the training institution was found to be based on authoritarian transmission methods and on rote learning of traditional syllabi, so that trainees were neither exposed to newer ideas about effective teaching, nor experienced for themselves the self‐direction and critical analysis essential for future organisational development. The culture and conditions surrounding teaching also militated against the notion of ‘the reflective practitioner’.     

New Destinations,New Trajectories? The Educational Progress of Hispanic Youth in North Carolina

Since 1990, Latin American immigrants to the United States have dispersed beyond traditional gateway regions to a number of "new destinations." Both theory and past empirical evidence provide mixed guidance as to whether the children of these immigrants are adversely affected by residing in a nontraditional destination. This study uses administrative public school data to study over 2,800 8- to 18-year-old Hispanic youth in one new destination, North Carolina. Conditional on third-grade socioeconomic indicators, Hispanic youth who arrive by age 9 and remain enrolled in North Carolina public schools close achievement gaps with socioeconomically similar White students by sixth grade and exhibit significantly lower high school dropout rates. Their performance resembles that of first-generation youth in more established immigration gateways.     

Understanding Differences in Academic Effectiveness: Practitioners’ Views∗

As part of a large‐scale study of the ability of 94 secondary schools to promote academic achievement, case studies of six outlier schools were conducted and headteachers (principals) and their deputies were questioned about the processes of effectiveness. The analysis of their responses ‐ grouped by the performance of their school (measured by value added analyses of three years of public examination results) as effective, ineffective or mixed ‐ reveals considerable support for previously cited characteristics of effective schooling. Despite the limitations imposed by a reliance on the use of retrospective accounts, the study takes forward the field of school effectiveness in its search for generalisable findings and coherent theory.

     

The Micropolitics of Educational Inequality: The Case of Teacher–Student Assignments

Politics of education researchers have long recognized the role of micropolitics in school decision-making processes. We argue that investigating micropolitical dynamics is key to an important set of school decisions that are fundamental to inequities in access to high-quality teachers: assignments of teachers and students to classrooms. Focusing on the intraorganizational political power of experienced teachers, our analysis of survey and administrative data from a large urban district suggests that more experienced teachers have more influence over which students are assigned to their classrooms. By a variety of measures, we also find that more experienced teachers are assigned fewer disadvantaged students, on average, a pattern inconsistent with goals of ameliorating educational inequality by matching more qualified teachers with the students who need them most.     

Developing Preschoolers’ Social Skills: The Effectiveness of Two Educational Methods

This study tested whether and how methods called ‘Play Time/Social Time’ and ‘I Can Problem Solve’ contribute to the improvement of social skills and the development of theory of mind (ToM) in children. The participants in the experiment were nearly 200 (N = 196) preschool children with low social functioning, with and without disabilities. The study showed that social skills and the level of ToM in children attending the lessons improved more than those of children from the control group. Both methods turned out to be effective in the case of social skills and ToM development among children with and without disabilities. It is worthwhile to popularise these methods among preschool teachers.     

Old Educational Ideas,New American Schools: Progressivism and the Rhetoric of Educational Revolution∗

Drawing on books and essays of prominent progressive educators, this essay seeks to identify key elements that should be included in a working definition of progressive education. Such a definition is crucial for determining the degree of influence progressive ideas have had on American schools. The essay looks at how these ideas appear to have influenced the designs of a number of highly visible "whole-school reform" programs in the United States, programs that are perhaps the leading educational reform effort in the United States today. The essay then assesses how effective these programs have been in boosting student achievement. The essay concludes by speculating that the oft-used rhetoric of educational revolution, which was employed by early progressive educators and by later whole-school reformers, is a serious obstacle to genuine improvement in educational policy and practice.     

Are Student Evaluations of Teaching Effectiveness Valid for Measuring Student Learning Outcomes in Business Related Classes? A Neural Network and Bayesian Analyses

In this study we investigate the underlying relational structure between student evaluations of teaching effectiveness (SETEs) and achievement of student learning outcomes in 116 business related courses. Utilizing traditional statistical techniques, a neural network analysis and a Bayesian data reduction and classification algorithm, we find little or no support for the validity of SETEs as a general indicator of teaching effectiveness or student learning. In fact, the underlying structure appears to be non-linear and possibly negatively bimodal where the most effective instructors are within the middle percentiles of student course ratings, while instructors receiving ratings in the top quintile or the bottom quintile are associated with significantly lower levels of student achievement.     

Multilevel Interaction Models and their Use in the Analysis of Large‐scale School Effectiveness Studies

In the present paper we discuss the merits of a general multilevel approach to model analysis of large scale school effectiveness studies.

We briefly review previous research framework which have been used in school effectiveness research and examine critically the assumptions on which they were based and the methodological implications of these assumptions. The general interaction models presented in this paper provide an answer to many of these critiques. Findings from different types of modeling supported the claim that the modeling of school effectiveness studies must be both multilevel and interactive. Achievement is found to be dependent in a very sensitive, non‐additive way, on the particular combination of pupil's home background, his or her general ability, teaching style and other teacher characteristics, the class and the school context in which the pupil learns. Comparisons between parsimonious main effect models with parsimonious interactive ones show that although the explanatory power of interaction models might be slightly smaller, the use of interaction models causes changes in the specification of the models that cannot be ignored.     

Formative Information Using Student Growth Percentiles for the Quantification of English Language Learners’ Progress in Language Acquisition

English language learners (ELLs) are the fastest growing subgroup in American schools. These students, by a provision in the reauthorization of the Elementary and Secondary Education Act, are to be supported in their quest for language proficiency through the creation of systems that more effectively measure ELLs’ progress across years. In the past, ELLs’ progress has been based on students’ prior scores measuring the same construct. To disentangle effectiveness from achievement, the reporting has generally targeted mean-group activity. In contrast, student growth percentiles (SGPs) provide a comparison of students’ growth with others who have the same achievement score history. By examining the construct measured by an English language proficiency test as manifested in student scores in Speaking, Listening, Reading and Writing, this article outlines the use of SGPs in providing information on how much each student needs to grow, which will allow educators to more effectively apply differential formative instructional strategies.     

Some Reflections on and Criticisms of China＇s Educational Management Studies

In spite of having undergone one Century＇s vicissitude, China＇s educational management studies are still lagging behind those abroad. Two research lines, one being induction and generalization, another being deduction and transplantation, have been roughly evolved over these studies. Both of them have reached the level of empirical science. Since 1980s, they have developed in an unprecedented way. However, there are still some problems existing in this field. The first one is tha（a clan of multi-echeloned and multi-typed educational management studies has been multiplied in the way of trying to jump over certain developing phases and to obtain instant success and quick profits. The second one is the concept of what is educational management studies is still unclear, thus leading to ambiguous ideas or even confusions on this issue. The third one is that the study on educational management and the study on educational economy are forced to be tied together and become one by human forces, thus hampering the development of either field. Therefore, it is necessary to make reflections on and criticisms on these confused phenomena so as to preserve standardization and identity of this discipline and promote its healthy development.     

The Potential Effectiveness of Communicative Approach in Xinjiang

Teaching English as a foreign language has been carried out in our Xinjiang     

Measuring Teaching Effectiveness: Correspondence Between Students’ Evaluations of Teaching and Different Measures of Student Learning

Relating students?? evaluations of teaching (SETs) to student learning as an approach to validate SETs has produced inconsistent results. The present study tested the hypothesis that the strength of association of SETs and student learning varies with the criteria used to indicate student learning. A multisection validity approach was employed to investigate the association of SETs and two different criteria of student learning, a multiple-choice test and a practical examination. Participants were N?=?883 medical students, enrolled in k?=?32 sections of the same course. As expected, results showed a strong positive association between SETs and the practical examination but no significant correlation between SETs and multiple-choice test scores. Furthermore, students?? subjective perception of learning significantly correlated with the practical examination score whereas no relation was found for subjective learning and the multiple choice test. It is discussed whether these results might be due to different measures of student learning varying in the degree to which they reflect teaching effectiveness.