The Accuracy of Aggregate Student Growth Percentiles as Indicators of Educator Performance

Mean or median student growth percentiles (MGPs) are a popular measure of educator performance, but they lack rigorous evaluation. This study investigates the error in MGP due to test score measurement error (ME). Using analytic derivations, we find that errors in the commonly used MGP are correlated with average prior latent achievement: Teachers with low prior achieving students have MGPs that underestimate true teacher performance and vice versa for teachers with high achieving students. We evaluate alternative MGP estimators, showing that aggregates of SGP that correct for ME only contain errors independent of prior achievement. The alternatives are thus more fair because they are not biased by prior mean achievement and have smaller overall variance and larger marginal reliability than the Standard MGP approach. In addition, the mean estimators always outperform their median counterparts.     

The Impact of Measurement Error on the Accuracy of Individual and Aggregate SGP

Student growth percentiles (SGPs) express students' current observed scores as percentile ranks in the distribution of scores among students with the same prior‐year scores. A common concern about SGPs at the student level, and mean or median SGPs (MGPs) at the aggregate level, is potential bias due to test measurement error (ME). Shang, vanIwaarden, and Betebenner (SVB; this issue) develop a simulation‐extrapolation (SIMEX) approach to adjust SGPs for test ME. In this paper, we use a tractable example in which different SGP estimators, including SVB's SIMEX estimator, can be computed analytically to explain why ME is detrimental to both student‐level and aggregate‐level SGP estimation. A comparison of the alternative SGP estimators to the standard approach demonstrates the common bias‐variance tradeoff problem: estimators that decrease the bias relative to the standard SGP estimator increase variance, and vice versa. Even the most accurate estimator for individual student SGP has large errors of roughly 19 percentile points on average for realistic settings. Those estimators that reduce bias may suffice at the aggregate level but no single estimator is optimal for meeting the dual goals of student‐ and aggregate level inferences.     

Referencing and borrowing from other systems: the Hong Kong education reforms

Abstract

Background: This paper analyses the role of, and approach to, policy referencing and borrowing in Hong Kong’s recent reforms that culminated in the creation of its New Academic Structure and the Hong Kong Diploma of Secondary Education.

Main argument: It argues that Hong Kong has gone further than most jurisdictions not just in responding to global influences on education reform, but in taking explicit steps to internationally benchmark its curriculum and assessment, and in involving the global community at multiple levels in the process of education policy planning and implementation.

Sources of evidence and method: The paper is based on the documentary analysis of policy documents in Hong Kong, and 23 interviews with key stakeholders in the policy network, including policy-makers, practitioners and community leaders.

Discussion and conclusions: While policy referencing and borrowing in the Hong Kong context can, in part, be traced to a colonial legacy, the Special Administrative Region of China demonstrates a collaborative approach to education reform involving local and international engagement that may be relevant to other systems. Its approach was informed by a measured use of policy referencing that involved ‘horizon scanning’ of other systems’ policies and practices; international benchmarking; and engaging international expertise to facilitate implementation.     

Situated learning,reflective practice and conceptual expansion: effective peer observation for tutor development

Despite tutors' importance, they often encounter inadequate professional development and support. This study describes the impact of peer observation of teaching activities on tutors' professional development using multiple data-sets over a three-year period. The data was analysed according to three themes: situated learning, reflective practice and conceptual expansion, in order to identify changes to teaching practice. Tutors reported that peer observation of teaching is an effective development activity that encouraged reflection on teaching and, for some, resulted in conceptual expansion and lasting change to teaching. Tutors found the situated aspect of the peer observation activity was an important factor, as it enabled them to observe how other tutors supported their students' engagement with the discipline-specific curriculum. Recommendations for future research and for the professional development of tutors in practice are provided.     

How to avoid burnout: A study of the relationship between burnout and worker,organizational and management characteristics in eleven child abuse and neglect projects

In conclusion this study affirms the hypothesis that burnout occurs when a constellation of factors--relating to worker, organizational and management--interact to create an alienating work environment. It suggests that solutions to the problem of worker burnout and performance require a systematic assessment of each of these areas to determine how and in what ways they contribute to burnout in any one particular agency. Further study and investigation that focuses on the interaction among worker, organizational and management characteristics is needed to affirm what common practices in social agencies lead to burnout and interfere with performance. Thereby, developing a relevant theory of management for social agencies that deals effectively with a high degree of uncertainty In the work task, that meets the needs and expectations of a professional manpower, and that can respond to changing and often hostile environmental conditions.     

Two reading assessments for youth in alternative basic skills and livelihood skills training programs

This article describes two assessment tools that have been used to assess the reading skills of youth participating in alternative basic skills and livelihood skills training programs. The Rapid Assessment of Reading Skills (RARS) was developed to identify potential participants who needed to improve their reading skills before beginning training and to assign them to the proper level of basic reading instruction. The Out-of-School Literacy Assessment incorporates RARS into a more complex impact evaluation tool. Both assessment tools can also be used to inform program design, define benchmarks, and assess impact.     

Associations between Fine Motor and Mathematics Instruction and Kindergarten Mathematics Achievement

Research Findings: This study investigates the role of fine motor and mathematics instruction in mathematics achievement in an international sample of kindergarteners from the United States and China. Multilevel modeling was used to assess the interaction between students’ entering skills and classroom time spent on basic math, higher-order math and fine-motor instruction. For American children, the effect of basic math and higher-order math instruction on student achievement depended on entering skills; however, fine motor instruction had negative average effects on student achievement and did not depend on students’ entering skills. Instruction time was not a significant predictor of achievement for Chinese students. Practice or Policy: Though fine motor skills have a robust correlation with mathematics achievement, a causal link has not been established. Our study indicates that time spent in fine motor instruction does not advance mathematics achievement in kindergarten and in fact may weaken mathematics achievement, given the limited time in the instructional day. American teachers in our sample who spent more time in fine-motor instruction tended to spend less time on basic math and higher-order mathematics instruction. Educators should weigh instructional trade-offs carefully and work to tailor instruction to students’ skill levels.     

Keeping Our Customers Happy: Myths and Management Issues in “Client-Led” Student Software Projects

In this paper we seek to dispel some of the myths and to address some of the real managerial issues which we believe may have deterred academics from involving commercial and industrial partners as customers in students' software project work. We contend that students know when a project is phoney, when requirements are dreamt up by academics and when the deliverables are of no practical use once a project mark has been assigned. The involvement of an external customer materially affects students' motivation, the quality of work that they produce and their sense of achievement. Put simply, authenticity is the key to a rich, challenging and meaningful educational project experience.     

Discussion Across Borders: Benefits for Collaborative Learning

Online learning is facilitated by various forms of computer-mediated communication (CMC). In higher education, CMC presents an opportunity to expand the learning community, even across national borders. This paper reports on two cycles of action research into the use of online discussion forums to enable groups of students from different countries to collaborate with each other in achieving learning outcomes specific to each group. The research data was obtained from questionnaires and focus groups with students and analysis of the content of the online discussion. The discussion was focused on the evaluation of prototype learning systems, produced by one group of students and evaluated by groups from two institutions in different countries, each of which had different, yet reciprocal, objectives to achieve from participating in the activity. The findings highlight the potential complexity of such an asymmetric collaborative activity, but indicate several advantages to students when the activity forms a distinct part of the pedagogical framework of their modules. For tutors planning similar trans-national collaborations, guidance points are offered that incorporate the findings from the research. Discussion par delÀ les frontières: les bénéfices de l’apprentissage coopératif. L’apprentissage en ligne est facilité par des formes variées de communication par ordinateurs (computer mediated communication CMC) Dans l’enseignement supérieur la CMC fournit l’occasion d’élarger la communauté d’apprentissage, même par delÀ les frontières. Dans cet article nous faisons rapport sur 2 cycles de recherche – action dans l’utilisation de forums de discussion en ligne pour donner la possibilité À des groupes d’étudiants de différents pays de callaborer l’un avec l’autre en réussiant des résultats d’apprentissage spécifiques À chaque groupe. Nos résultats de recherche ont été obtenues

À partir de questionnaires et de groupes choisis d’étudiants et de l’analyse de contenu des discussions en ligne. La discussion

était centrée sur l’évaluation de systèmes d’apprentissage prototypes produits par un groupe d’étudiants et évalués par des groupes de deux institutions dans des pays différents, chacun d’eux ayant des objectifs différents, et cependant réciproques, pour les réaliser par la participation à l’activité. Les résultats mettent en valeur la complexité potentielle d’une telle activité asymétrique coopérative, mais mentionnent divers avantages pour les étudiants lorsque l’activité forme une partie distincte du

cadre pédagogique de leurs modules. Tous les tuteurs qui font des plans de telles collaborations transnationales, nous proposons des conseils et des lignes directrices qui incorporent les résultats de notre recherche. Besprechungen über Grenzen hinweg: Hilfen für Gemeinschaftliches Lernen. Online-Learning wird durch verschiedene Formen computergestützter Kommunikation (CMC: computer mediated

communication) erleichtert. Im Bereich der höheren Bildung können durch CMC die Lerngemeinschaften ausgeweitet werden, selbst über nationale Grenzen hinaus. Mit diesem Beitrag berichten wir über zwei Zyklen von Wirkungsforschung in der Nutzung von Online-Diskussionsforen zur Ermöglichung von Zusammenarbeit zwischen Studentengruppen aus

verschiedenen Ländern durch Benutzung von Arbeitsergebnissen der anderen Gruppen. Die Daten für unsere Ergebnisse haben wir über Fragebogen und studentische Focus-Gruppen sowie durch Analyse des Inhalts der Online-Diskussion erhalten. Die Diskussion war auf die Auswertung von Muster-Lehrsystemen konzentriert, die von einer Studentengruppe produziert

und von Gruppen zweier Institutionen aus verschiedenen Ländern evaluiert. Jede Gruppe hatte unterschiedliche, sogar entgegengesetzte Aufgaben durch Teilnahme an der Aktion zu erfüllen. Die Ergebnisse unterstreichen die potentielle Komplexität einer solchen asymmetrischen Zusammenarbeit, weisen aber auf verschiedene Vorteile für Studenten hin, wenn die Aktivität einen deutlic hen Teil des pädagogisc hen Rasters ihrer Module dar stellt. Für Tutoren, die ähnlic he über nationale Zusammenarbeiten planen, stellen wir gerne Leitfäden mit den Ergebnissen unserer Forschung zur Verfügung.