Evaluating the performance of school psychologists: Current status and future directions with a 360-degree evaluation process

The purpose of this article is twofold: (a) to present the results of a study examining the current status of performance evaluations for school psychologist and (b) to use those findings to inform future directions. A content analysis of a national sample of 36 performance appraisal rubrics was conducted to examine their alignment with the professional standards established by the National Association of School Psychologists (NASP). Among the rubrics reviewed, 88.9% featured eight or more of the NASP domains of practice, yet only 56% of the rubrics featured all 10 domains, suggesting variability in the degree to which the rubrics aligned with the standards of practice, particularly for rubrics developed at the district-level. The results of this study informed the development of a 360-evaluation process based on the Framework for Personnel Evaluation of School Psychologists Utilizing the NASP Practice Model and Morrison's four key principles for evaluating the performance of school psychologists.     

Performance Indicators and League Tables: A Call for Standards

League tables are weighted combinations of scores on performance indicators. The Times newspaper publishes an annual league table of British universities and the focus of this article is the 1994 table. The scores of ninety-six universities on fourteen performance indicators are ranked using weights which 'reflect the interests of students'. The league table is intended as a resource which allows students to compare universities and therefore it is important for both students and universities that valid inferences can be drawn from its use. The table fails to meet any of the technical requirements which would assure its internal construct validity. The educational measurement literature requires that validity inquiry must include consideration of social consequences and it is demonstrated that the technical shortcomings of the Times table could have significant adverse impact, even in the top ten universities. It is argued that universities should forsake their current stance of non-co-operation and develop a shortlist of demanding technical standards, linked to the educational measurement literature, which could be referenced in effective attacks upon poorly constructed league tables.     

Addressing bullying through critical making

This paper explores how maker pedagogies helped middle school students develop transferable competencies, such as creativity, problem solving, self-directed learning and citizenship skills. We offer an in-depth look at how purposeful or critical making related to bullying awareness can help struggling students develop positive attitudes related to their roles as contributing members of a classroom community of learners.     

Predicting violence from school misbehavior: Promises and perils

This article considers the issue of predicting violence committed by students on school campuses through the use of school discipline indices such as office referrals, suspensions, and expulsions. The existing research on incidents of school discipline markers is reviewed as well as sources of variation due to student characteristics and classroom or school‐wide practices. Challenges in making accurate predictions about future behavior at school are outlined. © 2001 John Wiley & Sons, Inc.     

Research Collaboration Among University Scientists

Despite the growing importance of collaboration in research there have been very few investigations of the practice of research collaboration itself. The study we report investigated this practice by analysing 444 collaborative projects undertaken by staff in the Science Faculty of a New Zealand university. While the results support the sociology of science model of vertical collaboration up and down the academic hierarchy, we also show that significant collaboration now takes place across levels in the hierarchy, that is among peers, in what we call horizontal collaboration. This shift from vertical to horizontal collaboration has not been readily apparent in bibliographic studies of co-authored papers in top journals. One of the questions this study raises is the often assumed positive association between collaboration, research output and research quality, and the implications such assumptions have on the institutionalisation of research within the university. We end by suggesting that the shift that is occurring in the location of research from conventional departments to research centres within the university may signal an attempt to resurrect the practice of vertical collaboration.     

Average Effect Sizes in Developer-Commissioned and Independent Evaluations

Abstract

Rigorous evidence of program effectiveness has become increasingly important with the 2015 passage of the Every Student Succeeds Act (ESSA). One question that has not yet been fully explored is whether program evaluations carried out or commissioned by developers produce larger effect sizes than evaluations conducted by independent third parties. Using study data from the What Works Clearinghouse, we find evidence of a “developer effect,” where program evaluations carried out or commissioned by developers produced average effect sizes that were substantially larger than those identified in evaluations conducted by independent parties. We explore potential reasons for the existence of a “developer effect” and provide evidence that interventions evaluated by developers were not simply more effective than those evaluated by independent parties. We conclude by discussing plausible explanations for this phenomenon as well as providing suggestions for researchers to mitigate potential bias in evaluations moving forward.     

One course is not enough: Preservice elementary teachers' retention of improved views of nature of science

This study examined the views, and the retention of these views, of 19 preservice elementary teachers as they learned about nature of science (NOS). The preservice teachers participated in a cohort group as they took a science methods course during which they received explicit reflective instruction in nature of science. Through Views of Nature of Science version B (VNOS‐B) surveys and interviews it was found that most preservice teachers held inadequate ideas of nature of science prior to instruction, but improved their views after one semester of instruction in the science methods course. However, 5 months after instruction, the graduate preservice teachers were again interviewed and it was found that several of the students reverted back to their earlier views. The results are interpreted through Perry's scheme, and implications and recommendations for elementary science teacher education are made. © 2005 Wiley Periodicals, Inc. J Res Sci Teach 43: 194–213, 2006     

Setting Reliable National Curriculum Standards: a guide to the Angoff procedure

The Northern Ireland Curriculum, like the English National Curriculum, records pupil achievement on a 10‐level scale. The level to which a pupil is ‘assigned’ at the end of a Key Stage is based upon two sources of assessment information: classroom‐based measures provided by the teacher and summative information from Common Assessment Instruments (CAIs), which are pen‐and‐paper tests taken at the end of the Key Stage. CAIs play a central role in confirming the accuracy with which teachers judge the level at which a pupil is working. While the teacher might judge a pupil to have mastered level 7 in Algebra, for example, based upon observation in class, test data and homeworks, the CAI will only confirm this level if the pupil scores above the level 7 cutscore on the CAI. If this cutscore does not accord with a reliable measure of what constitutes level 7 performance in Algebra in the classroom, there is likely to be misclassification of pupils with attendant difficulties for the efficient planning of teaching and learning. Misclassifications can be minimised when examiners and teachers interpret level 7 achievement in Algebra similarly. The Angoff standard‐setting procedure was used to establish level 5 cutscores in the Number and Handling Data tests of the mathematics CAI so that comparisons might be made between the published level 5 cutscores and those which result from a judgemental standard‐setting procedure. The 21 teachers involved in the procedure were offered the opportunity to recommend a level 5 ‘standard’ using the Angoff methodology, and to review their recommendations in the light of test data from the February 1993 CAI administration. A further opportunity was offered following a discussion during which individual teachers articulated their reasons for the standards they recommended. The results confirm that the reliability of recommended standards increases both as a consequence of receiving normative data and of discussion. All statistical measures reported in this article indicate that the procedure could command the confidence of examiners, teachers and the public. While the recommended cutscore for Number is in close accord with that published by the examiners, the extent of the mismatch in the Handling Data test is such as might give rise to some misclassification of pupils. It is important to stress that this mismatch had no real consequences since 1993 was a pilot year and no test outcomes were reported. The article concludes with an outline of the contribution which the Angoff methodology can make to the resolution of some of the difficulties faced by English national assessment, as identified in Sir Ron Dealing's interim report “The National Curriculum and its Assessment”.