Principals’ use of rating scale categories in classroom observations for teacher evaluation

Teacher evaluation systems commonly rely on observation of teaching practice (OTP) by school principals. However, the value of OTP as evidence of teacher effectiveness depends on its psychometric quality. In this study, we address a key aspect of the psychometric quality of principals’ OTP ratings. Specifically, we investigate the degree to which rating scale categories have a consistent interpretation across teaching episodes and practices. Results suggest that the 1,324 principals’ use of the rating scale categories functioned as intended overall. However, we also found that the midpoint category is underutilized and that rating categories do not always reflect similar levels of teaching effectiveness across teaching episodes and practices. When such discrepancies occur, we cannot assume principals’ ratings reflect a consistent level of teacher effectiveness within and across classrooms. This is a critical component of validity evidence that can inform the interpretation of OTP ratings and point to areas for improvement in both the rubrics and in principals’ training for classroom observations.     

The use of scoring rubrics: Reliability, validity and educational consequences

Several benefits of using scoring rubrics in performance assessments have been proposed, such as increased consistency of scoring, the possibility to facilitate valid judgment of complex competencies, and promotion of learning. This paper investigates whether evidence for these claims can be found in the research literature. Several databases were searched for empirical research on rubrics, resulting in a total of 75 studies relevant for this review. Conclusions are that: (1) the reliable scoring of performance assessments can be enhanced by the use of rubrics, especially if they are analytic, topic-specific, and complemented with exemplars and/or rater training; (2) rubrics do not facilitate valid judgment of performance assessments per se. However, valid assessment could be facilitated by using a more comprehensive framework of validity when validating the rubric; (3) rubrics seem to have the potential of promoting learning and/or improve instruction. The main reason for this potential lies in the fact that rubrics make expectations and criteria explicit, which also facilitates feedback and self-assessment.     

Rubrics as a way of providing transparency in assessment

This paper reports on a study where rubrics have been used to convey assessment expectations to students (n?=?176) in three different assessment situations in professional education. These situations are: (1) the development of a survey instrument, which was part of a course in statistics and epidemiology; (2) an inspection of a house, which was part of a course about the functions of buildings for real estate brokers and (3) a workshop in communication with patients, which was part of a course in the evaluation of diagnostic procedures and treatments of oral infections in dental education. In all situations, students’ perceptions and uses of the rubrics were investigated. Findings suggest that it is indeed possible to convey expectations to students through the use of rubrics, in the sense that students not only appreciate the efforts to make assessment criteria transparent, but may also use the criteria in order to support and self-assess their performance. Important features of the rubrics, which were found to facilitate students’ understanding and use of the criteria in these situations, are presented and discussed.     

The reliability and sources of error of using rubrics-based assessment for student projects

Rubrics are widely used in higher education to assess performance in project-based learning environments. To date, the sources of error that may affect their reliability have not been studied in depth. Using generalisability theory as its starting-point, this article analyses the influence of the assessors and the criteria of the rubrics on the assessment of two service-learning projects. A sample of 365 novice students studying for three different undergraduate degrees was evaluated by eight student assessors and two teachers at three stages of assessment. Depending on the type of project and the stage of assessment, between 19.27 and 39.55% of the total variance was attributed to the quality of the projects, 0–7.49% to the main effect of the raters, and 3.44–17.3% to the main effect of the criteria. The results demonstrated that acceptable levels of reliability (≥.70) were obtained with three raters and eight criteria or four raters and nine criteria in contexts of relative or absolute decisions, respectively.     

Demystifying the rubric: a five-step pedagogy to improve student understanding and utilisation of marking criteria

Marking-criteria rubrics are commonly used to judge the quality of student work, but few students receive instruction to effectively use and apply rubrics. This study investigates an intervention designed to encourage effective utilisation of rubrics. The study, completed over two years (2011: n?=?189 respondents; 2012: n?=?464 respondents) in a large first-year course, explored how five instructional activities, used formatively and sequentially impacted student learning. This intervention comprised: (1) deconstruction of the rubric and standardising the marking method; (2) examples and exemplars; (3) peer review; (4) self-review; and (5) a reflective diary. Results showed an increase in student confidence with marking criteria and assessment (89% in 2011) and a statistically significant improvement in marks. Most students (77% in 2011 and 90% in 2012) rated each intervention as useful. Student feedback indicated that substantial changes occurred between initial drafts and final submission, which resulted in improvements in student performance scores.     

Self‐Assessment in Art Education through a Visual Rubric

A tool for self assessment in secondary art education was developed and tested. The tool includes rubrics for assessing production and reception activities in art education and consists of visual and text rubrics. The criteria in the rubrics are based on the Common European Framework of Reference for Visual Literacy which was developed by The European Network of Visual Literacy (ENViL). The way teachers and students use the rubrics, whether they consider them helpful and to what extent students’ self‐assessments are in line with teacher assessments was studied. It was concluded that teachers work with the rubrics intensively and both students and teachers appreciate its visual form. However, it was found that the agreement between teachers and students about the students’ scores was moderate and needed to improve. The results show that it is untrue that students, or boys in particular, overestimate their own performance in art education. The current study contributes to the development of feasible and valid assessment criteria and instruments in secondary art education.     

Evaluating the performance of school psychologists: Current status and future directions with a 360-degree evaluation process

The purpose of this article is twofold: (a) to present the results of a study examining the current status of performance evaluations for school psychologist and (b) to use those findings to inform future directions. A content analysis of a national sample of 36 performance appraisal rubrics was conducted to examine their alignment with the professional standards established by the National Association of School Psychologists (NASP). Among the rubrics reviewed, 88.9% featured eight or more of the NASP domains of practice, yet only 56% of the rubrics featured all 10 domains, suggesting variability in the degree to which the rubrics aligned with the standards of practice, particularly for rubrics developed at the district-level. The results of this study informed the development of a 360-evaluation process based on the Framework for Personnel Evaluation of School Psychologists Utilizing the NASP Practice Model and Morrison's four key principles for evaluating the performance of school psychologists.     

The use of scoring rubrics for formative assessment purposes revisited: A review

The mainstream research on scoring rubrics has emphasized the summative aspect of assessment. In recent years, the use of rubrics for formative purposes has gained more attention. This research has, however, not been conclusive. The aim of this study is therefore to review the research on formative use of rubrics, in order to investigate if, and how, rubrics have an impact on student learning. In total, 21 studies about rubrics were analyzed through content analysis. Sample, subject/task, design, procedure, and findings, were compared among the different studies in relation to effects on student performance and selfregulation. Findings indicate that rubrics may have the potential to influence students learning positively, but also that there are several different ways for the use of rubrics to mediate improved performance and self-regulation. There are a number of factors identified that may moderate the effects of using rubrics formatively, as well as factors that need further investigation.     

Attention deficit disorder and specific reading disability: separate but often overlapping disorders

Within a large (N = 182) heterogeneous sample of clinic-referred children with DSM-III-diagnosed attention deficit disorder (ADD), three behavioral subgroups were identified via cluster analysis of teacher ratings: 40% of the children had ADD with hyperactivity (ADDH), 30% had ADD with hyperactivity and aggressivity (ADDHA), and 31% had ADD without hyperactivity or aggressivity. Proportionally more girls were in the ADD-only subgroup. Over half the sample (n = 94) were poor readers, with 82 meeting discrepancy criteria for specific reading disability (RD). Proportionately more boys than girls met the RD criteria (9.2:1.0), whereas the sex ratio of males to females for the whole sample was 5.1 to 1.0. Focusing just on white males, the three behavioral subgroups were significantly different on convergent validity measures, such as other teacher ratings, parent ratings, and interview-elicited ratings of externalizing behavior, but were not different on such divergent validity measures as IQ and achievement scores, self-ratings, and laboratory performance tasks. Boys in the ADD sample who did not meet criteria for RD had significantly higher IQs than those who did, but subgroups with and without RD still differed significantly on WRAT-R reading and spelling scores with IQ covaried out. Both groups with and without RD could be differentiated from a control group on laboratory measures of sustained attention and impulse control. Methylphenidate benefited all subgroups equally, whether RD or not, and whether given a low (0.3 mg/kg) or high (0.6 mg/kg) dose.     

基于标准的教育:利用评分规则促进学生学习

教育应该成为基于标准的教育,在教育教学过程中应该有相应的表现性准则、评价规则贯穿始终,这样教育才能真正达到其应有的目标。表现性准则和评价规则在基于标准的教育中具有重要的作用,应利用表现性准则和评价规则促进教学与学习。     

Position Paper on the Use of Rubrics

<正>Introduction In the beginning of this paper,we need to clarify the definition of"rubric".According to Schmoker(2006),"a rubric simply means a rule or guide…by which students’performance or product is judged.It nails down the criteria,making them available to schools,teachers,parents,and students and providing clear direction and focus"(pp.70-71,cited in Cooper,&Gargan,2009,p.54).When it comes to the question whether we should use rubrics for assessing students’performance,there     

Evaluating standards-based assessment rubrics in a postgraduate public health subject

Standards-based assessment rubrics are recognised as an important tool for ensuring clear criteria in higher education. Most of the available evidence on rubrics comes from studies with face-to-face undergraduate students. Our study evaluated the introduction of rubrics into a postgraduate subject that included both online and face-to-face students. We conducted formative evaluation with former students to develop the rubrics, as well as surveying current students for process and impact evaluation, supplemented with a focus group. Process results showed that the rubrics were used widely and were generally considered to be of at least some use. Clarity of the rubrics was the main issue affecting perceived usefulness, particularly for the assessment tasks that had undergone comparatively less formative evaluation. Impact evaluation showed mixed results, with students that reported always using rubrics having higher marks on average compared to students who did not always use the rubrics. However, an expected improvement in marks across the whole student cohort compared to previous years did not eventuate, although this could be explained by a number of factors independent of the rubrics. Our results provide some evidence for the use of rubrics in postgraduate and online settings but highlight the need for rigorous formative evaluation.     

Co-creating rubrics: The effects on self-regulated learning,self-efficacy and performance of establishing assessment criteria with students

The aim of this study was to compare the effects of co-creating rubrics against just using rubrics. By co-creating rubrics, the students might have the opportunity to better internalize them and have a voice in the assessment criteria. Two groups undertaking a degree in Sport Sciences (N = 65) participated. Results showed that the students who co-created the rubrics had higher levels of learning self-regulation measured through thinking aloud protocols, whereas the results from the self-reported self-regulation and self-efficacy questionnaires did not show significant differences. The treatment group outperformed the control group in only one out of the three tasks assessed. Regarding the perceptions about rubrics use, there were no significant differences except for the process of co-creation, to which the co-created rubric group gave higher importance. Therefore, this study has opened an interesting venue on rubrics research: co-creating rubrics may influence students’ activation of learning strategies.     

Development of evidence-based,student-learning-oriented rubrics for pre-service science teachers’ pedagogical content knowledge

ABSTRACT

This paper offers pedagogical content knowledge (PCK) rubrics, that is, guides providing criteria for grading that are potentially applicable to a range of science topics and levels of teacher experience. Grading criteria applied in the rubrics are based on qualitative analyses of planned topic-specific professional knowledge (TSPK) and content knowledge (CK). Data were collected via three topic-specific vignettes from 239 pre-service science teachers (PSTs) starting a university-based, full-time, one year teacher education programme in England. The statements were analysed for TSPK and CK. PSTs’ statements proposed instructional strategies comprising demonstrations, explanations, illustrations and analogies, classified as Relevant to the science topics, others Irrelevant. Some Relevant strategies missed an aspect that may, if enacted, help students’ learning, so were judged Incomplete. Statements were also analysed for evidence of relevant and correct CK. CK and TSPK statements are aligned into grids, creating PCK rubrics. These demonstrate the precise nature of knowledge likely to lead to instruction that impacts positively on student learning. The rubrics present the possibility of PCK repertoires that contribute clarity and precision to teaching instruction. Although findings cannot be generalised, the methodology offers a strategy for supporting out-of-field teachers, and those seeking instructional strategies to add to existing repertoires.     

RECONSTRUCTING CHILDREN'S EXPERIENCE WHEN TEACHING AND ASSESSING THEM: LESSONS FROM DEWEY

This paper describes an online early childhood assessment course that was developed through a multi-university collaboration with support from a state improvement grant. Collaborators from three universities developed the course to address a new early childhood unified license (birth to age 8, regular and special education) in the state of Kansas. After reviewing the new state content standards, we identified targeted understandings, performance assessments, and online activities for 15 modules using a backward design process. Emphasis was placed on active learning through synchronous and asynchronous interactions facilitated by the use of a course management system. Positive evidence of learning was indicated by anonymous student feedback, pretest/posttest gain scores, and performance assessments evaluated with rubrics. We viewed the implementation of the course as a success and anticipate that it may lead to more sharing of online coursework in Kansas teacher education programs in the future.     

Sensitivity of School-Performance Ratings to Scaling Decisions

Policymakers usually leave decisions about scaling the scores used for accountability to their appointed technical advisory committees and the testing contractors. However, scaling decisions can have an appreciable impact on school ratings. Using middle-school data from New York State, we examined the consistency of school ratings based on two scaling approaches that differed in scaling decisions that are important in high-stakes testing contexts. We found that, depending on subject, grade, and year, a switch in scaling approach led to (1) average absolute shifts in ranks of between 50 and 132 positions (median = 69), which are appreciable shifts for a listing of 1,243 schools; and (2) between 7% and 45% (average = 20%) of schools experiencing shifts in assigned performance bands, depending on the classification scheme. Further, the effect of scaling approach was larger when the raw-score distribution has more severe ceiling effect, and in these cases, it was driven primarily by the difference in the location of the highest obtainable scale score from the two scaling approaches.