Examining rater accuracy and consistency with a special education observation protocol

Research indicates that instructional aspects of teacher performance are the most difficult to reach consensus on, significantly limiting teacher observation as a way to systematically improve instructional practice. Understanding the rationales that raters provide as they evaluate teacher performance with an observation protocol offers one way to better understand the training efforts required to improve rater accuracy. The purpose of this study was to examine the accuracy of raters evaluating special education teachers’ implementation of evidence-based math instruction. A mixed-methods approach was used to investigate: 1) the consistency of the raters’ application of the scoring criteria to evaluate teachers’ lessons, 2) raters’ accuracy on two lessons with those given by expert-raters, and 3) the raters’ understanding and application of the scoring criteria through a think-aloud process. The results show that raters had difficulty understanding some of the high inference items in the rubric and applying them accurately and consistently across the lessons. Implications for rater training are discussed.     

Teachers' use of rubrics to score non‐traditional tasks: factors related to discrepancies in scoring

This study considered middle school mathematics teachers’ use of rubrics to score non‐traditional tasks. A group of eighth‐grade teachers attended a two‐day workshop where they evaluated assessment tasks and discussed the use an associated scoring rubric. Scored samples of student work submitted by the teachers indicated that they had difficulty using the rubrics for scoring. When compared to expert ratings, all except one teacher had discrepancies in scoring and some discrepancies indicated major problems. These discrepancies appear to be related to whether the task contained familiar or unfamiliar content and the mix of procedure and explanation the task required. Several other factors related to discrepancies, such as leniency errors, teacher knowledge, and the halo effect are also discussed. With the expanded use of rubrics in many arenas, these results show the need for more professional development related to rubric use.     

Students’ use of a rubric for research theses

A rubric for research theses was developed, based on the manual of the American Psychological Association, to be used as an assessment tool for teachers and students. The aim was to make students aware of what is expected, get familiar with criteria, and interpret teacher and peer feedback. In two studies, it was examined whether students use and value these functions. In the first study, a rubric was provided to 105 Educational Sciences students working on their bachelor’s thesis. Questionnaire data indicated that students did value the rubric for the intended functions, although rubric use was not related to ability. In a panel interview, teachers stated that the number of proficiency levels should be increased to be able to distinguish between good and excellent students adequately, and that a criterion concerning student’s role during supervision should be added. Therefore, in the second study, 11 teachers were interviewed about their motives to give high grades and about the supervision process. This lead to an extra criterion concerning student’s role during supervision and an additional proficiency level to assess excellent performance. It is argued that an adequate course organisation is conditional for the rubric’s effectiveness.     

Teacher Candidate Dispositions: Perspectives of Professional Expectations

This study describes a programmatic effort to examine dispositions perceptions of teacher candidates entering the profession. Study participants included 114 master's level teaching candidates in their first semester of a nontraditional teacher education program. Teacher candidates scored themselves on a department disposition rubric designed to evaluate their professional behaviors and wrote reflective rationales to support their scoring. Results indicated that candidates positioned themselves as no fault, accepts fault, and assigns fault in writing their rationales. These positions were used as lenses to analyze candidates' positioning in relation to the program and faculty expectations. Candidates' disposition performance rationales relied on their novice status, the demands of work and family, and an admitted minimalist approach to some course demands. Insights gained from the study may allow teacher education faculty to better understand and support candidates' dispositional growth and development.     

High-stakes teacher evaluation policy: US principals’ perspectives and variations in practice

Principals’ implementation of new teacher evaluation policies in a suburban and rural southeastern area of the United States was examined over a five-year period. This study reports findings on two of eleven interview questions examining changes in principals’ perceptions over time regarding policy concerns and benefits. Findings indicate while initially overwhelmed, principals eventually managed implementation time challenges and later focused on the benefits of evaluation. Secondly, principals quickly integrated the instructional rubric criteria into classroom observations and professional development work. Third, increasing doubts emerged regarding the inconsistent application of the rubric criteria, the inclusion of student test scores in teacher evaluation, and the calculation of teacher effectiveness ratings. The authors conclude that mandating rigorous evaluation policy will not sufficiently address teacher effectiveness and may complicate principals’ instructional leadership. They assert that policy-makers must consider the long-term effects of implementation before substantial teacher evaluation change results.     

Cycles of Research and Application in Education: Learning Pathways for Energy Concepts

ABSTRACT— We begin this article by situating a methodology called developmental maieutics in the emerging field of mind, brain, and education. Then, we describe aspects of a project in which we collaborated with a group of physical science teachers to design developmentally informed activities and assessments for a unit on energy. Pen-and-paper assessments, called teasers , were employed, along with interviews, to study how students learned about the physics of energy. Results were used to describe students' learning pathways and to design a scoring rubric for teacher use. We hypothesized that (a) teasers, by themselves, could be used effectively to evaluate the developmental level of students' reasoning about energy and (b) teachers could employ the scoring rubric with minimal instruction. Encouraged by our findings, we went on to create a freely available online version of the energy teaser , including a new rubric designed to improve the accuracy with which teachers can assess the developmental level of students' energy conceptions.     

Development of a performance assessment task and rubric to measure prospective secondary school mathematics teachers’ pedagogical content knowledge and skills

The purpose of this paper is to share a performance assessment task and rubric designed to assess secondary school mathematics preservice teachers’ pedagogical content knowledge and skills. The assessment task and rubric were developed in collaboration with five education faculty, four arts and sciences faculty, and four high school teachers over a period of one year. Over this period, the group met more than 15 times and designed an assessment task and rubric to measure secondary school preservice teachers’ pedagogical content knowledge and skills. The assessment task and rubric were piloted at the end of the year and extensively revised using the pilot data. The redesigned task and rubric were successfully carried out in the second and subsequent years. Other teacher education programs can benefit from this assessment of teacher candidates’ pedagogical content knowledge and skills.     

The impact of the scoring system of a large-scale standardized EFL writing assessment on its score variability and reliability: Implications for assessment policy makers

Using generalizability (G-) theory and rater interviews as research methods, this study examined the impact of the current scoring system of the CET-4 (College English Test Band 4, a high-stakes national standardized EFL assessment in China) writing on its score variability and reliability. One hundred and twenty CET-4 essays written by 60 non-English major undergraduate students at one Chinese university were scored holistically by 35 experienced CET-4 raters using the authentic CET-4 scoring rubric. Ten purposively selected raters were further interviewed for their views on how the current scoring system could impact its score variability and reliability. The G-theory results indicated that the current single-task and single-rater holistic scoring system would not be able to yield acceptable generalizability and dependability coefficients. The rater interview results supported the quantitative findings. Important implications for the CET-4 writing assessment policy in China are discussed.     

Generalizability and Validity of the Use of a Case Analysis Assessment to Make Visible the Quality of Teacher Candidates

The use of case analysis as an accountability measure for demonstrating teacher candidates abilities to meet targeted course, institutional, and state teaching standards for student motivation and classroom management was evaluated. The findings supported the generalizability of case analysis ratings made using a standards-based scoring rubric, and the ratings were shown to differentiate levels of performance in accordance with those standards. Support for the content representativeness of the performance assessment was also obtained using criteria suggested by Crocker (1997), including the realism, frequency, necessity, and importance of the targeted teaching behaviors to actual teaching practice. The findings also provided initial support for the predictive validity of the case analysis assessment for appraising candidates overall abilities to manage a public school classroom.     

In search of a purpose for school history

Taking into account the merits of anticipatory reflection, instructional planning is perceived as an important process in the professionalization of teachers. When implementing a complex instructional strategy such as collaborative learning (CL), a thorough preparation becomes even more important. The purpose of the present study was to investigate the quality of lesson plans focusing on CL implementation. Based on the literature, a scoring rubric with 17 criteria in three domains (instruction, organization and evaluation) was developed and applied to analyse 323 lesson plans of second-year pre-service teachers. The results reveal both strengths (e.g. designing appropriate learning tasks, developing adequate learning materials) and weaknesses (e.g. including social objectives, rules and agreements for collaborative work) in the lesson plans. The rubric proves to be a useful instrument both for research and practice-oriented reflection. The findings provide significant insights for teacher training with regard to CL implementation.     

Evaluating Comparative Judgment as an Approach to Essay Scoring

ABSTRACT

As an alternative to rubric scoring, comparative judgment generates essay scores by aggregating decisions about the relative quality of the essays. Comparative judgment eliminates certain scorer biases and potentially reduces training requirements, thereby allowing a large number of judges, including teachers, to participate in essay evaluation. The purpose of this study was to assess the validity, labor costs, and efficiency of comparative judgments as a potential substitute for rubric scoring. An analysis of two essay prompts revealed that comparative judgment measures were comparable to rubric scores at a level similar to that expected of two professional scorers. The comparative judgment measures correlated slightly higher than rubric scores with a multiple-choice writing test. Score reliability exceeding .80 was achieved with approximately nine judgments per response. The average judgment time was 94 seconds, which compared favorably to 119 seconds per rubric score. Practical challenges to future implementation are discussed.     

教师绩效评价的理论与实践

教师绩效评价是教育人事制度改革的关键,20世纪80年代以来,它成为了教育改革的重点和难点。在以往有关教师评价的研究中,三种不同类型的评价常常被混用;而对教师绩效的评价又常常只关注任务绩效,没有关注关系绩效;此外,大多数教师评价研究均是关于大学教师的。教师绩效评价应以教师绩效结构的理论分析为基础。基于教师素质结构理论,在文献分析和关键事件调查、关键事件访谈的基础上,通过定性分析和实证分析,在反复研究基础上,提出了教师绩效的结构,并通过实证研究探讨了教师绩效评价的信度与效度。     

The Quality of Local District Assessments Used in Nebraska's School-Based Teacher-Led Assessment and Reporting System (STARS)

A sample of 293 local district assessments used in the Nebraska STARS (School-based Teacher-led Assessment and Reporting System), 147 from 2004 district mathematics assessment portfolios and 146 from 2003 reading assessment portfolios, was scored with a rubric evaluating their quality. Scorers were Nebraska educators with background and training in assessment. Raters reached an agreement criterion during a training session; however, analysis of a set of 30 assessments double-scored during the main scoring session indicated that the math ratings remained reliable during scoring, while the reading ratings did not. Therefore, this article presents results for the 147 mathematics assessments only. The quality of local mathematics assessments used in the Nebraska STARS was good overall. The majority were of high quality on characteristics that go to validity (alignment with standards, clarity to students, appropriateness of content). Professional development for Nebraska teachers is recommended on aspects of assessment related to reliability (sufficiency of information and scoring procedures).     

教师专业化视野下教师培训实效性研究——以初中英语骨干教师为例

英语骨干教师培训作为提高英语教师教育质量的重要举措之一,其示范与辐射作用的发挥对推动英语教师专业化发展有重要作用。采用问卷与访谈法对初中英语骨干教师培训实效性进行质性与量性的调查研究,在对调查结果分析的基础上,提出提高培训实效性的策略建议。     

Critical Thinking in Preservice Teachers: A Rubric for Evaluating Argumentation and Statistical Reasoning

The authors implemented a cognitive apprenticeship learning community (Collins, Brown, & Newman, 1989) in a class of preservice teachers at the University of Missouri - Columbia to enhance their argumentation and critical thinking skills about complex, educational problems. A detailed rubric based on the literature in argumentation and scientific reasoning was developed to evaluate the students' performance before and after the intervention. The rubric was designed to measure students' (a) conceptions and use of evidence, (b) notions about research and its applicability in evaluating complex social problems, and (c) ability to consider alternative perspectives. This article describes the intervention, the theoretical underpinnings of the rubric, and the use of the rubric in measuring the development of critical thinking in a group of preservice teachers. Strengths and weaknesses of the rubric are outlined and avenues for future research are discussed.     

真实性评价中的评分规则

评分规则是真实性评价中不可或缺的组成部分，它作为描述性评价方案具有四个构成要素；它不仅是评价性工具，也是教学性工具；更为重要的是，它在教师的专业发展和课堂教学评价中有着重要意义。当然，评分规则需要合理地开发和运用。     

Using rubrics in student self-assessment: student perceptions in the English as a foreign language writing context

The instructional value of rubrics for promoting student learning and aiding teacher feedback to student performance has been extensively researched in th educational literature. There is, nonetheless, a dearth of studies on students’ rubric use in second/foreign language contexts, and fewer studies have investigated the factors affecting rubrics’ effectiveness for promoting student learning. The paper reports a classroom-based inquiry into students’ perceptions of rubric use in self-assessment in English as a Foreign Language context and the factors moderating its effectiveness. Eighty students at a Chinese university participated in the study. The data collected included their reflective journals and six case study informants’ retrospective interviews. Results showed that the rubric was perceived as useful for fostering the students’ self-regulation by guiding them through the stages of goal-setting, planning, self-monitoring and self-reflection. Both within-rubric and rubric-user factors were identified as affecting the rubric’s effectiveness in student self-assessment. The findings are discussed with reference to the design features of rubrics. Implications are drawn for formative rubric use in student self-assessment.     

Reflection as a visible outcome for preservice teachers

As the standards movement progresses, efforts to encourage reflection by student teachers are often undermined. In this piece, we analyze exemplars of student teacher reflection coming from two very different approaches to outcomes-based teacher preparation. We use these exemplars to develop a rubric that illuminates the dimensions and qualities of reflection. This rubric helps clarify how meaningful reflection and an emphasis on learning are not incompatible if the focus is placed on the process of learning, rather than on outcomes alone. Finally, we contend that engagement in the process of reflection and reflection on the moral enterprise of teaching can be considered as important outcomes in their own right.     

Using Rubrics for Documentation of Clinical Work Supervision

In this article, the use of a scoring rubric for use in clinical supervision is discussed. The rubric is defined, a brief example is provided, and guidelines for development are included.