Evaluating assessment quality in competence-based education: A qualitative comparison of two frameworks

Because learning and instruction are increasingly competence-based, the call for assessment methods to adequately determine competence is growing. Using just one single assessment method is not sufficient to determine competence acquisition. This article argues for Competence Assessment Programmes (CAPs), consisting of a combination of different assessment methods, including both traditional and new forms of assessment. To develop and evaluate CAPs, criteria to determine their quality are needed. Just as CAPs are combinations of traditional and new forms of assessment, criteria used to evaluate CAP quality should be derived from both psychometrics and edumetrics. A framework of 10 quality criteria for CAPs is presented, which is then compared to Messick's framework of construct validity. Results show that the 10-criterion framework partly overlaps with Messick's, but adds some important new criteria, which get a more prominent place in quality control issues in competence-based education.     

The Regime of Self-Evaluation: Self-Conception for Teachers and Schools

Self-evaluation in inspection policy has become a global phenomenon. The idea is that it increases levels of teacher and school autonomy, wherein both schools and teachers have more ownership and responsibility over their work. In turn, such a process has allowed for greater accountability, which is then said to provide high quality education and, therefore, greater competitive advantage amongst knowledge-based economies. In both England and Ireland, self-evaluation has become a demanding procedure that is meant to complement external inspections of schools and teachers. In this article, I will argue that self-evaluation, whilst having the potential to become a worthwhile endeavour, does not live up to its name. In the first instance, the criteria used for self-evaluation are not internally generated but externally imposed. Thus, I would like to discuss the extent to which visions of ‘good’ or ‘bad’ education developed by inspecting bodies influence the way in which teachers and schools assess themselves. Furthermore, I will raise questions as to what appropriate criteria for teaching might look like. In doing so, I shall try to show that what is now current is a debased form of self-evaluation that is not only detrimental to the self-perception of teachers, but inadequate to what any coherent notion of the ‘self’ might be. In light of work by the philosopher Charles Taylor in particular, I will argue that the self is not something that can be examined in the way that is imagined in these inspection systems but is rather something dynamic and unfixed, constituted within a wider community of practice and, therefore, not amenable to evaluation in quite the way that is supposed.     

Child protective services utilization of child abuse pediatricians: A mixed methods study

Several children’s hospitals and medical schools across Texas have child abuse pediatricians (CAPs) who work closely with child protection workers to help ensure accurate assessments of the likelihood of maltreatment in cases of suspected abuse and neglect. Since the state does not mandate which cases should be referred to a CAP center, we were interested in studying factors that may influence workers’ decisions to consult a CAP. We used a mixed methods study design consisting of a focus group followed by a survey. The focus group identified multiple factors that impact workers’ decision-making, including several that involve medical providers. Responses from 436 completed surveys were compared to employees’ number of years of employment and to the state region in which they worked. Focus group findings and survey responses revealed frustration among many workers when dealing with medical providers, and moderate levels of confidence in workers’ abilities to make accurate determinations in cases involving medical information. Workers were more likely to refer cases involving serious physical injury than other types of cases. Among workers who reported prior interactions with a CAP, experiences and attitudes regarding CAPs were typically positive. The survey also revealed significant variability in referral practices by state region. Our results suggest that standard guidelines regarding CAP referrals may help workers who deal with cases involving medical information. Future research and quality improvement efforts to improve transfers of information and to better understand the qualities that CPS workers appreciate in CAP teams should improve CAP-CPS coordination.     

The effects of the use of a school self-evaluation instrument

School quality care has become important in many Western countries. A number of self-evaluation instruments have been developed in response. The goal of these instruments is improving the quality of education. One such system, ZEBO, was developed for performing quality assessment in Dutch primary schools. The use of this self-evaluation instrument was studied in 79 primary schools in The Netherlands over a period of 5 years. The results of the multilevel analyses with repeated measures show that the use of the self-evaluation results had no effect on pupil achievement scores (so far). However, it did have other effects, for example, an impact on the professional development of teachers.     

Exploring the Technical Quality of Using Assignments and Student Work as Indicators of Classroom Practice

This study presents findings from research developing indicators of classroom practice to monitor the influence of school reform initiatives on students' learning environments and to support the improvement of instructional practice. The work reported here entailed collecting assignments and student work and observing classrooms in schools participating in a large-scale urban school reform effort. Overall, the reliability and consistency of the assignment ratings were good. Results also indicate that the quality of the assignments was statistically significantly associated with the quality of observed instruction and student work. It appears that our method shows promise for use in large-scale evaluation settings and identifies important dimensions of practice that could support teacher self-evaluation and reflection.     

成长记录袋的创建与使用

成长记录袋有意识地收集学生的作品或有关证据,以反映学生在某一领域的努力、进步与成就。任课教师创建和使用成长记录袋一般要经过明确目的和用途、对内容选择提供指导、明确学生角色、确定评分程序和标准、交流和使用等几个主要步骤。在当前的评价实践中,学校和教师还要注意谨慎选择成长记录袋的应用领域、将成长记录袋的应用与教学有机结合、给教师和学生足够的自主权并加强教师培训。     

Development and Field Test of an Employment Selection Instrument for Teachers in Urban School Districts

Employment interviews are widely used in the selection of quality teachers, and indeed research confirms administrators’ belief in the validity of the procedure. However, many key recommendations for improving the general reliability of interviews including selecting questions that are job-related and research grounded, including well designed scoring rubrics, and incorporating adaptive variable-length interview designs are generally not well implemented in currently available instruments. Furthermore, emerging research suggests the need for specially tailored interviews that assess attitudes and pre-dispositions deemed essential to teachers’ effectiveness in certain high attrition environments like urban schools. This study describes the development and initial field-test results of a computer based, adaptive interview with an additional domain included for teachers in urban areas. The instrument is based on careful analysis of the suggestions from the extant literature base about effective employment interview techniques, effective general teaching practices, and sound teaching strategies in urban schools. By comparing the interview scores of 30 teachers with varying effectiveness ratings provided by administrators in one urban district, significant correlations were found. Regression analysis indicated a significant amount of variance in teachers’ effectiveness ratings could be predicted from their scores on the interview instrument.

An examination of the predictive relationships of self-evaluation capacity and staff competency on strategic planning in Hong Kong aided secondary schools

This article aims to examine the predictive relationships of self-evaluation capacity and staff competency on the effect of strategic planning in aided secondary schools in Hong Kong. A quantitative questionnaire survey was compiled to collect data from principals of the participating schools. Confirmatory factor analysis and reliability tests were applied to confirm the constructed validity and reliability of the survey instrument. Multiple regression analysis was applied to explore the relationships among variables. The capacity of self-evaluation mechanisms and staff competency are identified as predictive factors for effective strategic planning. An effective quality assurance (QA) mechanism could support the sustainable development of school organization. School administrators could embed self-evaluation mechanisms into strategic planning and provide staff development to improve the QA systems in their schools.     

School self-evaluation for development: The case for and reflections on a training programme

Abstract

This paper explores the case for schools conducting their own self-evaluation to serve the dual purpose of professional development within schools and accountability demands from external sources. The paper is in three parts. The first outlines the argument and benefits for schools conducting their own case study evaluation of their curriculum policies and practices, indicating how this approach is both similar to and different from external case study of schools. The second describes a training programme designed and implemented by the author to support schools in conducting evaluations that are valid, credible and useful. The third presents the participants' and author's reflections on the training and its outcomes and, through an example from one of the schools involved, illustrates what school self-evaluation looks like grounded in the actual experience of undertaking the process. From the evidence presented, the paper concludes that educational professionals, with appropriate support and training, are more than equal to the task of conducting credible and valid case study school self-evaluations.     

Classroom learning experiences and students’ perceptions of quality of school life

It is generally believed that classroom learning experiences very much influence students’ academic development. However, relatively little is known about whether classroom learning experiences have much effect on students’ affective and social development. In this study, we argued for the importance of learning experiences on students’ affective and social development. From research on Accelerated Schools Project and active learning, we conceptualised Positive Learning Experiences, Teacher Support in Learning, and Active Learning Experiences as three components of learning experiences and developed relevant measures to tap these learning experiences. Using research data from a large-scale student survey in Hong Kong (N = 19,477), we examined the construct validity of learning experiences and quality of school life. Confirmatory factor analysis provided very strong support for the measures and the underlying constructs that they tap. In two-way analysis of variance, the effects of gender and school level (secondary versus primary) on learning experiences and quality of school life were examined. Significant gender and school-level main effects, as well as gender by school-level interaction effects, were found for students’ ratings of their learning experiences and quality of school life. Female students gave more favourable ratings than male students, and primary students gave more favourable ratings than secondary students. In addition, gender differences in these ratings in secondary schools were relatively smaller than in primary schools. In subsequent multilevel modelling, learning experiences were strong predictors of quality of school life after controlling for the effects of gender, school level and average school achievement. These findings provide strong empirical support for the significance of classroom learning experiences for students’ social and affective development.

发展性学校评价在我国实施的个案研究

通过四所个案学校的实践历程发现,发展性学校评价在我国的实施具有一定的可行性,但仍存在一些问题和不足。具体表现在:政府督导室的支持和外部评价组的权威性有助于发展性学校评价的实施,但大部分学校自身参评的动力并不是很强;大多数学校均能组成自评小组,开发自评工具,进行自我评价,并形成了内容相对丰富的自评报告,但自评人员的参与面较窄,自评报告的"迎评化"色彩较浓,大多数教师和校长亦缺乏相应的评价知识和技能;外部评价组的专业性和投入度有力地保障了外评的效果,但也存在着评价时间较短、评价过程过于紧凑、评价队伍的构成不够多元等问题。     

Inter-rater reliability of physical abuse determinations in young children with fractures

As there is no “gold standard” in determining whether a fracture is caused by accident or abuse, agreement among medical providers is paramount. Using abstracted medical record data from 551 children <36 months of age presenting to a pediatric emergency department, we examined the extent of agreement between specialists who evaluate children with fractures for suspected abuse. To simulate clinical scenarios, two pediatric orthopaedists and two child abuse pediatricians (CAPs) reviewed the full abstraction and imaging, whereas the two pediatric radiologists reviewed a brief history and imaging. Each physician independently rated each case using a 7-point ordinal scale designed to distinguish accidental from abusive injuries. For any discrepancy in independent ratings, the two specialists discussed the case and came to a joint rating. We analyzed 3 types of agreement: (1) within specialties using independent ratings, (2) between specialties using joint ratings, and (3) between clinicians (orthopaedists and CAPs) with more versus less experience. Agreement between pairs of raters was assessed using Cohen's weighted kappa. Orthopaedists (κ = 0.78) and CAPs (κ = 0.67) had substantial within-specialty agreement, while radiologists (κ = 0.53) had moderate agreement. Orthopaedists and CAPs had almost perfect between-specialty agreement (κ = 0.81), while agreement was much lower for orthopaedists and radiologists (κ = 0.37) and CAPs and radiologists (κ = 0.42). More-experienced clinicians had substantial between-specialty agreement (κ = 0.80) versus less-experienced clinicians who had moderate agreement (κ = 0.60). These findings suggest the level of clinical detail a physician receives and his/her experience in the field has an impact on the level of agreement when evaluating fractures in young children.     

New regimes of truth: The impact of performative school self evaluation systems on teachers' professional identities

The revised inspection system in England requires schools to maintain a web-based self-evaluation, which is then used as the basis for high-stakes external inspection. This paper develops a Foucauldian analysis of interview data from a single case study site to illustrate the ways that new regimes of truth are created within schools and to consider the impact of disciplinary systems on teachers' professional identities.     

Critical facilitators: External supports for self-evaluation and improvement in schools

This paper explores approaches used to support schools to engage in self-evaluation for the purpose of improvement. General models of support are outlined which emphasise evaluation capacity building however the usefulness of this approach is questioned. Instead the role of the critical friend as a provider of external professional support is explored. An approach which emphasises facilitation and the engagement of an external professional to lead self-evaluation processes in schools is suggested with a view to making the process ‘easier’ and more ‘useful’. Research in relation to such an approach carried out in a range of educational contexts is presented. Implementation issues which will have relevance for schools, and for self-evaluation policy and practice in other jurisdictions are outlined.     

Student self-evaluation,teacher evaluation,and learner performance

A total of 341 Latvian students and eight teachers participated in this study of student self-evaluation and teacher evaluation. Students completed a 12-lesson teacher-directed instructional program on conducting and writing a report of their own experimental research. Sixteen classes were randomly assigned to one of four treatment conditions: (1) no in-program evaluation, (2) self-evaluation and revision at the research design and draft final report stages, (3) teacher evaluation and student revision at both stages, (4) self-plus-teacher evaluation and student revision at both stages. Students in the teacher-evaluation and self-plus-teacher evaluation conditions received significantly higher ratings from an independent rater on their final research reports. However, students under the self-evaluation conditions had greater confidence in their ability to conduct future experiments. This research is based on Dr. Olina's dissertation at Arizona State University, and was supported by grants from the Arizona State University Graduate Research Support Program and the Assessment Training Institute Foundaton in Portland, Oregon. We gratefully acknowledge their support.     

教师自我评价的多维透视

教师自我评价是教师评价的重要组成部分,无论从教师的职业特点还是从心理学、教育学理论等角度分析,教师自我评价都是合情合理的,对提高教师素质、提高教学质量起着重要作用。教师在进行自我评价时,应根据不同阶段的需求评价不同的内容,并采用多种评价方法,客观、合理的对自己进行评价,从而促进教师个人价值的实现,推动教师专业发展。