Interpretations of criteria‐based assessment and grading in higher education

The increasing use of criteria‐based approaches to assessment and grading in higher education is a consequence of its sound theoretical rationale and its educational effectiveness. This article is based on a review of the most common grading policies that purport to be criteria‐based. The analysis shows that there is no common understanding of what criteria‐based means or what it implies for practice. This has inhibited high‐quality discourse, research and development among scholars and practitioners. Additionally, the concepts of ‘criteria’ and ‘standards’ are often confused and, despite the use of criteria, the fundamental judgments teachers make about the quality of student work remain subjective and substantially hidden from the students’ view. As they stand, none of the approaches identified in the survey is fully capable of delivering on the aspirations of criteria‐based grading. Shifting the primary focus to standards and making criteria secondary could, however, lead to substantial progress.     

Student responses to criteria‐referenced self‐assessment

This paper reports on a study of undergraduate students’ experiences with criteria‐referenced self‐assessment. Fourteen students who had taken a course involving self‐assessment were interviewed in focus groups segregated by gender. The findings suggest that students had positive attitudes toward self‐assessment after extended practice; felt they can effectively self‐assess when they know their teacher’s expectations; claimed to use self‐assessment to check their work and guide revision; and believed the benefits of self‐assessment include improvements in grades, quality of work, motivation and learning. There were indications that some students sensed a tension between their own standards for good work and some of their teachers’ standards. There was no evidence of differences in the responses of male and female students. The paper concludes with the suggestion that self‐assessment involves a complex process of internalization and self‐regulation, and with implications for research and practice.     

The development of a criterion‐referenced profile: the Hertfordshire English Project

Many developments in profiling are taking place at present and a considerable number use a criterion‐referenced assessment model. This study reports the experiences of a local curriculum development project which has developed an English course and summative profile for lower‐attaining pupils. The way in which criterion‐referenced assessment was incorporated into the development of the profile is described, together with the results of a small‐scale survey of teachers’ views of the use of such a profile. Key issues for those developing such profiles in English and other subjects are identified.     

Building and Supporting a Validity Argument for a Standards‐Based Classroom Assessment of English Proficiency Based on Teacher Judgments

Using an argument‐based approach to validation, this study examines the quality of teacher judgments in the context of a standards‐based classroom assessment of English proficiency. Using Bachman's (2005) assessment use argument (AUA) as a framework for the investigation, this paper first articulates the claims, warrants, rebuttals, and backing needed to justify the link between teachers' scores on the English Language Development (ELD) Classroom Assessment and the interpretations made about students' language ability. Then the paper summarizes the findings of two studies—one quantitative and one qualitative—conducted to gather the necessary backing to support the warrants and, in particular, address the rebuttals about teacher judgments in the argument. The quantitative study examined the assessment in relation to another measure of the same ability—the California English Language Development Test—using confirmatory factor analysis of multitrait‐multimethod data and provided evidence in support of the warrant that states that the ELD Classroom Assessment measures English proficiency as defined by the California ELD Standards. The qualitative study examined the processes teachers engaged in while scoring the classroom assessment using verbal protocol analysis. The findings of this study serve to support the rebuttals in the validity argument that state that there are inconsistencies in teachers' scoring. The paper concludes by providing an explanation for these seemingly contradictory findings using the AUA as a framework and discusses the implications of the findings for the use of standards‐based classroom assessments based on teacher judgments.     

Constructive guidance and feedback for learning: the usefulness of exemplars,marking sheets and different types of feedback in a first year law subject

Criteria and standards‐based assessment models are increasingly being adopted by universities as effective practice. However the promise of these models of assessment may not be realised unless teachers can find ways of making criteria and standards understandable to students. Exemplars or examples of previous students’ work of high and low quality can make criteria and standards concrete. Recent research has focussed on the use of exemplars to help students understand criteria and standards, and less emphasis has been given to exemplars simply as guides for students. This mixed methods study explores students’ perceptions of the usefulness of exemplars and different types of feedback for guiding them in completing assessments. A combination of engaging in marking and discussing exemplars, and receiving individualised and standards‐based feedback provides the most helpful guidance for students’ effective learning.     

THE STUDENT ASSESSMENT PROJECT

Under a grant from Education Research and Development Committee, researchers at Macquarie University, Sydney, Australia, developed a set of instructional materials aimed at the inservice education of teachers on the topic of student assessment. The Student Assessment Project (SAP) now comprises seven modules in slide‐tape format covering the topics of test design, item writing, analysis of norm‐referenced and criterion‐referenced tests, combining scores from different components, moderation of test results, and grading and reporting. The kit also contains appropriate computer software (for an Apple II microcomputer), manuals and supplementary materials. This article gives some details of the project and its development and describes the widespread use of the first 20 kits from which evaluation data are being sought. Although SAP originally focussed on inservice education of secondary teachers, the present applicability to higher education and the further potential for modification and use at this level is discussed.     

Assessment of mathematics and reading performance: An examination of the correspondence between direct assessment of student performance and teacher report

Teacher judgments have been identified as a primary source of information regarding student academic achievement. Research examining the accuracy of teachers' judgments in assessing students' academic abilities has shown relatively high accuracy. However, previous studies have relied primarily on norm‐referenced measures to obtain estimates of students' achievement in reading and mathematics. Recent developments in the assessment of students' academic skills, such as Curriculum‐Based Measurement (CBM), provide a direct estimate of students' skill levels in basic areas such as reading and mathematics. The purpose of the present study was to examine the extent to which teachers' perceptions of students' reading and mathematics skills corresponded to direct estimates of students' reading and mathematics skills. Two second‐grade teachers estimated the reading and mathematics skills of 33 second‐grade students. Results of this study indicated that teachers were not accurate in assessing their students' mathematics functioning. Teachers were more accurate in assessing the occurrence of Mastery mathematics levels in basic addition, but were very inaccurate in assessing the occurrence of Mastery, Instructional, or Frustrational mathematics levels in all other skills assessed. In reading, teachers' judgment accuracy varied as a function of grade‐level material and instructional level. Specifically, teachers experienced considerable difficulty accurately identifying students who were reading at a Mastery level in grade‐level or above‐grade‐level material. © 2006 Wiley Periodicals, Inc. Psychol Schs 43: 247–265, 2006.     

The Common Core State Standards and Reading: Interpretations and Implications for Elementary Students with Learning Disabilities

The K‐5 reading standards within the English Language Arts Common Core State Standards provide guidance to teachers about grade level expectations for students. Though the authors of the standards acknowledge that some students may experience difficulty reaching the rigorous expectations, they explain that the standards outline a pathway to proficiency for all students, including those who struggle with literacy. Students with learning disabilities, who often have significant literacy difficulties, may face particular challenges when their instruction is framed by these standards. This article unpacks the complex K‐5 reading standards and provides a discussion of the implications for students with learning disabilities and their general and special education teachers. Examples from K‐5 lessons and recommendations for teachers and researchers are provided.     

Assessment as Judgment-in-Context: Analysing how teachers evaluate students' writing 1

In this paper, we analyse teachers' judgments of students' written texts. We document how teachers use evidence in ways that depend both on their knowledge of the students and on the assessment framework they need to use. We analyse teachers' judgments by contrasting the structures of assessments made using teachers' normal classroom judgment processes with those made using an external set of “benchmark” standards. We show how the tension between demands for system-wide assessment validity and localised contextually sensitive site validity impacts on the richness and consistency of the judgment processes. We conclude that current understandings of teacher judgment processes that operate in everyday assessment practices generally fail to account for the complexity and dynamism of this routine classroom activity. Furthermore, we demonstrate that the methodology of judgment analysis, combined with think-aloud protocols, has the potential to shed light on the complexities associated with the operation of judgment in educational assessment.     

Supporting Student Teachers' Professional Learning with Standards‐Referenced Assessment

Professional standards in teaching are developed in many education systems, with professional learning and quality assurance being the central purposes of these standards. This paper presents an initiative in developing a professional development progress map (hereafter, progress map) within a learning‐oriented field experience assessment (LOFEA) framework. The article examines the use of a progress map to support professional learning in teaching supervision in the field experience of a teacher education programme. Views of users, including 16 tertiary supervisors and 21 teacher participants of the in‐service programmes, were collected. Issues relating to supporting student teachers' professional learning with standards‐referenced assessment, are discussed around four themes, namely intention, instrumentation, interpretation and implementation.     

Educating Student Teachers to Use Small Groups in Secondary‐school Music Classrooms

The use of small groups in teaching classroom music is often accepted in the literature and by expert teachers as a natural part of a music‐education programme. However, research indicates that the classroom teacher commonly does not make use of small‐group activities and that, when small groups are used, the results are not always positive. This article describes a teacher‐education course aimed at developing student teachers’ abilities to manage small groups in the secondary‐school music classroom, and reports the perceptions of both student teachers and teacher‐observers when the students applied their learnings in the practicum setting.     

英语教育活动中档案袋评价应用分析

在英语教育活动中,现行的以成绩为主的终结性评价标准与体系已无法满足对师生以及教育活动效果进行评价的需要,而基于多元智能理论创建的档案袋评价则是一种可行性的解决方案。档案袋评价是教育评价体系中非常有效的一种质性评价方式,能够针对当前评价模式的弊端,促进师生角色转变,增加过程性评价,激发英语学习者的学习动机,丰富英语教育评价手段,真正发挥教育评价的积极作用。     

SETTING OBJECTIVES AND ASSESSING COMPETENCE IN PROFESSIONAL LEGAL EDUCATION

Professional Legal Education in the Australian Capital Territory (ACT) is undertaken at the Legal Workshop. A problem‐based learning orientation is a prominent feature of their programme. In this paper the assessment of the problem‐based learning used in the Workshop is described. All stages of this assessment process, including setting the course objectives, defining the conditions under which the problems (tasks) will be undertaken, and setting the criteria which are the standards for the assessment of success in the task, are described. The importance of criterion‐referenced assessment in the stage of setting assessment standards is emphasised. The results of the first attempt to use this assessment method are then described, and problems emerging with the process are discussed. Although this first “reality test” reveals some difficulties with the assessment method, it is concluded that this novel and innovative teaching and assessment technique has been appropriately applied to Professional Legal Education. While some refinements are required in the implementation of the method, the overall assessment concept has been successfully developed.     

Moderation and consistency of teacher judgement: Teachers’ views

Major curriculum and assessment reforms in Australia have generated research interest in issues related to standards, teacher judgement and moderation. This article is based on one related inquiry of a large‐scale Australian Research Council Linkage project conducted in Queensland. This qualitative study analysed interview data to identify teachers’ views on standards and moderation as a means to achieving consistency of teacher judgement. A complementary aspect of the research involved a blind review that was conducted to determine the degree of teacher consistency without the experience of moderation. Empirical evidence was gained that most teachers, of the total interviewed, articulated a positive attitude towards the use of standards in moderation and perceived that this process produces consistency in teachers’ judgements. Context was identified as an important influential factor in teachers’ judgements and it was concluded that teachers’ assessment beliefs, attitudes and practices impact on their perceptions of the value of moderation practice and the extent to which consistency can be achieved.     

Common Core State Standards,Writing, and Students with LD: Recommendations

This article examines the Common Core State Standards as they apply to writing and students with learning disabilities (LD). We first consider why the implementation of these standards is advantageous to writing instruction for students with LD as well as the challenges in implementing them. Next, we make the following four recommendations in terms of their implementation: (1) increase general and special education teachers’ knowledge about writing development; (2) create a writing environment in which students with LD can thrive; (3) employ evidence‐based writing practices in general education classes (where most students with LD are taught); and (4) use evidence‐based writing practices effective with students with LD. We conclude by considering research that still needs to be undertaken to help educators maximize the probability that students with and without LD meet the writing benchmarks proposed in these Standards.     

Meanings and Consequences in Standard Setting

Standard setting is defined as the identification of certain points on a mark scale with particular performance standards, with the intention of enhancing the inferences that are warranted from the test scores. It is argued that the selection of both the points on the mark‐scales and the performance standards with which they are equated are arbitrary and are driven by a set of values (which are often implicit). In ‘high‐stakes’ settings, it is shown how the values implicit in the standard can come to dominate the values inherent in the domain they represent. The validation of standards must therefore include consideration of their consequences as well as their meanings. It is then argued that standards, where they exist, cannot be accounted for purely in terms of norm‐referenced or criterion‐referenced interpretations, but exist rather by virtue of a shared construct in a community of practice. These theoretical positions are then developed to classify standard‐setting methods along two dimensions, the first relating to the role of performance data in the setting of standards and the second relating to the extent to which the meanings or the consequences of the assessment are emphasised in the process.     

Indeterminacy in the use of preset criteria for assessment and grading

When assessment tasks are set for students in universities and colleges, a common practice is to advise them of the criteria that will be used for grading their responses. Various schemes for using multiple criteria have been widely advocated in the literature. Each scheme is designed to offer clear benefits for students. Breaking down holistic judgments into more manageable parts is seen as a way to increase openness for students and achieve more objectivity in grading. However, such approaches do not adequately represent the full complexity of multi‐criterion qualitative judgments, and can lead to distorted grading decisions. Six anomalies in the ways assessors approach the grading task are identified, together with several likely contributing factors. Overall, the conclusion is that explicit grading models do not have as strong a theoretical foundation as is commonly supposed, and that holistic appraisal merits further investigation.