公开考试评卷工作的重要程序——香港的经验

公开考试的设计是一项专门的学问。公开考试一般包括三个重要元素：设计考试大纲、编制试题及评分、报告成绩。在考生人数众多的科目,评卷环节面临的重要问题之一就是须聘用大量评卷员参与评卷。但是,评卷是一项以专业判断为主的工作,为保证对所有考生公平,考评机构便须采取适当步骤,划一评卷尺度。本文旨在探讨香港考试及评核局为保持评卷信度,确保考生的答卷得到公平及客观的处理,在评卷环节所采取的程序,包括评卷参考的设计、评卷的配套程序、评卷教师的培训等,并分析这些程序的重要意义。     

But is it fair? Developing students’ understanding of grading complex written work through peer assessment

Peer assessment of long written tasks poses particular problems as these tasks typically involve complex learning and solving ill‐structured problems which require divergent responses. Marking reliability of this kind of writing task is difficult to achieve. The author illustrates this through an evaluation of two implementations of peer assessment, involving 81 students, in a UK university. In these implementations, all peer assessor grades were returned to students (not just mean grades). In this way students were exposed to subjectivity in marking. The implementations were evaluated through questionnaires, focus groups, observations of lectures and tutor interview. While students reported a better understanding of quality in student writing as a result of their experience, many complained that peer assessors’ marks were not ‘fair’. The article draws on recent research on the reliability of tutor marking to argue that marking judgements are subjective and that peer assessment offers the opportunity to explore subjectivity in marking, creating an opportunity for dialogue between tutors and students.     

数学作业批改模式初探

让学生参与作业批改，让教师从繁重的作业堆中获得解放，这是对传统数学作业批改的一次挑战．在新的形势下，对数学作业批改模式的探讨和实践，值得广大数学教师在教学实践中作认真思考和大胆的尝试．     

A Study in Self‐assessment: tutor and students’ perceptions of performance criteria

This paper reports a study which implemented and evaluated a method of student self‐assessment. The theme of the importance of marking criteria (Orsmond et al., 1996) is developed. Pairs of first‐year undergraduate biology students were asked to complete a poster assignment on a specific aspect of nerve physiology. The study was designed to allow the evaluation of (1) student self and tutor marking for individual marking criteria; and (2) student vs student marking of their poster work for individual marking criteria. A comparison between the tutor and the student self‐assessed mark reveals how important it is to consider the individual marking criteria rather than the overall mark. These results support our findings on peer assessment reported earlier (Orsmond et al., 1996). The results also illustrate the potential pitfalls that exist when making assumptions about the degree of student/student interaction and students’ marking ability. This study supports previous work which showed that self‐assessment is extremely useful in helping students reach their learning goals, i.e. it is a strong formative educational tool and can be used in order to bring about behavioural changes in students with regard to their own learning processes.     

From copying to learning: using exemplars to engage students with assessment criteria and feedback

Feedback is central to pedagogic theory, and if feedback is to be effective, students need to engage with it and apply it at some point in the future. However, student dissatisfaction with feedback – as evidenced in the National Student Survey – suggests that there are problems which limit student engagement with feedback, such as their perception that much of their feedback is irrelevant to future assignments. This article reports on a study which sought to enhance engagement by giving students exemplar assignments annotated with feedback before submission of their final assignments. This was done by providing an online facility where students could view exemplars and post comments or questions to tutors and peers on a discussion board. The exemplar facility was highly valued by students, although there were no quantitative effects such as an increase in students’ assignment marks when compared with the previous cohort. The article reflects on possible reasons for this result and discusses ways to improve the exemplar facility, for example by facilitating dialogue between tutors and students. The article concludes with lessons learned about how to construct exemplars, and considers how exemplars might also be used within marking teams to improve consistency of marking.     

The Importance of Marking Criteria in the Use of Peer Assessment

This paper reports a study which implemented and evaluated a method of peer assessment, as a formative and summative assessment procedure. Pairs of first‐year undergraduate biology students were asked to complete a poster assignment on a specific aspect of nerve physiology. This paper contains details of a method which allows student peer and tutor marking of work against the individual marking criteria to be evaluated. The results show that a comparison between the tutor and the student peer mark may be misleading as a guide to the validity of peer assessment. The importance of considering the individual sections of the marking criteria is illustrated. It was found that when the individual criteria were analysed the number of students marking the same as the tutor ranged from 31% to 62%. It also became clear that specific areas of the marking criteria were prone to over and under‐marking. Analysis of student feedback forms showed that students not only liked carrying out peer assessment, but felt the benefits in terms of developing facets of their learning process and heightening their awareness of their work. These results are discussed in the light of other studies.     

The Use of Student Derived Marking Criteria in Peer and Self-assessment

This paper reports a study which implemented and evaluated a method of student self and peer assessment involving student constructed marking criteria. The theme of the importance of marking criteria (Orsmond et al ., 1996, 1997) is developed. Pairs of first-year undergraduate biology students were asked to complete a poster assignment. The study was designed to allow the comparison and evaluation of (1) student self and tutor marking for individual marking criteria; (2) student self and peer marking for individual marking criteria; and (3) student and tutor marking for student constructed and tutor provided individual marking criteria. The present study shows that: (a) students may be less able to discriminate between individual marking criteria which they have constructed compared to marking criteria that have been provided; (b) asking students to construct their own marking criteria in discussion with tutors or fellow students does not enhance agreement between student/tutor or student/student marking; and (c) allowing students to construct their own marking criteria may lead to different learning outcomes compared to providing students with a marking criteria.     

Clarity,consistency and communication: using enhanced dialogue to create a course-based feedback strategy

This article examines the outcomes of a study across four discipline areas in order to develop course-based assessment strategies in closer cooperation with students. Second-year students (n = 48) from different disciplines were engaged in two phases of activity-orientated workshops. Phase 1 sought their perceptions of feedback. Phase 2 saw students design a proposed strategy to present to the respective staff teams. We discuss the emerging themes which appeared to be very similar amongst this diverse cross section of students: a lack of faith in marking consistency; the need for clear guidelines and criteria; the greater use of positive feedback language; and a close association with tutors. The emergence of strategies specific to each course is discussed, along with the alignment of the outcomes of this approach with pedagogic knowledge. It is suggested that enhanced dialogue enabled staff and students to develop a common understanding, and gave impetus to improving assessment feedback practices. Outcomes recommended here include changes to practice such as a team approach to feedback development, the content and style of feedback, developing the usefulness of feedback for future work and the need for teams to periodically revisit staff development in this area.     

Curtailing marking variation and enhancing feedback in large scale undergraduate chemistry courses through reducing academic judgement: a case study

Variation in marks awarded, alongside quality of feedback, is an issue whenever large-scale assessment is undertaken. In particular, variation between sessional teaching staff has been studied for decades resulting in many recorded efforts to overcome this issue. Attempts to curtail variation range from moderation meetings, extended training programmes, electronic tools, automated feedback or even audio/video feedback. Decreased marking variation was observed whenever automated marking was used, potentially due to less academic judgment being used by the markers. This article will focus on a case study of three interventions undertaken at Monash University that were designed to address concerns around the variability of marking and the feedback between sessional teaching staff employed in the chemistry teaching laboratories. The interventions included the use of detailed marking criteria, Excel marking spreadsheets and automated marked Moodle reports. Results indicated that more detailed marking criteria had no effect whilst automated processes caused a consistent decrease. This was attributed to a decrease in the academic judgment markers were expected to use. Only the Excel spreadsheet ensured the provision of consistent feedback to students. Sessional teaching staff commented that their marking loads were reduced and the new methods were easy to use.     

Science Teachers’ Use of a Concept Map Marking Guide as a Formative Assessment Tool for the Concept of Energy

In this study, we investigated the value of a concept map marking guide as an alternative formative assessment tool for science teachers to adopt for the topic of energy. Eight high school science teachers marked students’ concept maps using an itemized holistic marking guide. Their marking was compared with the researchers’ marking and the scores of a multiple-choice diagnostic test. Statistical analysis revealed that teachers evaluated students’ concept maps consistently, but the concept map scores were only mildly correlated with the multiple-choice test scores. We explored what each assessment method revealed in terms of students’ understanding with one student’s performance as a showcase. Teachers’ views on using the marking guide of concept maps as an assessment tool were also included. We discuss the value of the concept map marking guide as a formative assessment tool for science teachers to create and modify in order to encourage students’ conceptual learning.     

Quantitative analysis of approaches to group marking

Abstract

Group work, where students work on projects to overcome challenges together, has numerous advantages, including learning of important transferable skills, better learning experience and increased motivation. However, in many academic systems the advantages of group projects clash with the need to assign individualised marks to students. A number of different schemes have been proposed to individualise group project marks, these include marking of individual reflexive accounts of the group work and peer assessment. Here, we explore a number of these schemes in computational experiments with an artificial student population. Our analysis highlights the advantages and disadvantages of each scheme and particularly reveals the power of a new scheme proposed here that we call pseudoinverse marking.

Abbreviations

SOPP: Self organised peer assessment; RA: Reflexive accounts; MRA: Mark-adjusted reflexive accounts; NPA: Normalised peer assessment; PR: Peer ranking; PiM: Pseudoinverse marking     

The method of Adaptive Comparative Judgement

Adaptive Comparative Judgement (ACJ) is a modification of Thurstone’s method of comparative judgement that exploits the power of adaptivity, but in scoring rather than testing. Professional judgement by teachers replaces the marking of tests; a judge is asked to compare the work of two students and simply to decide which of them is the better. From many such comparisons a measurement scale is created showing the relative quality of students’ work; this can then be referenced in familiar ways to generate test results. The judges are asked only to make a valid decision about quality, yet ACJ achieves extremely high levels of reliability, often considerably higher than practicable operational marking can achieve. It therefore offers a radical alternative to the pursuit of reliability through detailed marking schemes. ACJ is clearly appropriate for performances like writing or art, and for complex portfolios or reports, but may be useful in other contexts too. ACJ offers a new way to involve all teachers in summative as well as formative assessment. The model provides strong statistical control to ensure quality assessment for individual students. This paper describes the theoretical basis of ACJ, and illustrates it with outcomes from some of our trials.     

The Use of Exemplars and Formative Feedback when Using Student Derived Marking Criteria in Peer and Self-assessment

This paper reports a study which implemented a method of student self and peer assessment involving student constructed marking criteria in the presence of exemplars. Pairs of first-year undergraduate biology students were asked to complete a poster assignment. The study was designed to allow the evaluation of (1) student self and peer marking and tutor marking for individual marking criteria following the use of exemplars; (2) the role of exemplars in providing a focus for formative feedback about subject standards. The present study shows that: (a) the use of exemplars can help students demonstrate greater understanding of both marking criteria and subject standards; (b) the use of exemplars can help students learning so that higher quality outcomes are produced; (c) the use of exemplars forms a focus for meaningful formative feedback; (d) students may make more objective judgements as a result of peer assessment compared to self-assessment.     

Honoring and Suffering the Thorn: Marking,Naming, and Eldering Depth Psychology II

This is the second article advocating and explicating a depth psychological approach to the education of the talented. In this analysis we take the “thorn” on the Piirto Pyramid of Talent Development and relate it to the processes of marking, naming, and eldering/mentoring as teachers practice them. The article presents 20 ways that the imaginal, symbolic aspects of life can be acknowledged and brought forth in students, thus invoking the educare within education.     

A design science research methodology for developing a computer-aided assessment approach using method marking concept

Assessment has been defined as an authentic method that plays an important role in evaluating students’ learning attitude in acquiring lifelong knowledge. Traditional methods of assessment including the Computer-Aided Assessment (CAA) for mathematics show limited ability to assess students’ full work unless multi-step questions are sub-divided into sub questions. This issue persisted significant drawback especially within the notion of method marking approach. To address this issue, the aim of the study is to develop a methodological framework that will create an information and communications technology (ICT) artefact prototype. The prototype (termed as method marking assessment (MMA) artefact) implements a method-marking assessment concept to assess through multi-step questions. Extensive literature reviews have revealed that there are features in common between complex-problem solution characteristics and multi-steps questions assessment using ICT; therefore complex problems paradigm is used in the study for developing the MMA prototype.     

Quality and Qualms in the Marking of University Assignments by Sessional Staff: An Exploratory Study

The higher education sector is increasingly reliant upon casual (‘sessional’) staff for teaching and marking purposes. While this practice has been little examined in the past, over the last few years increasing attention has been paid to the quality of marking, mainly because students and academic staff alike are becoming increasingly likely to question examples of poor practice. Hence, many universities in Australia are now attempting to introduce stricter procedures relating to marking. Despite current concerns, there is little published research on marking practices in Australian universities. This paper adds to the body of knowledge by reporting on two pieces of empirical research into the use of casual markers. A project at Charles Sturt University comprised focus groups of, respectively, students, lecturers and markers, and a survey of distance education students. Research at the University of South Australia focused on pedagogical issues relating to marking, comparing the approaches of permanent lecturing staff with those of sessional markers. The results of these projects provide a useful insight into areas of current concern to university staff and management.     

Investigating the relationship between quality,format and delivery of feedback for written assignments in higher education

Feedback can have a great impact on student learning. However, in order for it to be effective, feedback needs to be of high quality. Electronic marking has been one of the latest adaptations of technology in teaching and offers a new format of delivering feedback. There is little research investigating the impact the format of feedback has on quality of feedback and subsequently on student learning. This study sets out to investigate the impact paper-based and electronic methods of assignment submission and return have on students’ and markers’ perceived quality of feedback. Students and markers on an undergraduate course were asked to complete an anonymous online survey investigating their perceptions of quality, format and timeliness of feedback delivered electronically and on paper. The results showed that marking and providing feedback electronically was an acceptable method for markers, reporting improved speed and consistency of marking. There was no increase or decrease in satisfaction with the feedback received. Overall, electronic marking was found to be an acceptable method of delivery of feedback on written assignments by both students and markers. The findings of this study suggest that electronic marking can result in more timely feedback for students without impacting on quality.