Aiming for autonomy: formative peer assessment in a final-year undergraduate course

This article takes a dual focus on the theme of student formative peer assessment. On the one hand it offers a thorough literature review in this field, while on the other it unpacks a case study of curriculum design where peer assessment has been adopted. The practical example draws on recent changes made to a third- and final-year undergraduate research dissertation course in a UK architectural school. Although peer assessment worked quite well in this small setting, similar findings might be uncovered when scaled up to larger cohorts, as well as other disciplines and year groups. The research findings are certainly relevant for other contexts. Qualitative research from a focus group with six students on the course informs the empirical body of the paper. Through engaging with students’ reflections, as well as the relevant literature, and reflexively looking at the curriculum changes, this paper discusses some of the benefits and drawbacks that can arise from peer assessment in this context. Peer assessment is not without its challenges and hiccups, but, despite these, the benefit to autonomous critical thinking cannot be understated. And arguably peer assessment is especially germane for final-year undergraduates as they embark on employment or more advanced studies.     

A longitudinal quantitative investigation into the concurrent validity of self and peer assessment applied to English-Chinese bi-directional interpretation in an undergraduate interpreting course

One of the productive lines of research on self-assessment (SA) and peer assessment (PA) concerns their concurrent validity with respect to a criterion measure. However, similar research has rarely been conducted for spoken-language interpreting. This article therefore reports on a longitudinal study that investigated the validity of self and peer ratings on three performance dimensions of English-Chinese consecutive interpretation (i.e., information completeness, fluency of delivery, and target language quality), taking teachers’ ratings as a yardstick. Major findings include: although the students as a group were unable to replicate teachers’ ratings, they were able to rank-order their performances in a fairly accurate manner and improved their SA and PA accuracy over time. Interpreting directionality seems to moderate the correlational strength of self/teacher ratings and peer/teacher ratings. These results are discussed in relation to previous literature, and pedagogical suggestions are provided to improve SA and PA for bi-directional interpretation.     

The assessment cycle: a model for learning through peer assessment

This paper advances a model describing how peer assessment supports self-assessment. Although prior research demonstrates that peer assessment promotes self-assessment, the connection between these two activities is underspecified. This model, the assessment cycle, draws from theories of self-assessment to elaborate how learning takes place through peer assessment. The model is applied to three activity structures described in the literature to analyse their potential to support learning by promoting self-assessment. Broadly speaking, the model can be used to understand learning that takes place in a variety of peer assessment activities: marking/grading, analysis, feedback, conferencing and revision. This approach contrasts most studies on peer assessment, which have focused on calibration of instructor and peer grades, rather than learning opportunities.     

Using scaffolded rubrics to improve peer assessment in a MOOC writing course

This study explored the value of using a guided rubric to enable students participating in a massive open online course in writing to produce more reliable assessments of their fellow students’ writing. To test the assumption that training students to assess will improve their ability to provide quality feedback, a multivariate factorial analysis was used to determine differences in assessments made by students who received guidance on using a rating rubric and those who did not. Although results were mixed, on average students who were provided no guidance in scoring writing samples were less likely to successfully differentiate between novice, intermediate, and advanced writing samples than students who received rubric guidance. Rubric guidance was most beneficial for items that were subjective, technically complex, and likely to be unfamiliar to the student. Items addressing relatively simple and objective constructs were less likely to be improved by rubric guidance.     

Uncertainty analysis for peer assessment: oral presentation skills for final year project

Peer assessment plays an important role in engineering education for an active involvement in the assessment process, developing autonomy, enhancing reflection, and understanding of how to achieve the learning outcomes. Peer assessment uncertainty for oral presentation skills as part of the FYP assessment is studied. Validity and reliability for complex assessments were conceptualised in the context of quantification, hierarchical system of uncertainties, aleatory uncertainty, and epistemic uncertainty. Systems as assessment methods were also conceptualised to define elements (i.e. examiner, examinee, and difficulty) as variables or/and constants, and to study the uncertainty behaviour of the peer assessment within the framework of the parametric uncertainty model. As a result, a singularity, at which the hierarchical structure of uncertainties breaks down, and its related behaviour were theoretically predicted and experimentally verified. Also, a validity paradox, when the ‘difficulty’ level is sufficiently lowered, was found to be possible and verified for oral presentation skills. It may be useful for understanding fundamental characteristics of the peer assessment in which epistemic uncertainties are dependent on how the assessment is close to the singularity point.     

Peer assessment in thesis oral presentation

Peer assessment has been studied in various situations and actively pursued as a means by which students are given more control over their learning and assessment achievement. This study investigated the reliability of staff and student assessments in two oral presentations with limited feedback for a school-based thesis course in engineering where the different disciplines ran their own assessment. Staff scores were generally found to be more reliable than student scores, but it was not consistent. The engineering disciplines displayed widely varying differences between the staff and student scores. A large variation in reliability was found for the different disciplines making it difficult to reconcile to a standard school-based overall grading scheme for the course when the reliability of marking was low. The results also showed that the average scores for oral presentations are generally much higher than for written examinations, based on the grade point average. Future research will need to investigate how best to develop a consistent assessment framework for oral presentations and moderation of scores for consistency, particularly for large classes where assessment are done by different groups of staff and students and collated into a single set of marks for the class.     

Students' perceptions of fairness in peer assessment: evidence from a problem-based learning course

This paper describes the experience of peer assessment in a problem-based learning course in a Portuguese university, and presents the results of a survey of 120 students who attended the course over seven semesters. Cluster analysis revealed that students group into two distinct clusters with significantly different experiences of peer assessment. Although the larger cluster (70%) expressed a positive experience, the other group perceived peer assessment results and final marks to be unfair, and reported incidents of friendship-marking and conflict arising from peer assessment. Results show that perceptions of fairness (both regarding peer assessment and final marks) are very much associated with those problems. We reflect on these findings to explore implications for practice as well as future research.     

An empirical review of anonymity effects in peer assessment,peer feedback,peer review,peer evaluation and peer grading

Abstract

Peer assessment has proven to have positive learning outcomes. Importantly, peer assessment is a social process and some claim that the use of anonymity might have advantages. However, the findings have not always been in the same direction. Our aims were: (a) to review the effects of using anonymity in peer assessment on performance, peer feedback content, peer grading accuracy, social effects and students’ perspective on peer assessment; and (b) to investigate the effects of four moderating variables (educational level, peer grading, assessment aids, direction of anonymity) in relation to anonymity. A literature search was conducted including five different terms related to peer assessment (e.g., peer feedback) and anonymity. Fourteen studies that used a control group or a within group design were found. The narrative review revealed that anonymous peer assessment seems to provide advantages for students’ perceptions about the learning value of peer assessment, delivering more critical peer feedback, increased self-perceived social effects, a slight tendency for more performance, especially in higher education and with less peer assessment aids. Some conclusions are that: (a) when implementing anonymity in peer assessment the instructional context and goals need to be considered, (b) existent empirical research is still limited, and (c) future research should employ stronger and more complex research designs.     

Assessment training effects on student assessment skills and task performance in a technology-facilitated peer assessment

This study examines the impact of an assessment training module on student assessment skills and task performance in a technology-facilitated peer assessment. Seventy-eight undergraduate students participated in the study. The participants completed an assessment training exercise, prior to engaging in peer-assessment activities. During the training, students reviewed learning concepts, discussed marking criteria, graded example projects and compared their evaluations with the instructor’s evaluation. Data were collected in the form of initial and final versions of students’ projects, students’ scoring of example projects before and after the assessment training, and written feedback that students provided on peer projects. Results of data analysis indicate that the assessment training led to a significant decrease in the discrepancy between student ratings and instructor rating of example projects. In addition, the degree of student vs. instructor discrepancy was highly predictive of the quality of feedback that students provided to their peers and the effectiveness of revisions that they made to their own projects upon receiving peer feedback. Smaller discrepancies in ratings were associated with provision of higher quality peer feedback during peer assessment, as well as better revision of initial projects after peer assessment.     

Latent trait modelling of rater accuracy in formative peer assessment of English-Chinese consecutive interpreting

Despite the increasing popularity of peer assessment in tertiary-level interpreter education, very little research has been conducted to examine the quality of peer ratings on language interpretation. While previous research on the quality of peer ratings, particularly rating accuracy, mainly relies on correlation and analysis of variance, latent trait modelling emerges as a useful approach to investigate rating accuracy in rater-mediated performance assessment. The present study demonstrates the use of multifaceted Rasch partial credit modelling to explore the accuracy of peer ratings on English-Chinese consecutive interpretation. The analysis shows that there was a relatively wide spread of rater accuracy estimates and that statistically significant differences were found between peer raters regarding rating accuracy. Additionally, it was easier for peer raters to assess some students accurately than others, to peer-assess target language quality accurately than the other rating domains, and to provide accurate ratings to English-to-Chinese interpretation than the other direction. Through these findings, latent trait modelling demonstrates its capability to produce individual-level indices, measure rater accuracy directly, and accommodate sparse data rating designs. It is therefore hoped that substantive inquiries into peer assessment of language interpretation could utilise latent trait modelling to move this line of research forward.     

Quantification for complex assessment: uncertainty estimation in final year project thesis assessment

A quantitative method for estimating an expected uncertainty (reliability and validity) in assessment results arising from the relativity between four variables, viz examiner's expertise, examinee's expertise achieved, assessment task difficulty and examinee's performance, was developed for the complex assessment applicable to final year project thesis assessment including peer assessment. A guide map can be generated by the method for finding expected uncertainties prior to the assessment implementation with a given set of variables. It employs a scale for visualisation of expertise levels, derivation of which is based on quantified clarities of mental images for levels of the examiner's expertise and the examinee's expertise achieved. To identify the relevant expertise areas that depend on the complexity in assessment format, a graphical continuum model was developed. The continuum model consists of assessment task, assessment standards and criterion for the transition towards the complex assessment owing to the relativity between implicitness and explicitness and is capable of identifying areas of expertise required for scale development.     

Peer assessment in the digital age: a meta-analysis comparing peer and teacher ratings

Given the wide use of peer assessment, especially in higher education, the relative accuracy of peer ratings compared to teacher ratings is a major concern for both educators and researchers. This concern has grown with the increase of peer assessment in digital platforms. In this meta-analysis, using a variance-known hierarchical linear modelling approach, we synthesise findings from studies on peer assessment since 1999 when computer-assisted peer assessment started to proliferate. The estimated average Pearson correlation between peer and teacher ratings is found to be .63, which is moderately strong. This correlation is significantly higher when: (a) the peer assessment is paper-based rather than computer-assisted; (b) the subject area is not medical/clinical; (c) the course is graduate level rather than undergraduate or K-12; (d) individual work instead of group work is assessed; (e) the assessors and assessees are matched at random; (f) the peer assessment is voluntary instead of compulsory; (g) the peer assessment is non-anonymous; (h) peer raters provide both scores and qualitative comments instead of only scores; and (i) peer raters are involved in developing the rating criteria. The findings are expected to inform practitioners regarding peer assessment practices that are more likely to exhibit better agreement with teacher assessment.     

The quality of peer assessment in a wiki-based online context: a qualitative study

Peer assessment (PA) provides opportunities for authentic assessment, autonomy and collaboration. Several authors advocate that students can benefit from PA and put forward the effects of PA on the students’ learning outcomes. Questions concerning the validity and reliability of PA and PA competences are also addressed by different researchers. This qualitative study is part of a wider project that seeks to develop and test evaluation and assessment strategies in online contexts. In a doctoral module, PA was used for summative and formative purposes. Formative PA aimed to give feedback about the ongoing group work, but also to increase online interaction between the different groups of students. The main module task was to write a literature review, about a selected topic, using a wiki. Criteria and indicators to assess the literature review were negotiated with the students. Different criteria were used to assess the quality of PA, such as, the use of the negotiated criteria, the adequacy of the chosen vocabulary or the provision of constructive feedback. The results show that overall the quality of PA can be improved. Groups did not provide sufficient criticism, questions and suggestions for improvement.     

Small group discussion as a key component in online assessment training for enhanced student learning in web-based peer assessment

The purpose of this study is to examine the effect of online assessment training, with synchronous group discussion as a key component, on subsequent web-based peer assessment results. Participants included 81 college students, mostly women, taking a business writing class. After initial submission of a draft counter-offer letter, they completed online assessment training by engaging in a consensus-seeking, synchronous group discussion of the rubric-based ratings they gave to sample counter-offer letters. They then engaged in web-based assessment of randomly assigned peer counter-offer letters and upon receiving peer feedback, submitted revision of their own counter-offer letter. The discussion groups were randomly assigned to either anonymous (using a pseudo name) or identified conditions (using real name). Findings indicate that the quality of student counter-offer letters improved after online assessment training, and improved even more after web-based peer assessment. There were no significant differences between anonymous discussion groups and identified groups.     

The role of anonymity in peer assessment

This quasi-experimental study aimed to examine the impact of anonymity and training (an alternative strategy when anonymity was unattainable) on students’ performance and perceptions in formative peer assessment. The training in this study focused on educating students to understand and appreciate formative peer assessment. A sample of 77 students participated in a peer assessment activity in three conditions: a group with participants’ identities revealed (Identity Group), a group with anonymity provided (Anonymity Group) and a group with identities revealed but training provided (Training Group). Data analysis indicated that both the Anonymity Group and Training Group outperformed the Identity Group on projects. In terms of perceptions, however, the Training Group appreciated the value of peer assessment more and experienced less pressure in the process than the other two groups.     

Peer assessment of class participation: applying peer nomination to overcome rating inflation

Having students actively engaged with each other in discussions has become an increasingly important and common aspect of the classroom environment. This increased emphasis has also meant that instructors need to find ways to effectively and efficiently evaluate class participation. In this paper, we describe the most common method used for these assessments and highlight some of its inherent challenges. We then propose an alternative method based on peer nominations. Two case studies illustrate the advantages of this method; we find that it is both easy for students to complete and provide instructors with valuable diagnostic information with which to provide feedback and assign grades.     

The alignment of written peer feedback with draft problems and its impact on revision in peer assessment

Despite the wide use of peer assessment, questions about the helpfulness of peer feedback are frequently raised. In particular, it is unknown whether, how and to what extent peer feedback can help solve problems in initial texts in complex writing tasks. We investigated this research gap by focusing on the case of writing literature reviews in an academic writing course. The dataset includes two drafts from 21 students, sampled to represent a wide range of document qualities, and 84 anonymous peer reviews, involving 1,289 idea units. Our study revealed that: (1) at both substance and high prose levels, drafts of all quality levels demonstrated more common problems on advanced writing issues (e.g. counter-argument); (2) peer feedback was driven by difficulty of the problem rather than overall draft quality, peer comments were not well aligned with the relative frequency of problems, more comments were given to less difficult problems; (3) peer feedback had a moderate impact on revision, and importantly, receiving multiple comments on the same issue led to more repairs and improvement of draft quality, but consistent with the comments received, authors tended to fix basic problems more often. Implications for practice and research are drawn from these findings.     

The assessment of lesson plans in teacher education: a case study in assessment validity and reliability

This paper forms part of an exploration of assessment on one part‐time higher education (HE) course: an in‐service, professional qualification for teachers and trainers in the learning and skills sector which is delivered on a franchise basis across a network of further education colleges in the north of England. This paper proposes that the validity and reliability of portfolio‐based assessment, a key component of many HE programmes in addition to the course being researched here, is contestable. Analysis of the processes of compiling portfolios for assessment, through the conceptual framework of the New Literacy Studies, suggests that the ways in which portfolios are assessed and the ways in which the crucial requisites of validity and reliability are assigned to them, mask complexities and contradictions in their creation by the student. This paper argues for a new, critical analysis of portfolio production and raises a number of questions about the validity, reliability and authenticity of the assessment process that the portfolios reify.