Assessment training effects on student assessment skills and task performance in a technology-facilitated peer assessment

This study examines the impact of an assessment training module on student assessment skills and task performance in a technology-facilitated peer assessment. Seventy-eight undergraduate students participated in the study. The participants completed an assessment training exercise, prior to engaging in peer-assessment activities. During the training, students reviewed learning concepts, discussed marking criteria, graded example projects and compared their evaluations with the instructor’s evaluation. Data were collected in the form of initial and final versions of students’ projects, students’ scoring of example projects before and after the assessment training, and written feedback that students provided on peer projects. Results of data analysis indicate that the assessment training led to a significant decrease in the discrepancy between student ratings and instructor rating of example projects. In addition, the degree of student vs. instructor discrepancy was highly predictive of the quality of feedback that students provided to their peers and the effectiveness of revisions that they made to their own projects upon receiving peer feedback. Smaller discrepancies in ratings were associated with provision of higher quality peer feedback during peer assessment, as well as better revision of initial projects after peer assessment.     

The road to self-assessment: exemplar marking before peer review develops first-year students’ capacity to judge the quality of a scientific report

Lack of clarity about assessment criteria and standards is a source of anxiety for many first-year university students. The Developing Understanding of Assessment for Learning (DUAL) programme was designed as a staged approach to gradually familiarise students with expectations, and to provide opportunities for the development of the skills required to successfully complete assessment tasks. This paper investigated the students’ perceptions of the first two components of the DUAL programme, which assist first-year biology students to engage with stated assessment criteria and standards in order to develop their capacity to make judgements about scientific report exemplars, their peers’ scientific reports and ultimately their own. The study found strong evidence (96% of responses) that the marking and discussion of exemplar reports with peers and demonstrators clarified expectations of scientific report writing. A key feature of this element of DUAL was the opportunity for structured discussion about assessment criteria and standards between peers and markers (demonstrators). During these discussions, students can clarify explicit statements and develop a tacit knowledge base to enhance their ability to judge the quality of others’ work and their own. The peer review exercise (the second element of DUAL) was not rated as highly, with 65% of students finding the process helpful for improving their report. The negative reactions by a sizeable minority of students highlight the need to clearly communicate the expectations and benefits of peer review, with a focus on how the process of giving feedback to peers might benefit a student as much as receiving feedback on their own report.     

Implementation and outcomes of online self and peer assessment on group based honours research projects

Honours research projects in the School of Civil, Environmental and Mining Engineering at the University of Adelaide are run with small groups of students working with an academic supervisor in a chosen area for one year. The research project is mainly self-directed study, which makes it very difficult to fairly assess the contribution of individual students to a group-based research project. Until recently, a paper-based method of self and peer assessment (SPA) for the honours research projects was used to assess the projects. This was unsatisfactory for both students and academic staff, who found that the same mark was often awarded by the peers to each student without feedback. In 2010, therefore, an online SPA tool, SPARK, together with a set of newly developed assessment criteria, were used for SPA by the honours students. The new SPA criteria are based on teamwork and research skills, and the new rating scales reflect standard academic grades. Surveys of the students confirmed that they found the criteria and scales much more suitable for the assessment of group work and design projects.     

How Well Am I Doing? Using a Corpus-based Analysis to Investigate Tutor and Institutional Messages in Comment Sheets

Current Quality Assurance Agency for Higher Education guidelines highlight the necessity for institutional assessment to be clear and transparent. Whilst institutions spend much energy on making sure that criteria on which they assess work are published and clear, much of the feedback to students on different courses is discursive in nature and consists of both formative and summative assessment. Students also receive information about grades and criteria from a variety of institutional documents. This paper uses a concordance of feedback sheets to investigate the way that tutors use institutional discourse in written feedback sheets on a part-time MA(Ed) and the functions performed by the written feedback. The paper finds a high degree of congruence between the institutional and tutor corpora. Both corpora place considerable emphasis on academic conventions, yet this is not reflected in the criteria used for assessment.     

Teacher learning through participation in a negotiated assessment procedure

This article focuses on the impact of a specific formative assessment procedure, negotiated assessment, on teacher professional learning. Negotiations between the assessor and the teacher as assessee seem to be especially promising for this teacher learning. However, there is no empirical evidence yet that has confirmed this. We explored teachers’ opinions about the usefulness of the different elements in a negotiated assessment procedure for their professional learning and the learning benefits they reported as a result of being engaged in this procedure. Our findings show that teachers found the negotiated assessment procedure useful for their learning and reported different types of learning benefits in terms of change: change in their knowledge, beliefs and attitudes, change in their teaching practice and change in their students’ learning outcomes.     

Learning from reviewing peers’ concept maps in an inquiry context: Commenting or grading,which is better?

In peer assessment, both receiving feedback and giving feedback (reviewing peers’ products) have been found to be beneficial for learning. However, the different ways to give feedback and their influence on learning have not been studied enough. This experimental study compared giving feedback by writing comments and by grading, to determine which contributes more to the feedback providers’ learning. Secondary school students from Russia (n = 51) and the Netherlands (n = 42) gave feedback on concept maps during a physics lesson. The lesson was given in an online inquiry learning environment that included an online lab. Students gave feedback in a special Peer Assessment tool, which also provided assessment criteria. Findings indicate that post-test knowledge scores were higher for students from the commenting group. The difference between the groups was largest for the low prior knowledge students. Possible educational implications and directions for further research are discussed.     

The development of science activities via on-line peer assessment: the role of scientific epistemological views

This study implemented an online peer assessment learning module to help 36 college students with the major of pre-school education to develop science activities for future instruction. Each student was asked to submit a science activity project for pre-school children, and then experienced three rounds of peer assessment. The effects of the online peer assessment module on student learning were examined, and the role of Scientific Epistemological Views (SEVs) in the learning process was carefully investigated. This study found that student peers displayed valid scoring that was consistent with an expert’s marks. Through the online peer assessment, the students could enhance the design of science activities for future instruction; for instance, the science activities became more creative, science-embedded, feasible and more suitable for the developmental stage of pre-school children. More importantly, students with more sophisticated (constructivist-oriented) SEVs tended to progress significantly more for designing science activities with more fun, higher creativity and greater relevancy to scientific knowledge, implying that learners with constructivist-oriented SEVs might benefit more from the online peer assessment learning process. These students also tended to offer more feedback to their peers, and much of the peer feedback provided by these students was categorized as guiding or helping peers to carefully appraise and plan their science activity projects. This study finally suggested that an appropriate understanding regarding the constructivist epistemology may be a prerequisite for utilizing peer assessment learning activities in science education.     

Investigating secondary school students’ unmediated peer assessment skills

The purpose of this study was to investigate secondary school students’ unmediated peer assessment skills. Specifically, 36 seventh graders, without receiving any kind of support, were anonymously assigned to reciprocally assess their peers’ science web-portfolios. Additionally, students’ attitudes towards and intentions about the use of unsupported reciprocal peer assessment were examined. Three data sources were used, namely, interviews, screen-video captured data, and the peer assessors’ feedback produced. Findings showed that the students have positive attitudes towards unsupported reciprocal peer assessment and that they intend to implement it again. It was also found that students have the skills, at least the beginnings, needed for the implementation of peer assessment. Specifically, they were found to be able to define and use their own assessment criteria, whose overall validity and reliability, however, were found to be low. Finally, the feedback they produced included grades, positive and negative judgments, as well as suggestions for changes.     

Peer assessment in a project-based engineering course: comparing between on-campus and online learning environments

As the number of participants in online distance learning courses increases, peer assessment is becoming a popular strategy for evaluating open assignments and for breaking the social isolation surrounding distance education. Yet, the quality and characteristics of peer assessment in massive online courses has received little attention. Hence, this study was set to examine peer feedback quality and grading accuracy in a project-based course. The study applied a sequential exploratory mixed methods design. It included 339 participants who studied the same engineering course, but in three different modes: on-campus (n = 77), small private online course (n = 110), and massive open online course (MOOC) (n = 152). Content analysis of feedback comments identified four categories: reinforcement, statement, verification and elaboration, arranged in an ascending scale of cognitive ability. The findings indicated that the MOOC participants provided more feedback comments and volunteered to assess more projects than their counterparts did. However, the on-campus students provided higher quality feedback and their peer grading was better correlated with the grades assigned by the teaching assistants.     

Student perceptions of quality feedback in teacher education

Many reports have identified a perceived lack of quality in regard to assessment feedback in higher education contexts. One research study in 2007 on undergraduate university students found that less than half of the students (46%) collected their formative feedback, suggesting that from their perspective feedback clearly was not fulfilling the role it should. This is a study of 465 graduate students and 101 undergraduate students studying teacher education at a major Australian university. The study investigated what students perceived to be effective, quality feedback based upon their extensive higher education experiences. Students identified preferences in regard to form, detail and timing of assessment feedback. The data were collected by means of pen and paper survey and identified which strategies the students perceived to be the most effective, particularly within the context of large cohort teaching and written assessment formats. Findings agreed with research elsewhere regarding problems with assessment feedback quality and quantity, but students also provided clear indications of how realistic improvements could be made in terms of assessment feedback processes and strategies.     

Automatic evaluation of open-ended questions for online learning. A systematic mapping

The assessment of students’ performances in Higher Education is one of the essential components of teaching activities. Open-ended tasks allow a more in-depth assessment of students’ learning levels, but their evaluation and grading are time-consuming and prone to subjective bias. Since the Covid-19 pandemic, most traditional Higher Education courses converted to online courses; automatic grading and feedback tools and methods (AGFTM) have become critical components of online learning systems, especially with regards to short answers and essays assessment. This work frames the recent advancement in AGFTM through a systematic mapping of the research field and a literature review. This analysis gives an overview of the trends, specific goals, methods, quality of proposals, challenges and limitations in this research area. The results indicate that it is a growing research area, with a large set of techniques involved, but still not mature, where practical implementations have yet to come.     

Peer review in the classroom: student perceptions,peer feedback quality and the role of assessment

Abstract

Peer review in the classroom can enhance numerous employability skills such as critical appraisal, writing skills, reflection practices and collaborative experiences. This study takes place over two years and discusses the implementation of a repeating blind peer review cycle across a single semester for final year chemistry students enrolled on a compulsory employability module. The feedback cycle promotes personal reflection through the use of mini-reflective questionnaires. The process was assessed by academic tutors at the resubmission stage and/or the peer feedback stage where the quality of peer feedback was directly assessed. The research investigates the quality of peer feedback, the importance of assessment and student perceptions of what is most useful. Methods include directed content analysis of feedback produced, student opinions and a focus group. Students were capable of offering useable feedback across a range of assessment criteria but tend to focus on the important criteria best aligned to the particular assignment. A range of motivational factors and tactics were noticed but students tended to find reviewing the most beneficial.     

The potential formative benefits of portfolio assessment in second and foreign language writing contexts: A review of the literature

This article identifies the potential formative benefits of using portfolio assessment (PA) in second/foreign language classrooms by reviewing the up-to-date research. Three methods were used to search for relevant research: database search, manual search, and citation search. According to the literature, PA has potential benefits due to its process-oriented, authentic, integrated, interactive, and learner-centered characteristics. PA leads to increased motivation, learner autonomy, and improved writing performance. However, not many studies include observation as a method of collecting data. Through observation a researcher can investigate how the benefits of PA are acted upon and negotiated. There is a need for more empirical research in the juxtaposition of PA and formative assessment, particularly in primary and secondary education.     

Promoting deep learning in a teacher education programme through self- and peer-assessment and feedback

The incorporation of self- and peer-assessment and feedback has significant potential as a pedagogical strategy to promote deep learning in project based coursework. This study examined the impact of a deeper approach to learning on pre-service teachers’ critical thinking and metacognitive skills. It also examined the impact on student learning outcomes within a project based module with a significant design element. Forty-seven students participated in the pilot of an online peer feedback system. Results suggest that the quality of students’ reflections through peer feedback and overall satisfaction with the module remained high despite students’ citing a preference for instructor feedback. The data also indicate that the incorporation of self- and peer-assessment and feedback resulted in higher quality learning outcomes and enhanced critical thinking skills.     

Combining peer‐assessment with negotiated learning activities on a day‐release undergraduate‐level certificate course (ECTS level 3)

Peer‐assessment was used within a negotiated curriculum in a module on training and development at ECTS level 3. The students on the programme were exclusively day‐release and all had a major responsibility for the management and delivery of work‐based training programmes. Analysis of student evaluations, supplemented by those of university assessors and the external examiner, showed that the learning on the programme was transformative in that it changed the way students perceived their own abilities and their potential to make improvements in their work both as students and trainers. Peer‐assessment encouraged critical reflection, helped develop skills of autonomous learning and provided feedback that took account of individual learning needs. Weaker students benefited from the talents of their more able peers. Students planned, monitored and assessed their learning activities in a way that significantly increased motivation and raised academic standards. The learning achieved led to a transformation of personal and professional perspective on the part of the students leading to greater personal autonomy.     

Dancing with the devil: formative peer assessment and academic performance

Peer assessment can be important in developing active and independent learners, as well as providing more and faster feedback in large classes, compared to marking done by tutors. In addition, the evaluative, critical stance required by students in order to assess their peers' work encourages the development of higher-order cognitive skills. Changing roles from being assessed to being an assessor can also improve students' ability to judge and improve on their own work. However, peer assessment does have potential problems and there is some debate as to the appropriate academic level at which to implement it, the kinds of feedback that are given and the ways in which students respond. In addition, there is little evidence that peer assessment has an impact on academic performance. This research reports the results of an online peer assessment exercise for a macroeconomics essay conducted in a large Economics 1 class at Rhodes University. Of the 800 students, about half participated in the peer assessment exercise. Data were collected from students via a formal course evaluation. In addition, a sample of 50 essays was evaluated in terms of the relationship between peer marks and final (tutor) marks received and the impact that peer assessment had on the quality of the final essay submitted. An Ordinary Least Squares regression was used to investigate the impact of peer assessment participation on marks. Results showed that peer marks tended to ‘bunch’ in the 60–68% range, indicating the reluctance of peers to give very high or low marks. In general, peers gave more useful feedback on technical aspects, such as presentation and referencing (which were also the categories in which students most often made improvements), than on content. Regression analysis showed that peer assessment participation was not a significant determinant of final essay mark, but that economics ability and English language proficiency were.     

Group peer assessment for summative evaluation in a graduate-level statistics course for ecologists

Peer assessment is often used for formative learning, but few studies have examined the validity of group-based peer assessment for the summative evaluation of course assignments. The present study contributes to the literature by using online technology (the course management system Moodle^?) to implement structured, summative peer review based on an anchored rubric in an ecological statistics course taught to graduate students. We found that grade discrepancies between students and the instructor were fairly common (60% of assignments), relatively low in value (mean = 3.3 ± 2.5% on assignments that had discrepancies) and proportionally higher for criteria related to interpretation of statistical results and code quality and organisation than for criteria related to the successful completion of analysis or instructional tasks (e.g. fitting particular statistical methods, de-identification of one’s submission). Students reported that the peer assessment process increased their exposure to alternative ways of approaching statistical and computational problem-solving, but there were concerns raised about the fairness of the process and the effectiveness of the group component. We conclude with some recommendations for implementing peer assessment to maximise student learning and satisfaction.     

Designing online assessment tools for disengaged youth

This article reports on the development of online assessment tools for disengaged youth in flexible learning environments. Sociocultural theories of learning and assessment and Bourdieu's sociological concepts of capital and exchange were used to design a purpose-built content management system. This design experiment engaged participants in assessment that led to the exchange of self, peer and teacher judgements for credentialing. This collaborative approach required students and teachers to adapt and amend social networking practices for students to submit and judge their own and others' work using comments, ratings, keywords and tags. Students and teachers refined their evaluative expertise across contexts, and negotiated meanings and values of digital works, which gave rise to revised versions and emergent assessment criteria. By combining social networking tools with sociological models of capital, assessment activities related to students' digital productions were understood as valuations and judgements within an emergent, negotiable social field of exchange.     

基于多层面Rasch模型的在线学生自评、师评和学生互评研究

在线学习评价是检测学习质量的重要方式,在评价中加入自评和互评会使评价结果更具合理性。研究针对在线学习评价中自评、师评和同伴评价中的问题,基于Moodle平台,应用多层面Rasch模型探究学生自评、师评和同伴评价的一致性。研究结果表明,师评最为严厉,然后依次是同伴评价和自评;评价量规的制定需要有科学有效的方法,需要让学生参与其中;在线环境中评价应增加即时反馈,从而使评价促进学习。多层面Rasch模型使在线学习评价更具合理性,对评价标准和一致性的检验有重要意义。     

Where angels fear to tread: online peer-assessment in a large first-year class

In the context of widening participation, large classes and increased diversity, assessment of student learning is becoming increasingly problematic in that providing formative feedback aimed at developing student writing proves to be particularly laborious. Although the potential value of peer assessment has been well documented in the literature, the associated administrative burden, also in relation to managing anonymity and intellectual ownership, makes this option less attractive, particularly in large classes. A potential solution involves the use of information and communication technologies to automate the logistics associated with peer assessment in a time-efficient way. However, uptake of such systems in the higher education community is limited, and research in this area is only beginning. This case study reports on the use of the Moodle Workshop module for formative peer assessment of students’ individual work in a first-year introductory macro-economics class of over 800 students. Data were collected through an end-of-course evaluation survey of students. The study found that using the feature-rich Workshop module not only addressed many of the practical challenges associated with paper-based peer assessments, but also provided a range of additional options for enhancing validity and reliability of peer assessments that would not be possible with paper-based systems.