Assessing assessment quality: Criteria for quality assurance in design of (peer) assessment for learning - A review of research studies

The interest in assessment for learning (AfL) has resulted in a search for new modes of assessment that are better aligned to students’ learning how to learn. However, with the introduction of new assessment tools, also questions arose with respect to the quality of its measurement. On the one hand, the appropriateness of traditional, psychometric criteria is being questioned in the context of assessment for learning. On the other hand, it has been argued that new criteria need to be formulated to acknowledge the unique features of assessment for learning. The issue dealt with in this review is what quality criteria are specifically relevant to AfL. Studies using (peer) assessment for learning were evaluated with regard to use of quality criteria under two perspectives: their recognition of educational measurement criteria, as well as their consideration of student involvement in the assessment of learning. The selected studies were analyzed to determine what specific criteria were utilized in (successive parts of) the assessment cycle (i.e., the process of construction, administration and follow up of an assessment). Our results indicate that quality criteria are dissimilarly connected to the distinct steps of an assessment cycle, which holds as well for the attention given to student involvement in assessment for learning.     

Achieving inclusive excellence: Strategies for creating real and sustainable change in quality and diversity

Since the 1990s, the University of Connecticut has made several shifts in its culture and practice that have resulted in improved educational quality and greater success rates for students from traditionally underrepresented populations. Damon Williams shares his institution's approach.     

The role of peer review in identity development for engineering education researchers

ABSTRACT

Peer review has been the focus of an ongoing study at a series of recent annual conferences of the Australasian Association for Engineering Education (AAEE). A further development of this study has been to explore the perspective/s of the authors of these conference papers and the impact that peer review can have on their development as researchers. This paper uses the identity-trajectory framework to illustrate relationships between peer review and academic identity construction for engineering education authors in the AAEE community. Participants’ responses illustrate how various aspects of responding to reviews and writing reviews for other authors, contribute to the development of the networking and intellectual strands of their academic identity as engineering education researchers. We suggest that members of the global engineering education community should be mindful of how they write their peer reviews of conference papers to ensure the opportunity to constructively contribute to their peers’ successful transition into this different research paradigm is not missed.     

Peer review in the classroom: student perceptions,peer feedback quality and the role of assessment

Abstract

Peer review in the classroom can enhance numerous employability skills such as critical appraisal, writing skills, reflection practices and collaborative experiences. This study takes place over two years and discusses the implementation of a repeating blind peer review cycle across a single semester for final year chemistry students enrolled on a compulsory employability module. The feedback cycle promotes personal reflection through the use of mini-reflective questionnaires. The process was assessed by academic tutors at the resubmission stage and/or the peer feedback stage where the quality of peer feedback was directly assessed. The research investigates the quality of peer feedback, the importance of assessment and student perceptions of what is most useful. Methods include directed content analysis of feedback produced, student opinions and a focus group. Students were capable of offering useable feedback across a range of assessment criteria but tend to focus on the important criteria best aligned to the particular assignment. A range of motivational factors and tactics were noticed but students tended to find reviewing the most beneficial.     

Responding to diversity and striving for excellence: The case of Finland

The implementation of inclusive education has become an internationally accepted goal. In this process a substantial challenge is to simultaneously increase both equality and quality in inclusive education. This article discusses ways of achieving this goal in light of recent research findings which indicate that it is possible to meet both goals at once. The findings of various studies on the Finnish comprehensive school reform, along with recent learning outcomes, support this conclusion. During the comprehensive school era, equality has increased, performance gaps have decreased, and student achievement has improved overall. Possible reasons for this are that Finnish educational policy development has not followed international mainstream trends and that flexible and extensive special education provisions have been built into our school system. Internationally acknowledged requirements for a good education are competent teachers and a school system’s commitment to take on the responsibility of educating children of all abilities.     

提高科技期刊专家审稿质量的途径

为了使专家审稿的过程更加快速有效,从审稿专家的选取、审稿方式和方法的选择、审稿过程的控制等方面对提高科技期刊专家审稿质量的途径进行了分析,只有在正确的审稿专家选择原则的指导下,采用正确的审稿方式和方法,并对审稿过程进行合理的控制,才能使审稿专家对刊物有更加深入的了解,从而更加准确地把握稿件的学术质量。     

International comparisons and trends in external quality assurance of higher education: Commonality or diversity?

The paper explores international comparisons ofthe extent of commonality or diversity in themain national external quality assuranceframeworks for higher education. It has beensuggested, from an European survey, that thereare common features in national qualityassurance frameworks (van Vught andWesterheijden (Quality Management and QualityAssurance in European Higher Education: Methodsand Mechanisms) 1993; Luxembourg, Commission ofthe European Communities, Education TrainingYouth). The paper extends the survey, tappingother comparative reports.These comparisons show that a `general model'of external quality assurance does notuniversally apply, but that most elements of itdo apply in most countries. We conclude thatthe `general model' provides a starting pointfrom which to map deviations. In each country,there may be specific additions of elements oromissions from the model, but more usuallythere are modifications or extensions ofelements. These variations are determined bypracticalities, the size of the highereducation sector, the rigidity/flexibility ofthe legal expression of quality assurance (orthe absence of enshrinement in law), and thestage of development from state control of thesector. Some additions to the `general model'are suggested.The paper also considers efforts to produce aninternational scheme for external qualityassurance of higher education, and theapplicability of the `general model' to thetransfer of quality assurance frameworks fromcountry to country.     

The geography of access and excellence: spatial diversity in higher education system design

In a “settler society” such as Canada, inequalities in higher education have geo-political dimensions, necessitating a theoretical framework that considers the geographic distribution of material wealth and power. This paper explores both the economic and social implications of the geographical distribution of the higher education system in British Columbia, Canada, including the ways in which stratification by institutional type may reinforce spatial inequalities. The study shows that system expansion has not improved regional representation, which has socio-economic implications.

The role of teachers in peer conflict: implications for teacher reflections

Abstract

The purpose of this qualitative study was to explore early childhood teachers’ interventions during peer conflict. Fifteen ethnically diverse teachers in central Texas were asked to reflect on videotaped peer conflict situations. Using thematic analysis, five strategy themes were identified: prevent aggression, consider timing, stop conflict, promote social competence, and use conflict resolution. A majority of teachers indicated a goal to enhance social competence; however, many disagreements resulted in teachers preemptively stopping conflict without discussion of alternatives. We emphasize the benefits of teacher training, including reflections, as a tool in discovering conflict resolution strategies that promote socialization.     

An empirical review of anonymity effects in peer assessment,peer feedback,peer review,peer evaluation and peer grading

Abstract

Peer assessment has proven to have positive learning outcomes. Importantly, peer assessment is a social process and some claim that the use of anonymity might have advantages. However, the findings have not always been in the same direction. Our aims were: (a) to review the effects of using anonymity in peer assessment on performance, peer feedback content, peer grading accuracy, social effects and students’ perspective on peer assessment; and (b) to investigate the effects of four moderating variables (educational level, peer grading, assessment aids, direction of anonymity) in relation to anonymity. A literature search was conducted including five different terms related to peer assessment (e.g., peer feedback) and anonymity. Fourteen studies that used a control group or a within group design were found. The narrative review revealed that anonymous peer assessment seems to provide advantages for students’ perceptions about the learning value of peer assessment, delivering more critical peer feedback, increased self-perceived social effects, a slight tendency for more performance, especially in higher education and with less peer assessment aids. Some conclusions are that: (a) when implementing anonymity in peer assessment the instructional context and goals need to be considered, (b) existent empirical research is still limited, and (c) future research should employ stronger and more complex research designs.     

The role of anonymity in peer assessment

This quasi-experimental study aimed to examine the impact of anonymity and training (an alternative strategy when anonymity was unattainable) on students’ performance and perceptions in formative peer assessment. The training in this study focused on educating students to understand and appreciate formative peer assessment. A sample of 77 students participated in a peer assessment activity in three conditions: a group with participants’ identities revealed (Identity Group), a group with anonymity provided (Anonymity Group) and a group with identities revealed but training provided (Training Group). Data analysis indicated that both the Anonymity Group and Training Group outperformed the Identity Group on projects. In terms of perceptions, however, the Training Group appreciated the value of peer assessment more and experienced less pressure in the process than the other two groups.     

The quality assurance of a national English writing assessment: Policy implications for quality improvement

The purpose of this study was to examine the quality assurance issues of a national English writing assessment in Chinese higher education. Specifically, using generalizability theory and rater interviews, this study examined how the current scoring policy of the TEM-4 (Test for English Majors – Band 4, a high-stakes national standardized EFL assessment in China) writing could impact its score variability and reliability. Eighteen argumentative essays written by nine English major undergraduate students were selected as the writing samples. Ten TEM-4 raters were first invited to use the authentic TEM-4 writing scoring rubric to score these essays holistically and analytically (with time intervals in between). They were then interviewed for their views on how the current scoring policy of the TEM-4 writing assessment could affect its overall quality. The quantitative generalizability theory results of this study suggested that the current scoring policy would not yield acceptable reliability coefficients. The qualitative results supported the generalizability theory findings. Policy implications for quality improvement of the TEM-4 writing assessment in China are discussed.     

Identifying the potential organizational impact of an educational peer review program

The literature on educational peer review (EPR) has focused on evaluating EPR's impact on faculty and/or student learning outcomes; no literature exists on the potential organizational impact. A qualitative (case study) research design explored perceptions of 17 faculty and 10 administrators within a school of nursing in an Ontario university regarding the potential impact of an EPR program on organizational goals. Findings suggested that the implementation of an EPR program may increase consistency in the delivery of the curriculum, provide opportunities for faculty development, act as a method of teaching evaluation, and enhance faculty retention. Higher education institutions may be more likely to provide the resources required for EPR if the organizational benefits can be demonstrated.

La documentation au sujet de “l'évaluation éducative par les pairs” (educational peer review) (EPR) est centrée sur l'étude d'impact de l'EPR sur les objectifs d'apprentissage (learning outcomes) des enseignants et/ou des étudiants; il n'existe aucune documentation portant sur l'impact organisationnel potentiel. Cette recherche qualitative (études de cas) a permis d'explorer les perceptions à l'égard de l'impact potentiel d'un programme d'EPR sur les objectifs organisationnels de 17 enseignants et 10 administrateurs provenant d'une école de sciences infirmières rattachée à une université de l'Ontario (Canada). Les résultats suggèrent que l'implantation d'un programme d'EPR peut accroître la constance de mise en ?uvre du cursus, fournir des occasions de développement pédagogique, servir de méthode d'évaluation de l'enseignement, et améliorer la rétention du corps enseignant. Les institutions d'enseignement supérieur seront peut‐être davantage susceptibles de fournir les ressources nécessaires à l'EPR si l'on peut faire la preuve de ses bénéfices organisationnels.     

On the role of impact evaluation of quality assurance from the strategic perspective of quality assurance agencies in the European higher education area

In this paper the current national legislations, the quality assurance approaches and the activities of impact analysis of three quality assurance agencies from Romania, Spain and Germany are described from a strategic perspective. The analysis shows that the general methodologies (comprising, for example, self-evaluation reports, peer reviews, on-site visits, assessment reports, follow-up measures) and main subjects of quality assurance in higher education (such as study programmes and institutional structures and processes) are very similar in the sample cases. However, up to now, impact evaluation of quality assurance has not been implemented systematically in the sample agencies (as in many others). This is the more relevant since the European standards of quality assurance in higher education oblige quality assurance agencies to analyse their general findings and observe the effects of their activities. Against that background, it is argued that methodologically sound impact analyses of quality assurance interventions in higher education institutions should be seen as an integral part of the agencies’ own quality assurance because it would make their work more transparent and easier to improve systematically. The paper identifies some professionalisation needs required for impact evaluation competences: staff and peers who are qualified by methodological knowledge but also by ‘soft’ skills such as project and conflict management.