Using rubrics in student self-assessment: student perceptions in the English as a foreign language writing context

The instructional value of rubrics for promoting student learning and aiding teacher feedback to student performance has been extensively researched in th educational literature. There is, nonetheless, a dearth of studies on students’ rubric use in second/foreign language contexts, and fewer studies have investigated the factors affecting rubrics’ effectiveness for promoting student learning. The paper reports a classroom-based inquiry into students’ perceptions of rubric use in self-assessment in English as a Foreign Language context and the factors moderating its effectiveness. Eighty students at a Chinese university participated in the study. The data collected included their reflective journals and six case study informants’ retrospective interviews. Results showed that the rubric was perceived as useful for fostering the students’ self-regulation by guiding them through the stages of goal-setting, planning, self-monitoring and self-reflection. Both within-rubric and rubric-user factors were identified as affecting the rubric’s effectiveness in student self-assessment. The findings are discussed with reference to the design features of rubrics. Implications are drawn for formative rubric use in student self-assessment.     

Students’ use of a rubric for research theses

A rubric for research theses was developed, based on the manual of the American Psychological Association, to be used as an assessment tool for teachers and students. The aim was to make students aware of what is expected, get familiar with criteria, and interpret teacher and peer feedback. In two studies, it was examined whether students use and value these functions. In the first study, a rubric was provided to 105 Educational Sciences students working on their bachelor’s thesis. Questionnaire data indicated that students did value the rubric for the intended functions, although rubric use was not related to ability. In a panel interview, teachers stated that the number of proficiency levels should be increased to be able to distinguish between good and excellent students adequately, and that a criterion concerning student’s role during supervision should be added. Therefore, in the second study, 11 teachers were interviewed about their motives to give high grades and about the supervision process. This lead to an extra criterion concerning student’s role during supervision and an additional proficiency level to assess excellent performance. It is argued that an adequate course organisation is conditional for the rubric’s effectiveness.     

Mind the gap! Students’ use of exemplars and detailed rubrics as formative assessment

The current study examined efficient modes for providing standardized feedback to improve performance on an assignment for a second year college class involving writing a brief research proposal. Two forms of standardized feedback (detailed rubric and proposal exemplars) were utilized is an experimental design with undergraduate students (N = 100) at three urban college campuses. Students completed a draft of a proposal as part of their course requirements and were then randomly assigned to receive a detailed rubric, proposal exemplars, or a rubric and proposal exemplars for use in revising their work. Analyses of students’ writing from first draft to second draft indicated that all three conditions led to improvements in writing that were significant and strong in terms of effect size, with the stand-alone detailed rubric leading to the greatest improvement. Follow-up focus groups with students indicated that a stand-alone rubric potentially engages greater mindfulness on the part of the student. Practical implications are discussed.     

Computer-Assisted Rubric Evaluation: Enhancing Outcomes and Assessment Quality

This case study examines the use of computer-assisted grading rubrics using Blackboard’s ? Rubric Evaluation Report to compare expected performance for each criterion of an assignment to the students’ level of achievement. Several studies have reported on the innovative and effective delivery of how learning management systems (LMS) can deliver real-time (immediate data delivery) antecedents that inform students of performance requirements (Atkinson & Lim, 2013; Goomas, in press). The impetus, then, is to perform an analysis on each criterion of the assignment in order to determine how closely student performance matched the requirements of the rubric. Used as an analytical tool, the instructor was able to direct students to particular learning activities and to adjust teaching tactics in order to meet or exceed the rubric requirements.     

Analysing how Scientists Explain their Research: A rubric for measuring the effectiveness of scientific explanations

The present article presents a rubric we developed for assessing the quality of scientific explanations by science graduate students. The rubric was developed from a qualitative analysis of science graduate students’ abilities to explain their own research to an audience of non‐scientists. Our intention is that use of the rubric to characterise explanations of science by scientists, some of whom become professors, would lead to better teaching of science at the university level. This would, in turn, improve retention of qualified and diverse scientists, some of whom may elect to become science teachers. Our rubric is useful as an instrument to help evaluate scientific explanations because it distinguishes between the content knowledge and pedagogical knowledge of scientists, as well as a scientist’s ability to integrate the two in the service of a clear and coherent explanation of his or her research. It is also generally useful in evaluating, or self‐evaluating, science explanations by science professors and researchers, graduate students preparing to be scientists, science teachers and pre‐service teachers, as well as students who are explaining science as part of learning.     

Assessment literacy and student learning: the case for explicitly developing students ‘assessment literacy’

In this paper, we report on a study to quantify the impact on student learning and on student assessment literacy of a brief assessment literacy intervention. We first define ‘assessment literacy’ then report on the development and validation of an assessment literacy measurement instrument. Using a pseudo-experimental design, we quantified the impact of an assessment literacy-building intervention on students’ assessment literacy levels and on their subsequent performance on an assessment task. The intervention involved students in the experimental condition analysing, discussing and applying an assessment rubric to actual examples of student work that exemplified extremes of standards of performance on the task (e.g. poor, excellent). Results showed that such a procedure could be expected to impact positively on assessment literacy levels and on student performance (on a similar or related task). Regression analyses indicated that the greatest predictor of enhanced student marks (on the assessment task that was the subject of the experiment), was the development of their ability to judge standards of performance on student work created in response to a similar task. The intervention took just 50 minutes indicating a good educational return on the pedagogical investment.     

Straddling rows and columns: Students’ (mis)conceptions of an assessment rubric

Abstract

Assessment rubrics intend to make criteria explicit to students. However, an understanding of assessment criteria requires tacit knowledge about which students may not be aware. The lack of such knowledge is notable in undergraduate health science students taking a compulsory course in the social sciences. This study performed a content analysis of how 1^st year health sciences students interpret a SOLO assessment rubric for a sociology ‘reflection piece’ essay. The findings suggest that empiricism, standardisation and lack of tacit knowledge limit students’ ways of reasoning about the rubric. These limitations manifest in a methodical difference between students’ expectations and lecturers’ intentions of using a rubric. This difference is illustrated by the students’ need for procedural knowledge and the lecturer’s expectation of conceptual knowledge. These differences have implications for teaching, learning and assessment in health sciences undergraduate education. The findings also justify a dialogical approach to assessments through facilitating students’ epistemological development.     

When is the best time to use rubrics in flipped learning? A study on students’ learning achievement,metacognitive awareness,and cognitive load

ABSTRACT

Rubrics can guide teachers to provide effective feedback and students’ peer- and self-assessment. This paper examines the effect of rubric use in flipped learning activities on students’ learning achievement, metacognitive awareness, and cognitive load through a quasi-experiment. A total of 83 university students were randomly assigned to an experimental group or a control group. The students in the experimental group were given rubrics in performing pre-class learning activities on the cloud platform, while those in the control group used rubrics in post-class assessment and modification of their finished works. The results show that rubric use can assist flipped learning by improving students’ learning achievement and metacognitive awareness, while reducing their cognitive load. Using rubrics before in-class work can yield stronger benefits in raising metacognitive awareness and lessening cognitive load. Moreover, the study reveals that students tend to hold a generally favorable attitude toward rubric use. These findings imply that appropriate use of rubrics in flipped learning has the potential to create a more efficient and learner-friendly condition to enhance self-regulated learning.     

Reliability and validity of rubrics for assessment through writing

This experimental project investigated the reliability and validity of rubrics in assessment of students’ written responses to a social science “writing prompt”. The participants were asked to grade one of the two samples of writing assuming it was written by a graduate student. In fact both samples were prepared by the authors. The first sample was well written in terms of sentence structure, spelling, grammar, and punctuation; however, the author did not fully answer the question. The second sample fully answered each part of the question, but included multiple errors in structure, spelling, grammar and punctuation. In the first experiment, the first sample was assessed by participants once without a rubric and once with a rubric. In the second experiment, the second sample was assessed by participants once without a rubric and once with a rubric. The results showed that raters were significantly influenced by mechanical characteristics of students’ writing rather than the content even when they used a rubric. Study results also indicated that using rubrics may not improve the reliability or validity of assessment if raters are not well trained on how to design and employ them effectively.     

The impact of a rubric and friendship on peer assessment: Effects on construct validity,performance, and perceptions of fairness and comfort

Construct validity of peer assessment (PA) is important for PA application, yet difficult to achieve. The present study investigated the impact of an assessment rubric and friendship between the assessor and assessee on construct validity of PA. Two-hundred nine bachelor students participated: half of them assessed a peer's concept map with a rubric whereas the other half did not use a rubric. The results revealed a substantial reliability and construct validity for PA. All students over-score their peers’ performance, but students using a rubric were more valid. Moreover, when using a rubric a high level of friendship between assessor and assessee resulted in more over-scoring. Use of a rubric resulted in higher quality concept maps for peer and expert ratings.     

Systems thinking of pre- and in-service science and engineering teachers

Systems thinking is an important skill in science and engineering education. Our study objectives were (1) to create the basis for a systems thinking language common to both science education and engineering education, and (2) to apply this language to assess science and engineering teachers’ systems thinking. We administered two assignments to teacher teams: first, modelling the same adapted scientific text, and second, modelling a synthesis of peer-reviewed articles in science and engineering education, with teams selecting a topic from a list and summarising them. We assessed those models using a rubric for systems thinking we had developed based on our literature review of this topic. We found high interrater reliability and validated the rubric’s theoretical construct for the system aspects of function, structure and behaviour. We found differences in scores between the assignments in favour of the second assignment, for two attributes of systems thinking: ‘expected outcome/intended purpose’ and ‘main object and its sub-objects’. We explain the first attribute difference as stemming from the modellers’ domain expertise as science or engineering teachers, rather than as scientists or engineers, and the second attribute difference – from the larger amount of information available for modelling the articles synthesis assignment. The theoretical contribution of this study lies in the definition of the systems thinking construct as a first step in establishing a common language for the science education and engineering education communities. The study's methodological contribution lies in the rubric we developed and validated, which can be used for assessing the systems thinking of teachers and potentially also of undergraduate students.     

Effects of a rubric for mathematical reasoning on teaching and learning in primary school

Rubrics are assessment tools that help students gain complex competencies. Our quasi-experimental study aimed to evaluate whether rubrics help teachers teach and assess mathematical reasoning in primary school and whether such an instrument might support student learning. In two Swiss cantons, 762 students in 44 5th- and 6th-grade primary classes worked on their reasoning competencies, and half of them additionally employed our standards-based rubric. All of the teachers received a 1-day training and participated in the final project evaluation. To standardise and support the teachers during the implementation phase, they received a detailed curriculum. An achievement test and questionnaires for students and teachers were administered before and at the end of the intervention. The results of our quantitative longitudinal analyses indicate that the rubric fosters the teachers’ perceived diagnostic skills but only indirectly impacts their use of formative feedback. Based on the students’ perceptions, however, we observed a direct effect of the rubric on formative feedback and student self-assessment. Effects on students’ outcomes could not be observed, but there are indications of effects mediated by self-regulation and self-efficacy.     

Rubric‐referenced self‐assessment and middle school students’ writing

This study investigated the relationship between middle school students’ scores for a written assignment (N = 162) and a process that involved students in generating criteria and self‐assessing with a rubric. Gender, time spent writing, grade level, prior rubric use, and previous achievement in English were also examined. The treatment involved using a model essay to scaffold the process of generating a list of criteria for an effective essay, reviewing a written rubric, and using the rubric to self‐assess first drafts. The comparison condition involved generating a list of criteria and reviewing first drafts. Findings include a main effect of treatment, gender, grade level, writing time, and previous achievement on total essay scores, as well as main effects on scores for every criterion on the scoring rubric. The results suggested that reading a model, generating criteria, and using a rubric to self‐assess can help middle school students produce more effective writing.     

Using Online Studio Groups to Improve Writing Competency: A Pilot Study in a Quality Improvement Methods Course

Communication skills are a significant contributor to an individual's success in the workplace. Unfortunately, students often have trouble expressing their ideas in written form and the poor quality of students’ written work often impedes the learning process. This pilot study investigates the use of online writing studios within a quality improvement methods course and the impact of this intervention on students’ writing competency and perceptions about the writing process. The impact of this approach on student performance is evaluated within and across semesters using comparative statistical analyses. Results show that students who participated in online writing studios performed better (i.e., at least one rubric level higher) on their final written assignments for the course compared to students who did not, and that this approach is associated with improved students’ perceptions about the writing process. This research suggests that the use of online writing studios within a content‐specific course can incrementally improve students’ writing competency over the course of one semester, and can positively affect the learning of written communication skills and change students’ perceptions about various aspects of the writing process.     

Oral presentations in higher education: a comparison of the impact of peer and teacher feedback

In this paper the authors analyse the competency of a sample of economics and business students in orally presenting academic content, as well as the changes that occurred after receiving feedback. The students’ presentations were videotaped, and a total of 96 were analysed at three stages: pre-test, post-test and follow-up. The participants were divided into two groups. In one, each student received feedback from a peer as part of a peer assessment with rubric activity. In the other, they received feedback from the teacher immediately after their presentation. In the post-test, the peer assessment with rubric students improved by 10% in the valuation of their presentation, while the teacher feedback students only improved by 5%. These results support the idea that undergraduates’ evaluations of their peers can be effective in improving oral presentation skills, especially when they are provided with some support instruments (videos and rubrics). However, the improvements in the peer assessment with rubric group were not maintained in the follow-up re-test. This suggests that a single session of peer assessment with rubric is insufficient to generalise any improvements in the said competency. The implications of these results with regard to the implementation of methods of formative peer assessment in higher education are discussed.     

Using scaffolded rubrics to improve peer assessment in a MOOC writing course

This study explored the value of using a guided rubric to enable students participating in a massive open online course in writing to produce more reliable assessments of their fellow students’ writing. To test the assumption that training students to assess will improve their ability to provide quality feedback, a multivariate factorial analysis was used to determine differences in assessments made by students who received guidance on using a rating rubric and those who did not. Although results were mixed, on average students who were provided no guidance in scoring writing samples were less likely to successfully differentiate between novice, intermediate, and advanced writing samples than students who received rubric guidance. Rubric guidance was most beneficial for items that were subjective, technically complex, and likely to be unfamiliar to the student. Items addressing relatively simple and objective constructs were less likely to be improved by rubric guidance.     

Ninth grade students’ mental models of the marine environment and their implications for environmental science education in Taiwan

Abstract

The purpose of this study is to investigate students’ mental models of the marine environment and to further examine how these models are related to their perceptions of marine problems. One hundred twenty-eight ninth grade students from a large harbor city in southern Taiwan completed a survey including a drawing activity and a set of two-tier questions. Using a four-factor rubric to analyze student drawings, we found that students’ mental models of the marine environment are generally partial and unconnected. In particular, the human component was often missing in the drawing. Students showed a strong awareness of marine problems; however, their awareness is focused on waste pollution. Moreover, correlation analysis showed that students’ mental models are associated with the scope of the perceived problems.     

The Use and Effectiveness of an Argumentation and Evaluation Intervention in Science Classes

This study explored teachers’ use of the Argumentation and Evaluation Intervention (AEI) and associated graphic organizer to enhance the performance of students in middle and secondary science classrooms. The results reported here are from the third year of a design study during which the procedures were developed in collaboration with teachers. A quasi-experimental pretest–posttest design with 8 experimental and 8 control teachers was used with a total of 282 students. An open-ended test assessed students’ abilities to evaluate a scientific argument made in an article. The students were asked to identify the claim and its qualifiers, identify and evaluate the evidence given for the claim, examine the reasoning in support of the claim, consider counterarguments, and construct and explain a conclusion about the claim. The quality of students’ responses was assessed using a scoring rubric for each step of the argumentation process. Findings indicated a significantly higher overall score and large effect size in favor of students who were instructed using the AEI compared to students who received traditional lecture–discussion instruction. Subgroup and subscale scores are also presented. Teacher satisfaction and student satisfaction and confidence levels are reported.     

Everything is illuminated: What big data can tell us about teacher commentary

What happens to writing instructors’ feedback when they use a common rubric and an online tool to respond to student papers in a first-year composition course at a large state university in the United States? To investigate this question, we analyze the 118,611 comments instructors made when responding to 17,433 student essays. Using concordance software to quantify teachers’ use of rubric terms, we found instructors were primarily concerned with global, substantive, higher-order concerns—such as responding to students’ rhetorical situations, use of reason, and organization—rather than lower-order concerns about grammar or formatting. Given past research has determined teachers overemphasize lower-order concerns such as grammar, mechanics, and punctuation (Connors and Lunsford, 1988, Lunsford and Lunsford, 2008, Moxley and Joseph, 1989, Moxley and Joseph, 1992, Schwartz, 1984, Sommers, 1982, Stern and Solomon, 2006), these results may suggest the possibility of a generational shift when it comes to response to student writing. Aggregating teacher commentary, student work, and peer review responses via digital tools and employing concordance software to identify big-data patterns illuminates a new assessment practice for Writing Program Administrators—the practice of Deep Assessment.