Using scaffolded rubrics to improve peer assessment in a MOOC writing course

This study explored the value of using a guided rubric to enable students participating in a massive open online course in writing to produce more reliable assessments of their fellow students’ writing. To test the assumption that training students to assess will improve their ability to provide quality feedback, a multivariate factorial analysis was used to determine differences in assessments made by students who received guidance on using a rating rubric and those who did not. Although results were mixed, on average students who were provided no guidance in scoring writing samples were less likely to successfully differentiate between novice, intermediate, and advanced writing samples than students who received rubric guidance. Rubric guidance was most beneficial for items that were subjective, technically complex, and likely to be unfamiliar to the student. Items addressing relatively simple and objective constructs were less likely to be improved by rubric guidance.     

Evaluating the reliability of a detailed analytic scoring rubric for foreign language writing

The demand for valid and reliable methods of assessing second and foreign language writing has grown in significance in recent years. One such method is the timed writing test which has a central place in many testing contexts internationally. The reliability of this test method is heavily influenced by the scoring procedures, including the rating scale to be used and the success with which raters can apply the scale. Reliability is crucial because important decisions and inferences about test takers are often made on the basis of test scores. Determining the reliability of the scoring procedure frequently involves examining the consistency with which raters assign scores. This article presents an analysis of the rating of two sets of timed tests written by intermediate level learners of German as a foreign language (n = 47) by two independent raters who used a newly developed detailed scoring rubric containing several categories. The article discusses how the rubric was developed to reflect a particular construct of writing proficiency. Implications for the reliability of the scoring procedure are explored, and considerations for more extensive cross-language research are discussed.     

Evaluating Comparative Judgment as an Approach to Essay Scoring

ABSTRACT

As an alternative to rubric scoring, comparative judgment generates essay scores by aggregating decisions about the relative quality of the essays. Comparative judgment eliminates certain scorer biases and potentially reduces training requirements, thereby allowing a large number of judges, including teachers, to participate in essay evaluation. The purpose of this study was to assess the validity, labor costs, and efficiency of comparative judgments as a potential substitute for rubric scoring. An analysis of two essay prompts revealed that comparative judgment measures were comparable to rubric scores at a level similar to that expected of two professional scorers. The comparative judgment measures correlated slightly higher than rubric scores with a multiple-choice writing test. Score reliability exceeding .80 was achieved with approximately nine judgments per response. The average judgment time was 94 seconds, which compared favorably to 119 seconds per rubric score. Practical challenges to future implementation are discussed.     

Learning to teach writing through tutoring and journal writing

Structured reflection on practical teaching experiences may help pre‐service teachers to integrate their learning and analyze their actions to become more effective learners and teachers. This study reports on 12 pre‐service English as a second language (ESL) teachers’ individual tutoring of learners of English language writing. The data of the study are the writing journal entries that the pre‐service ESL teachers maintained during their tutoring experience. These journals had common elements: all were used by the pre‐service teachers to consider what funds of knowledge they bring to their teaching of ESL learners, to evaluate their roles as writers, learners and teachers and to reflect on the educational, social and cultural implications of teaching writing in English to speakers of other languages. This article describes ways in which both native and non‐native English speaking pre‐service teachers adapted their instruction to meet the particular needs of individual ESL writers and what they learned in the process. It provides insight regarding the value of using tutoring and reflection generally in teacher education and specifically in the preparation of teachers of ESL.     

Understanding the growth of ESL paragraph writing skills and its relationships with linguistic features

This study sought to examine the development of paragraph writing skills of 116 English as a second language university students over the course of 12 weeks and the relationship between the linguistic features of students’ written texts as measured by Coh-Metrix – a computational system for estimating textual features such as cohesion and coherence – and the scores assigned by human raters. The raters’ reliability was investigated using many-facet Rasch measurement (MFRM); the growth of students’ paragraph writing skills was explored using a factor-of-curves latent growth model (LGM); and the relationships between changes in linguistic features and writing scores across time were examined by path modelling. MFRM analysis indicates that despite several misfits, students’ and raters’ performances and scale’s functionality conformed to the expectations of MFRM, thus providing evidence of psychometric validity for the assessments. LGM shows that students’ paragraph writing skills develop steadily during the course. The Coh-Metrix indices have more predictive power before and after the course than during it, suggesting that Coh-Metrix may struggle to discriminate between some ability levels. Whether a Coh-Metrix index gains or loses predictive power over time is argued to be partly a function of whether raters maintain or lose sensitivity to the linguistic feature measured by that index in their own assessment as the course progresses.     

How accurate are ESL students’ holistic writing scores on large-scale assessments?—A generalizability theory approach

Using generalizability theory, this study examined both the rating variability and reliability of ESL students’ writing in the provincial English examinations in Canada. Three years’ data were used in order to complete the analyses and examine the stability of the results. The major research question that guided this study was: Are there any differences between the rating variability and reliability of the writing scores assigned to ESL students and to Native English (NE) students in the writing components of the provincial examinations across three years? A series of generalizability studies and decision studies was conducted. Results showed that differences in score variation did exist between ESL and NE students when adjudicated scores were used. First, there was a large effect for both language group and person within language-by-task interaction. Second, the unwanted residual variance component was significantly larger for ESL students than for NE students in all three years. Finally, the desired variance associated with the object of measurement was significantly smaller for ESL students than for NE students in one year. Consequently, the observed generalizability coefficient for ESL students was significantly lower than that for NE students in that year. These findings raise a potential question about the fairness of the writing scores assigned to ESL students.     

A comparative judgement approach to the large-scale assessment of primary writing in England

ABSTRACT

Writing assessment is a key feature of most education systems, yet there are limitations with traditional methods of assessing writing involving rubrics. In contrast, comparative judgement appears to overcome the reliability issues that beset the assessment of performance assessment tasks. The approach presented here extends previous work on comparative judgement by directly involving teachers in a large number of schools in the judging of young pupils’ writing. To ensure quality control the process incorporated a process of ‘anchoring’ that ensured that teachers could not artificially inflate their own pupils’ scores. The approach was used to assess the writing of 55,599 primary pupils in England in 2017–2018. Overall, the results showed that a comparative judgement approach to writing incorporating anchoring shows promise in providing a fair and robust large-scale method to assess writing.     

电子邮件与英语写作教学活动设计

电子邮件是网络应用中使用最多的交流工具，英语教师可以使用电子邮件的交互功能设计出多种英语写作教学活动，使学生通过真实的语言写作交流来不断提高其英语应用能力和社会交际能力。     

ESL写作中母语语法的负迁移及教学对策

英汉语言的结构不同,给中国英语学习者的写作带来了负面影响。根据迁移理论,通过从五个方面对英汉两种语言在语法层面上的异同进行比较,探讨中国学生在英语写作中由于受母语干扰而容易产生的典型语法错误,希望对改进英语写作教学有所启示。     

Putting Rubrics to the Test: The Effect of a Model, Criteria Generation, and Rubric-Referenced Self-Assessment on Elementary School Students' Writing

The purpose of this study was to investigate the effect of reading a model written assignment, generating a list of criteria for the assignment, and self-assessing according to a rubric, as well as gender, time spent writing, prior rubric use, and previous achievement on elementary school students' scores for a written assignment (N = 116). Participants were in grades 3 and 4. The treatment involved using a model paper to scaffold the process of generating a list of criteria for an effective story or essay, receiving a written rubric, and using the rubric to self-assess first drafts. The comparison condition involved generating a list of criteria for an effective story or essay, and reviewing first drafts. Findings include a main effect of treatment and of previous achievement on total writing scores, as well as main effects on scores for the individual criteria on the rubric. The results suggest that using a model to generate criteria for an assignment and using a rubric for self-assessment can help elementary school students produce more effective writing.     

The impact of the scoring system of a large-scale standardized EFL writing assessment on its score variability and reliability: Implications for assessment policy makers

Using generalizability (G-) theory and rater interviews as research methods, this study examined the impact of the current scoring system of the CET-4 (College English Test Band 4, a high-stakes national standardized EFL assessment in China) writing on its score variability and reliability. One hundred and twenty CET-4 essays written by 60 non-English major undergraduate students at one Chinese university were scored holistically by 35 experienced CET-4 raters using the authentic CET-4 scoring rubric. Ten purposively selected raters were further interviewed for their views on how the current scoring system could impact its score variability and reliability. The G-theory results indicated that the current single-task and single-rater holistic scoring system would not be able to yield acceptable generalizability and dependability coefficients. The rater interview results supported the quantitative findings. Important implications for the CET-4 writing assessment policy in China are discussed.     

Evaluating fifth- and sixth-grade students’ expository writing: task development, scoring, and psychometric issues

Drawing from multiple theoretical frameworks representing cognitive and educational psychology, we present a writing task and scoring system for measurement of students’ informative writing. Participants in this study were 72 fifth- and sixth-grade students who wrote compositions describing real-world problems and how mathematics, science, and social studies information could be used to solve those problems. Of the 72 students, 69 were able to craft a cohesive response that not only demonstrated planning in writing structure but also elaboration of relevant knowledge in one or more domains. Many-facet Rasch Modeling (MFRM) techniques were used to examine the reliability and validity of scores for the writing rating scale. Additionally, comparison of fifth- and sixth-grade responses supported the validity of scores, as did the results of a correlational analysis with scores from an overall interest measure. Recommendations for improving writing scoring systems based on the findings of this investigation are provided.     

The quality assurance of a national English writing assessment: Policy implications for quality improvement

The purpose of this study was to examine the quality assurance issues of a national English writing assessment in Chinese higher education. Specifically, using generalizability theory and rater interviews, this study examined how the current scoring policy of the TEM-4 (Test for English Majors – Band 4, a high-stakes national standardized EFL assessment in China) writing could impact its score variability and reliability. Eighteen argumentative essays written by nine English major undergraduate students were selected as the writing samples. Ten TEM-4 raters were first invited to use the authentic TEM-4 writing scoring rubric to score these essays holistically and analytically (with time intervals in between). They were then interviewed for their views on how the current scoring policy of the TEM-4 writing assessment could affect its overall quality. The quantitative generalizability theory results of this study suggested that the current scoring policy would not yield acceptable reliability coefficients. The qualitative results supported the generalizability theory findings. Policy implications for quality improvement of the TEM-4 writing assessment in China are discussed.     

略论ESL写作教学中的"产品法"与"过程法"

对英语作为第二语言(ESL)的写作教学来说,"以过程为中心"的方法较诸"以产品为中心"的方法可能更为有效.作为一种写作教学方法,"过程法"注重写作的过程,因而在实施中特别重视写作过程中的不同阶段,并就各阶段设置了多种多样的练习活动,以使学生写出更有意义的作品.然而,我们不能将"过程法"降为一种具有规定技巧和惯例的"办法",而应创设有效的写作学习环境,在这种环境中,学生不仅对写作感到轻松愉快,而且能自主探索并培育个性化的写作方法.     

多侧面模式在教育测试中的应用研究

主观测试实施过程中,由于存在多种因素导致最终测试结果的信度和效度降低,因此,对影响测试信度和效度各种因素的发现和分析就显得格外重要.本文主要介绍基于试题反应理论的多侧面模式产生背景、基本框架、在国内外教育测评上的典型应用以及此模式的局限性,从而说明多侧面模式作为一种新的测评模式,可以较全面地找出影响测试信度和效度的因素,特别是评分员主观效应因素,并能够对其进行客观分析.近年来,该模式在国内外教育测评上的应用也越来越广泛.     

The impact of two types of peer assessment on students' performance and satisfaction within a Wiki environment

The purpose of this study was to compare the effects of two peer assessment methods on university students' academic writing performance and their satisfaction with peer assessment. This study also examined the validity and reliability of student generated assessment scores. Two hundred and thirty-two predominantly undergraduate students were selected by convenience sampling during the fall semester of 2007. The results indicate that students in the experimental group demonstrated greater improvement in their writing than those in the comparison group, and the findings reveal that students in the experimental group exhibited higher levels of satisfaction with the peer assessment method both in peer assessment structure and peer feedback than those in the comparison group. Additionally, the findings indicate that the validity and reliability of student generated rating scores were extremely high. Using Wiki interactive software and providing an online collaborative learning environment to facilitate peer assessment added value to peer assessment.     

The use of exemplars in English writing classrooms: from theory to practice

Recent literature on the use of exemplars in the context of higher education has shown that exemplar-based instruction is implemented in various disciplines; nevertheless, how exemplar-based instruction can be implemented in English-as-a-Second-Language (ESL) writing classrooms in higher education institutions remains under-explored. In this connection, this article reports on a textbook development project which adopts an exemplar-based instruction approach to be used by university English instructors to prepare students for IELTS writing (academic module). The goal of the textbook is to cultivate students’ understanding of the assessment standards of the two IELTS writing tasks through the design and use of exemplar-based dialogic and reflective activities. In this article, theoretical underpinnings of the use of exemplars, namely tacit knowledge, assessment as learning and dialogic feedback, will first be discussed in detail. Then, an overview of an ongoing project which aims to develop an exemplar-based IELTS writing textbook will be given. The last section of this article suggests practical strategies for ESL writing teachers who are interested in using exemplars to develop students’ understanding of assessment standards.