The impact of bilingual dictionaries on lexical sophistication and lexical accuracy in tests of L2 writing proficiency: A quantitative analysis

Writing assessment essentially juxtaposes two elements: how ‘good writing’ is to be defined, and how ‘good measurement’ of that writing is to be carried out. The timed test is often used in large-scale L2 writing assessments because it is considered to provide reliable measurement. It is, however, highly inauthentic. One way of enhancing authenticity in a way that might also promote good writing is to include access to support resources in the test. This article reports a study into whether allowing a bilingual dictionary enhances good writing in German writing tests. Participants (n = 47) took two tests, one with and one without a bilingual dictionary. The essays of sub-sets of the participants were compared in terms of lexical sophistication, lexical accuracy and test score. Lexical sophistication was measured by means of the Lexical Frequency Profile (Laufer & Nation, 1995). Lexical accuracy was determined through a quantitative analysis of words looked up in the dictionary. The results revealed an increase in lexical sophistication in ‘with dictionary’ tests but frequent misuse of look-ups. There was also no improvement in test scores. Subsequent issues for the use of dictionaries in writing tests are raised.     

“This test makes no freaking sense”: Criticism, confusion, and frustration in timed writing

Large-scale writing programs can add value to the traditional timed writing assessment by using aspects of the essays to assess the effectiveness of institutional goals, programs, and curriculums. The “six learning goals” prompt in this study represents an attempt to provide an accurate writing assessment that moves beyond scores. This paper focuses on student challenges to the prompt and testing situation to reveal that many students successfully challenge the task as a deliberate strategy, while less savvy test-takers clearly resist in response to anger, confusion or frustration. While only a small minority of test-takers openly protest the prompt or testing situation, the paper suggests that all students could be better prepared to reflect upon their university experience in timed essays through more coaching and experience with reflective impromptu questions. This finding offers both encouragement and caution for writing programs seeking a single test that can generate both placement scores and valuable feedback.     

Formative assessment of writing in English as a foreign language

Recognizing the importance of formative assessment, this mixed-methods study investigates how four teachers and 100 students respond to the new emphasis on formative assessment in English as a foreign language (EFL) writing classes in Norway. While previous studies have examined formative assessment in oral classroom interactions and focused on either studying students or teachers, little research has been conducted on formative assessment of writing where both students and teachers are studied. As such, this study provides new insight. The findings mostly indicate that contradictions are prevalent amongst teachers’ and students’ perceptions of formative assessment of writing. The contradictions revolve around feedback, grades, text revision, self-assessment, and student involvement. The identified contradictions suggest the need for developing a mutual understanding of formative assessment in order to make it useful and meaningful.     

Predicting human judgments of essay quality in both integrated and independent second language writing samples: A comparison study

This study explores whether linguistic features can predict second language writing proficiency in the Test of English as a Foreign Language (TOEFL iBT) integrated and independent writing tasks and, if so, whether there are differences and similarities in the two sets of predictive linguistic features. Linguistic features related to lexical sophistication, syntactic complexity, cohesion, and basic text information were investigated in relation to the writing scores for both integrated and independent samples. The results of this study show that linguistic features can be used to significantly predict essay scores in the integrated as well as the independent writing. When comparing across the two writing tasks, there are both similarities and differences in the two sets of predictive features. For instance, lexical sophistication was found to be a significant predictor for both tasks while features such as verbs in 3rd person singular form and semantic similarity were only significant predictors for the integrated task. These findings demonstrate that evaluation of the two writing tasks rely on similar and distinct features, and are at least partially assessed using different linguistic criteria. Implications of these findings for the assessment of second language (L2) writing are also discussed.     

The potential formative benefits of portfolio assessment in second and foreign language writing contexts: A review of the literature

This article identifies the potential formative benefits of using portfolio assessment (PA) in second/foreign language classrooms by reviewing the up-to-date research. Three methods were used to search for relevant research: database search, manual search, and citation search. According to the literature, PA has potential benefits due to its process-oriented, authentic, integrated, interactive, and learner-centered characteristics. PA leads to increased motivation, learner autonomy, and improved writing performance. However, not many studies include observation as a method of collecting data. Through observation a researcher can investigate how the benefits of PA are acted upon and negotiated. There is a need for more empirical research in the juxtaposition of PA and formative assessment, particularly in primary and secondary education.     

Students' perceptions of Wiki-based collaborative writing for learners of English as a foreign language

This study proposes a Wiki-based collaborative writing approach to the writing process for EFL (English as a foreign language) learners. A five-stage computer-mediated collaborative writing project including collaborative planning, partitioned drafting, peer-revising, peer-editing, and individual publishing was blended with on-campus English composition course. Fifty-one L2 learners at a university in central Taiwan participated in this project. Procedural scaffolding and collective scaffolding were provided to promote students' self-regulation and thus to foster the development of students' writing skills. A cross-referencing questionnaire survey was adopted to investigate students' perceptions of Wiki-based collaborative writing and students' perceptions of their work in each stage of collaborative writing. As the results indicated, a high percentage of students' satisfaction showed positive perceptions of this Wiki-based collaborative writing environment, and the instructional design of implementing a Wiki-based collaborative writing project with a five-stage writing process does assist EFL learners to accomplish a collaborative writing task on the internet with less limitation of time. This article also points to new possibilities for future research.     

When a “sloppy copy” is good enough: Results of a state writing assessment

Students in grades 5 and 8 completed a state writing assessment, and their first and final drafts on the extended writing portion of the test were copied and scored using the state writing rubric. The rubric consisted of three primary traits: Content and Organization, Style and Fluency, and Language Use. Scorers were blind to the study purpose and scored either a student's first or final draft. No significant difference was found between the first and final drafts written by students in special education at both grade levels. Likewise, no significant difference was found for the writing of general education students in grade 8. A significant difference was found, however, between first and final drafts written by fifth-grade students in general education. Cross tabulations conducted at grades 5 and 8 revealed that over 50% of the first drafts received the same score or a better score than what was earned on the final draft.     

A closer look at integrated writing tasks: Towards a more focussed definition for assessment purposes

The use of integrated tasks is increasingly common in second language assessment in both high stakes and classroom contexts. However, because of the vague definition of what constitutes an integrated task, the types of tasks included in this area and the assessment of the resulting writing is highly varied. In this paper, we argue for a better definition of the term ‘integrated writing task’. We start by reviewing current definitions available in the literature, discuss the construct underlying integrated tasks and then propose a number of features which we feel need to be present in integrated tasks. We then propose a new, narrower definition of the term. We hope that this will result in better task design, improved assessment outcomes for test takers, more detailed test specifications, better replicability of research and a shared research agenda.     

Evaluating the impact of first-year writing course grades on college-student persistence

Informed by Tinto’s model of persistence, this large-scale evaluation study (N = 8916) centers on intersections of first-year composition (FYC), grades, and persistence at a career-focused, polytechnic university in a midwestern U.S. state. It presents a replicable design exploring a pathway through Composition 1 and Composition 2 to graduation and suggests theoretical and practical implications for early-alert, student-support intervention. Results of binary logistic regression indicate that, accounting for demographic and enrollment factors, grades significantly predict whether students take Composition 2 immediately after Composition 1, whether they persist to Composition 2 at all, and whether they graduate. We propose that the linked sequence of writing courses may allow students to build skills over a year-long sequence, enhancing self-efficacy and grade-reinforced senses of belonging, while factors related to ethnic identity and first-generation status pose barriers at critical points along the first-year sequence. We conclude with suggested points of student-support intervention and future research on grades, first-year writing coursework, and college-student persistence related to writing self-efficacy and senses of belonging.     

Incorporating formative assessment in Iranian EFL writing: a case study

Undergraduate students' experience of assessment in universities is usually of summative assessment which provides only limited information to help students improve their performance. By contrast, formative assessment is informative and forward-looking, possessing the leverage to inform students of their day-to-day progress and inform teachers of how to better tailor their instruction to students' immediate learning needs. Despite these potentials, studies carried out on the use of formative assessment in English as a foreign language (EFL) contexts are somehow rare. The current study reports on incorporating formative assessment in an L2 writing course in Iran. The analysis of data from pre- and post-study writing tasks, pre- and post-study questionnaires, and semi-structured interviews revealed that first-year undergraduate students were offered opportunities to improve various aspects of their writing and to develop positive attitudes toward writing as well as formative assessment. However, the students reported several challenges that could have implications for the further implementation of formative assessment in similar contexts.     

The use of scoring rubrics: Reliability, validity and educational consequences

Several benefits of using scoring rubrics in performance assessments have been proposed, such as increased consistency of scoring, the possibility to facilitate valid judgment of complex competencies, and promotion of learning. This paper investigates whether evidence for these claims can be found in the research literature. Several databases were searched for empirical research on rubrics, resulting in a total of 75 studies relevant for this review. Conclusions are that: (1) the reliable scoring of performance assessments can be enhanced by the use of rubrics, especially if they are analytic, topic-specific, and complemented with exemplars and/or rater training; (2) rubrics do not facilitate valid judgment of performance assessments per se. However, valid assessment could be facilitated by using a more comprehensive framework of validity when validating the rubric; (3) rubrics seem to have the potential of promoting learning and/or improve instruction. The main reason for this potential lies in the fact that rubrics make expectations and criteria explicit, which also facilitates feedback and self-assessment.     

The impact of the scoring system of a large-scale standardized EFL writing assessment on its score variability and reliability: Implications for assessment policy makers

Using generalizability (G-) theory and rater interviews as research methods, this study examined the impact of the current scoring system of the CET-4 (College English Test Band 4, a high-stakes national standardized EFL assessment in China) writing on its score variability and reliability. One hundred and twenty CET-4 essays written by 60 non-English major undergraduate students at one Chinese university were scored holistically by 35 experienced CET-4 raters using the authentic CET-4 scoring rubric. Ten purposively selected raters were further interviewed for their views on how the current scoring system could impact its score variability and reliability. The G-theory results indicated that the current single-task and single-rater holistic scoring system would not be able to yield acceptable generalizability and dependability coefficients. The rater interview results supported the quantitative findings. Important implications for the CET-4 writing assessment policy in China are discussed.     

Self-assessment of writing in independent language learning programs: The value of annotated samples

The availability of multi-media technologies in education has made the option of independent learning increasingly attractive. Whilst independent learning presents learners with a more flexible learning context, it also presents new challenges in assessment in that the onus is placed upon the learners themselves to monitor and evaluate their own learning.This paper reports on the development of an approach to learner self-assessment which aims to facilitate the self-assessment of performance on extended-response writing tasks. Whilst such tasks are typical within current approaches to communicative language learning, a review of the self-assessment literature reveals no precedent for how actual task performance might be effectively and reliably self-assessed. A methodology was therefore developed which used annotated samples of learner-produced texts, where the annotations would elucidate both the criteria by which performance should be judged and the standards expected, thus enabling learners to assess their own performances by comparing their work with these samples. The methodology was trialled on a small sample of students. It was found to be both reliable and useful, allowing students not only to accurately evaluate their own performance but also to learn new language skills from the samples.     

Modeling the process of summary writing of Chinese learners of English as a foreign language

In language learning contexts, writing tasks that involve reading of source texts are often used to elicit more authentic integrative language use. Thus, interests in researching these read-to-write tasks in general and as assessment tasks keep growing. This study examined and modeled the process of summary writing as a read-to-write integrated task. Data of test-taking process were collected from 94 Chinese English as a foreign language (EFL) learners who completed two summarization tasks. Seven students first completed the tasks while thinking aloud their mental process. The researcher observed their test-taking process and conducted post-test interviews to learn more about how they completed the tasks. Participants’ verbal reports were coded into mental moves, which were then grouped according to their purposes. These formed the basis for identifying the processes and sub-stages participants followed while completing the tasks. Based on the analysis of the think-aloud, observation, and interview data, a questionnaire was developed containing items specifying the processes task takers went through. The questionnaire was completed by the remaining 87 participants to test the generalizability of the findings. A process model of summarization was constructed subsequently. The implications of the study are discussed with reference to construct validity of summarization tasks, EFL writing instruction and assessment.     

身体写作在中国:别有一番滋味的感性呈现

在西方20世纪60年代以埃莱娜·西苏、波伏娃等为代表的女权主义者提出的身体写作是从意识形态的角度以一种男女平等的信念,旨在反对包括性别歧视在内一切不平等的社会现象。在这种浪潮的影响下,我国女性文学迅速发展,充分的挖掘了女性躯体的革命性,同时也让身体写作成为一股女性文学的重要潮流。     

Promoting writing amongst peers: establishing a community of writing practice for early career academics

In the current research-focused climate, academics are facing increasing pressure to produce research outputs. This pressure can prove particularly daunting for early career (EC) academics, who are simultaneously attempting to master new teaching and administrative demands while establishing their own independent research trajectories. Previous reports suggest that academic writing retreats can be an effective way of increasing research outputs. Such retreats generally involve academics from a range of career stages and require expert facilitators. Through organising a series of structured writing events, this project aims to cultivate an enduring community of practice for academic writers. Reflecting on our EC retreat and subsequent writing days with academics from different career stages, we suggest that success hinged on three key factors: (1) A formal structure comprising bounded periods of intense writing, flanked by group reviewing and goal-setting; (2) Co-located writing with participants based in a shared space, away from their usual workstation and distractions; (3) Peer discussions involving participants at a similar career stage. Specifically we found that writing amongst ‘equals’ increased productivity and confidence amongst EC academics.     

Using rubrics in student self-assessment: student perceptions in the English as a foreign language writing context

The instructional value of rubrics for promoting student learning and aiding teacher feedback to student performance has been extensively researched in th educational literature. There is, nonetheless, a dearth of studies on students’ rubric use in second/foreign language contexts, and fewer studies have investigated the factors affecting rubrics’ effectiveness for promoting student learning. The paper reports a classroom-based inquiry into students’ perceptions of rubric use in self-assessment in English as a Foreign Language context and the factors moderating its effectiveness. Eighty students at a Chinese university participated in the study. The data collected included their reflective journals and six case study informants’ retrospective interviews. Results showed that the rubric was perceived as useful for fostering the students’ self-regulation by guiding them through the stages of goal-setting, planning, self-monitoring and self-reflection. Both within-rubric and rubric-user factors were identified as affecting the rubric’s effectiveness in student self-assessment. The findings are discussed with reference to the design features of rubrics. Implications are drawn for formative rubric use in student self-assessment.     

The effect of rating augmentation on inter-rater reliability: An empirical study of a holistic rubric

A two-stage process by which a holistic rubric is applied to the assessment of open-ended items, such as writing samples, is defined. The first stage involves scoring a performance by the assignment of an integer rating that is congruent with the proficiency level that is exhibited in the performance. The second stage is the subsequent assignment by the rater of an augmentation that indicates whether or not the writing competency reflected in the paper is a bit higher or lower than the competency level reflected in the benchmark paper for the given proficiency level. If the rater feels that the paper represents benchmark proficiency for the given level, no augmentation is assigned to the rating. The results of this study indicate that the use of rating augmentation can improve the inter-rater reliability of holistic assessments, as indicated by generalizability phi coefficients, correlation coefficients, and percent agreement indices. Implications and suggestions for follow-up research are discussed.     

Story writing skills of adults with a history language-impairment

The purpose of this study was to investigate the story writing skills of adults with a history of oral language impairment. It was hypothesized that writing text would pose difficulty for adults with a history of language impairment (LI), and that this difficulty would manifest itself as reduced grammatical complexity and increased errors in spelling, grammar and punctuation. The story writing of 10 adults with a history of LI was compared to a group of 51 unimpaired individuals. Participants were asked to write the story of Cinderella. Stories were analysed for length, grammatical complexity and accuracy of grammar, punctuation and spelling. Data were analysed to determine group trends as well as individual profiles. As a group, the adults with LI showed no difference from the comparison group in the length of their stories as measured by total number of words. The LI group did, however, show reduced grammatical complexity in their writing, as measured by mean length of t-unit. The LI group made more errors in grammar, spelling and punctuation than the comparison group. Individual analyses indicated substantial variability within the LI group. It is concluded that LI in childhood is associated with writing difficulties in adulthood.