A question of choice: The implications of assessing expressive writing in multiple genres

The Maryland School Performance Assessment Program (MSPAP) tests include an expressive writing task in which students at grades 3, 5, and 8 can choose to write about any topic they wish in the form of either a story, poem, or play. This test design feature provided the opportunity to investigate what factors contribute to students' choice of genre, how scorers apply a single expressive writing rubric to a range of genres, and whether these genres constitute equivalent tasks for measurement and reporting purposes. Our study, which combined analysis of statewide score data, 300 randomly selected student texts, questionnaires given to teacher-scorers, and interviews with students, argues strongly for the validity of this choice task as a measure of expressive writing and demonstrates that choice of genre both increases writers' engagement and enhances the fairness of the assessment by giving all students the best opportunity to demonstrate proficiency in this learning outcome. By highlighting several features of student texts that complicate scoring, the study also suggests that accuracy and consistency might be improved by

• 1.1) providing additional sample papers during training,
• 2.2) attending to scorers' assumptions regarding several key concepts, especially “originality,” and
• 3.3) adjusting the ways that training for focused holistic scoring generally takes place.

The study concludes that the perceptions of students, scorers, and classroom teachers are critical to the ongoing development of writing assessments that offer students increasing control and choice.     

Comparability of essay question variants

Writing task variants can increase test security in high-stakes essay assessments by substantially increasing the pool of available writing stimuli and by making the specific writing task less predictable. A given prompt (parent) may be used as the basis for one or more different variants. Six variant types based on argument essay prompts from a large-scale, high-stakes North American writing assessment and six based on issue prompts from the same test were created and evaluated in the research section of the test administrations in the winter of 2009. Examinees were asked to volunteer to write an essay on one of the new prompt/variant tasks. Essays were obtained from 7573 examinees for argument prompts and 10,827 examinees for issue prompts. Results indicated that all variant types produced reasonably similar means, standard deviations, and rater reliabilities, suggesting that the variant strategy should be useable for operational administrations in high stakes essay assessments. Variant type did not interact with gender, ethnicity, or language (self-report that English or another language is the examinee's “best” language).     

An investigation of academic writing in secondary schools to inform the development of diagnostic classroom assessments

Despite the high stakes attached to students’ performance on assessments of academic writing, we still know little about the challenges students face when composing in academic contexts. To begin to address this problem, two studies were designed with the following aims: to identify and describe the most prevalent types of academic writing at the secondary level, and to characterize the challenges that both English Language Learners (ELLs) and non-English-Language Learners (non-ELLs) experience with this type of writing. Findings from these studies reveal that (1) in the context of New York City schools, exposition/argument is the most prevalent genre assigned as a writing task in secondary classrooms, as well as the most valued; and that (2) while both Intermediate ELLs and non-ELLs articulated Translating (the process of articulating ideas in the conventions of written English) as the most frequent challenge, the two groups differed in the kinds of Translating challenges and in the types and range of other challenges they articulated: ELLs articulated fewer challenges specifically related to the genre of exposition than did non-ELLs. Based on these findings we suggest several new approaches to classroom diagnostic assessment of writing, and raise issues to be considered in these approaches.     

Constructs of writing proficiency in US state and national writing assessments: Exploring variability

Persistent gaps between optimistic state and pessimistic national academic performance assessment results are increasingly leading to calls for unified national standards in the US. Critics argue that these gaps reveal vast differences in how proficiency is conceptualized; however, little is known about how conceptualizations compare among large-scale US assessments. To explore this issue, the present study investigated constructs of writing proficiency implicated in 41 US state and national high school direct writing assessments by analyzing the relationships between prompt-genre demands and assessment scoring criteria. Results of this analysis suggest that national writing assessments differ as a group from state assessments in the extent to which they emphasize genre distinctions and present coherent conceptualizations of writing proficiency. The implications of these assessment variations for college preparedness are discussed.     

The relationship between measures of vocabulary and narrative writing quality in second- and fourth-grade students

The purpose of this study was to examine the relationship between measures of written vocabulary and writing quality. Participants included 92 second-grade students and 101 fourth-grade students. Students completed two writing samples: one an experimenter-developed writing task and the other, a standardized assessment of writing quality. Research questions examined whether four vocabulary measures (vocabulary diversity, less frequent vocabulary, mean syllable length, number of polysyllabic words) demonstrated developmental differences, whether the vocabulary measures remained stable across two different writing prompts, and whether the vocabulary measures explained unique and shared variance beyond that explained by compositional length and compositional spelling. The results indicated that vocabulary diversity and less frequent vocabulary showed developmental differences across the two writing tasks. Vocabulary diversity was the only variable to remain stable across the two writing tasks. Commonality analysis revealed that vocabulary measures explained unique and shared variance in writing quality in all four models (2 grades and 2 writing prompts). Generally, vocabulary diversity was the most stable and consistent of the four vocabulary variables.     

Uneasy writing: The defining moments of high-stakes literacy testing

The authors report on an exploratory study that investigated Grade 10 students’ responses to a large-scale, high-stakes literacy test, the results of which determine high school graduation in Ontario, Canada. Through focus-group and individual interviews, the authors found that students perceived the test as evaluating their English skills rather than their cross-curricular literacy as is the stated purpose by the test administrators. Furthermore, the test contributed to students’ narrow definition of writing. Rather than promoting writing as a powerful tool for thinking and expressing ideas, students understood writing to be a fill-in-the-blanks response to particular genres. The authors conclude by pointing out the power of such tests to define unintended learning in schools.     

Are poor Chinese text comprehenders also poor in written composition?

We studied the performance in three genres of Chinese written composition (narration, exposition, and argumentation) of 158 grade 4, 5, and 6 poor Chinese text comprehenders compared with 156 good Chinese text comprehenders. We examined text comprehension and written composition relationship. Verbal working memory (verbal span working memory and operation span working memory) and different levels of linguistic tasks—morphological sensitivity (morphological compounding and morphological chain), sentence processing (syntax construction and syntax integrity), and text comprehension (narrative and expository texts)—were used to predict separately narrative, expository, and argumentation written compositions in these students. Grade for grade, the good text comprehenders outperformed the poor text comprehenders in all tasks, except for morphological chain. Hierarchical multiple regression analyses showed differential contribution of the tasks to different genres of writing. In particular, text comprehension made unique contribution to argumentation writing in the poor text comprehenders. Future studies should ask students to read and write parallel passages in the same genre for better comparison and incorporate both instructional and motivational variables.     

Testing times: the impact of prior knowledge on written genres produced in examination settings

This article explores the relationship between children's prior knowledge of spoken and written discourses and how this influences their attempts at examination writing. Particular emphasis is given to task demand in examination questions and how frequently this fails to acknowledge how prior knowledge might be realized in response to the question. School writing has specific genres of its own, genres which, although they may partially mirror genres found outside school, are nonetheless very specific. A significant factor in school genres is they emphasize an asymmetric power relationship between the teacher and the writer, with the teacher not only knowing the conventions of the genre, but often also acting as the determiner of the title, and as the arbiter of the finished piece of writing. Moreover, children whose home background has socioculturally prepared them for production of written genres are advantaged over those with different cultural and meaning‐making resources available to them.     

Improving the writing performance,knowledge, and self-efficacy of struggling young writers: The effects of self-regulated strategy development

Writing is a complex task. Its development depends in large part on changes that occur in children’s strategic behavior, knowledge, and motivation. In the present study, the effectiveness of an instructional model, Self-Regulated Strategy Development (SRSD), designed to foster development in each of these areas, was examined. Adding a peer support component to SRSD instruction to facilitate maintenance and generalization was also examined. Struggling, third grade writers, the majority of whom were minority students attending schools that served primarily low-income families, received SRSD instruction focused primarily on learning writing strategies and knowledge for planning and composing stories and persuasive essays. These students wrote longer, more complete, and qualitatively better papers for both of these genres than peers in the comparison condition (Writers’ Workshop). These effects were maintained over time for story writing and generalized to a third uninstructed genre, informative writing. SRSD instruction boosted students’ knowledge about writing as well. The peer support component augmented SRSD instruction by increasing students’ knowledge of planning and enhancing generalization to informative and narrative writing. In contrast, self-efficacy for writing was not influenced by either SRSD condition (with or without peer support).     

Writing evaluation: rater and task effects on the reliability of writing scores for children in Grades 3 and 4

We examined how raters and tasks influence measurement error in writing evaluation and how many raters and tasks are needed to reach a desirable level of .90 and .80 reliabilities for children in Grades 3 and 4. A total of 211 children (102 boys) were administered three tasks in narrative and expository genres, respectively, and their written compositions were evaluated in widely used evaluation methods for developing writers: holistic scoring, productivity, and curriculum-based writing scores. Results showed that 54 and 52% of variance in narrative and expository compositions were attributable to true individual differences in writing. Students’ scores varied largely by tasks (30.44 and 28.61% of variance), but not by raters. To reach the reliability of .90, multiple tasks and raters were needed, and for the reliability of .80, a single rater and multiple tasks were needed. These findings offer important implications about reliably evaluating children’s writing skills, given that writing is typically evaluated by a single task and a single rater in classrooms and even in some state accountability systems.     

Reading and writing to learn in secondary education: online processing activity and written products in summarizing and synthesizing tasks

The research reported here employed a multiple-case study methodology to assess the online cognitive and metacognitive activities of 15-year-old secondary students as they read informational texts and wrote a new text in order to learn, and the relation of these activities to the written products they were asked to generate. To investigate the influence of the task, students were required to perform two different tasks which differed in complexity and familiarity. The first task was reading a single text and making a written summary of it, while the second consisted in reading two texts and making a written synthesis of them. To gather information about how students construct meaning from informational texts, we asked students to think aloud as they read and wrote in order to provide us with information about their comprehension and composition processes. We also examined their reading and writing activities during the tasks. The results show that to a large extent secondary school students lack the cognitive and metacognitive processes that would enable them to make strategic use of reading and writing. They also show that, although there are no major differences in the way secondary school students tackle these different tasks, those who create the most elaborate products evidence a more recursive and flexible use of reading and writing. The most obvious conclusion as far as the repercussions of these findings are concerned is that there is an urgent need for work on tasks of this kind in the classroom.     

Use of Writing Samples on Standardized Tests: Susceptibility to Rule-Based Coaching and the Resulting Effects on Score Improvement

Despite the growing use of writing assessments in standardized tests, little is known about coaching effects on writing assessments. Therefore, this study tested the effects of short-term coaching on standardized writing tests, and the transfer of those effects to other writing genres. College freshmen were randomly assigned to either training (receiving nine hours of rule-based coaching) or a control group. Both groups completed pretest essays on the first day, and posttest and transfer essays (writing in a different genre) on the last day of the experiment. Group membership (training versus control) and an interaction between pretest scores and group membership were significant predictors of posttest scores after controlling for pretest performance. Training also produced higher transfer task scores after controlling for pretest and ACT scores. Implications are discussed.     

Examining fourth-grade mathematics writing: features of organization,mathematics vocabulary,and mathematical representations

Increasingly, students are expected to write about mathematics. Mathematics writing may be informal (e.g., journals, exit slips) or formal (e.g., writing prompts on high-stakes mathematics assessments). In order to develop an effective mathematics-writing intervention, research needs to be conducted on how students organize mathematics writing and use writing features to convey mathematics knowledge. We collected mathematics-writing samples from 155 4th-grade students in 2 states. Each student wrote about a computation word problem and fraction representations. We compared mathematics-writing samples to a norm-referenced measure of essay writing to examine similarities in how students use writing features such as introductions, conclusions, paragraphs, and transition words. We also analyzed the mathematics vocabulary terms that students incorporated within their writing and whether mathematics computation skills were related to the mathematics vocabulary students used in writing. Finally, we coded and described how students used mathematics representations in their writing. Findings indicate that students use organizational features of writing differently across the norm-referenced measure of essay writing and their mathematics writing. Students also use mathematics vocabulary and representations with different levels of success. Implications for assessment, practice, and intervention development are discussed.     

The quality assurance of a national English writing assessment: Policy implications for quality improvement

The purpose of this study was to examine the quality assurance issues of a national English writing assessment in Chinese higher education. Specifically, using generalizability theory and rater interviews, this study examined how the current scoring policy of the TEM-4 (Test for English Majors – Band 4, a high-stakes national standardized EFL assessment in China) writing could impact its score variability and reliability. Eighteen argumentative essays written by nine English major undergraduate students were selected as the writing samples. Ten TEM-4 raters were first invited to use the authentic TEM-4 writing scoring rubric to score these essays holistically and analytically (with time intervals in between). They were then interviewed for their views on how the current scoring policy of the TEM-4 writing assessment could affect its overall quality. The quantitative generalizability theory results of this study suggested that the current scoring policy would not yield acceptable reliability coefficients. The qualitative results supported the generalizability theory findings. Policy implications for quality improvement of the TEM-4 writing assessment in China are discussed.     

Analyzing language demands facing english learners on science performance assessments: The sald framework

This article report on the development and use of an analytical framework intended to map the language demands encountered by English learners as they engage in science performance assessments. Grounded in functional and interactional views of language and language use, the authors—two science education researchers and a language scholar—developed the framework via an inductive, iterative, and systematic review of written assessment materials associated with three fifth grade science performance tasks. The resulting Science Assessment Language Demands (SALD) framework is organized around three dimensions: participant structures, communicative modes, and written texts and genres that students are called upon to read and produce. The authors used textual analysis to conduct an expert review of the written documents associated with the three assessment tasks. The results indicate that the framework can be used to document a wide range of functional and interactional language demands involved in science performance assessments. The demands revealed by the SALD framework highlight both potential challenges facing English learners during science performance assessments as well as opportunities afforded by such assessments for demonstrating their knowledge and skills and further developing language proficiency. A major implication of the study is the potential use of the framework to evaluate the language demands and opportunities of science assessments used in classrooms with English learners. © 2010 Wiley Periodicals, Inc. J Res Sci Teach 47: 909–928, 2010     

体裁分析在商务英语写作教学中的应用

写作是一项需要综合知识的语言技能,把语篇语言学知识应用于商务英语写作具有十分重要的意义。文章从语篇语言学的体裁分析理论入手,探讨体裁教学法在商务英语写作教学中的应用。结合写作过程,帮助学生了解体裁的社会功能和交际目的,认识体裁结构和认知模式及语言运用策略,从而使他们更好地理解和创造出符合特定体裁惯例的商务语篇。     

The relationship between vocabulary and writing quality in three genres

The purpose of this study was to examine the role of vocabulary in writing across three genres. Fifth graders (N = 105) wrote three compositions: story, persuasive, and informative. Each composition revolved around the topic of outer space to control for background knowledge. Written compositions were scored for holistic writing quality and several different vocabulary constructs: diversity, maturity, elaboration, academic words, content words, and register. The results indicated that students vary their vocabulary usage by genre. Story text had higher diversity than informative text as well as higher maturity as compared to persuasive text. Persuasive text contained higher diversity than informative text, and higher register than both of the other genres. Informative text included more content words and elaboration than the other text types as well as more maturity than persuasive text. Additionally, multiple regression and commonality analysis indicated that the vocabulary constructs related to writing quality differed by genre. For story text, vocabulary diversity was a unique predictor, while for persuasive text, content words and register were unique predictors. Finally, for informative text content words was the strongest unique predictor explaining almost all of the total variance in the five factor model, although maturity was also a unique predictor.