Evaluating the reliability of a detailed analytic scoring rubric for foreign language writing

The demand for valid and reliable methods of assessing second and foreign language writing has grown in significance in recent years. One such method is the timed writing test which has a central place in many testing contexts internationally. The reliability of this test method is heavily influenced by the scoring procedures, including the rating scale to be used and the success with which raters can apply the scale. Reliability is crucial because important decisions and inferences about test takers are often made on the basis of test scores. Determining the reliability of the scoring procedure frequently involves examining the consistency with which raters assign scores. This article presents an analysis of the rating of two sets of timed tests written by intermediate level learners of German as a foreign language (n = 47) by two independent raters who used a newly developed detailed scoring rubric containing several categories. The article discusses how the rubric was developed to reflect a particular construct of writing proficiency. Implications for the reliability of the scoring procedure are explored, and considerations for more extensive cross-language research are discussed.     

Do university students,alumni, educators and employers link assessment and graduate employability?

Within higher education literature, assessment and graduate employability are linked and co-presented, in that quality student assessment is purported to enhance employability. This research was designed to query the extent to which these same conceptual links are perceived by those actively involved in higher education. Four stakeholder groups from multiple disciplines and eight Australian states and territories (students, alumni, educators and employers) were interviewed about graduate employability (n?=?127). Interviewers intentionally omitted any mention of assessment to determine whether the various stakeholders would bring it up themselves when asked questions such as what is and is not effective for nurturing employability. The results indicated that among the educators, assessment emerged as a dominant theme. While the three other stakeholder groups infrequently used the term assessment, they did discuss related educational concepts and practices in the context of enhanced employability. All stakeholder groups identified a missing link between theory and practice, with educators specifying that link as assessment. Recommendations to improve employability through assessment are the key takeaways from this research.     

我国研究生教育规模扩张与质量提升凝视———从三部委发布《关于深化研究生教育改革的意见》谈起

改革开放30 多年来,我国逐步建立起布局合理、学科齐全、类型多样的研究生教育体系,从根本上改变了高等教育的层次结构,基本实现了立足国内培养研究生的战略目标。早在2005 年,我国在校研究生人数超过100 万,成为名副其实的研究生教育大国。但是,近年来伴随着规模扩张而教育质量却出现了下降趋势,受到了媒体愈来愈多的质疑。教育质量下降的原因很多,归纳起来大约有三个方面：其一,研究生教育资源跟不上招生规模的扩张。其二,研究生教育模式与社会需求不适应。其三,研究生教育重外在功利需求而轻内在修养需求。对此,教育部、国家发改委、财政部颁布《关于深化研究生教育改革的意见》并召开全国研究生教育工作视频会议,对研究生教育改革进行了全面部署。虽然《关于深化研究生教育改革的意见》提出了“优化类型结构”等一系列改革举措,但依然是基于研究生教育外在功利需求而不是内在修养需求的。其实,研究生教育改革应该更加重视对研究生个体未来发展奠定基础的内在修养需求。     

Does inequality in opportunity perpetuate inequality in outcomes? International evidence from four TIMSS cycles

Entrenched socioeconomic inequalities in achievement have been observed in international assessments for decades, with previous research suggesting that these inequalities may be exacerbated by differentiation in the opportunities provided to various social groups. Much previous research in this field has been US-based or subject to measurement issues. This study investigated socioeconomic inequalities in outcomes and opportunities using four cycles of international assessment data across 78 countries. Further, the paper questions whether inequalities in opportunities are related to inequalities in outcomes. The findings indicate global social inequalities in mathematics and science outcomes, while inequality of opportunity was particularly concentrated in economically advanced countries. Little evidence of the perpetuation of socioeconomic inequality in achievement through differentiated educational provision was found.     

创造教育评估初探

培养具有创造精神和创造能力的人才必须依靠创造教育,而创造教育必须有创造教育评估作保证,创造教育评估应充分考虑：第一,基本评估与特色评估的关系;第二,创造型教师及其评估;第三,大学生的个性评估;第四,大学生的能力评估;第五,大学生创造性测验与评估。     

Test Design with Cognition in Mind

One of the primary themes of the National Research Council's 2001 book Knowing What Students Know was the importance of cognition as a component of assessment design and measurement theory ( NRC, 2001 ). One reaction to the book has been an increased use of sophisticated statistical methods to model cognitive information available in test data. However, the application of these cognitive-psychometric methods is fruitless if the tests to which they are applied lack a formal cognitive structure. If assessments are to provide meaningful information about student ability, then cognition must be incorporated into the test development process much earlier than in data analysis. This paper reviews recent advancements in cognitively-based test development and validation, and suggests various ways practitioners can incorporate similar methods into their own work.     

浅析日本研究生院大学的形成与发展

近十几年来,日本的研究生教育体制改革思路新颖、富于创新;独立研究生院、联合研究生院等新的办学模式纷纷出现。始于20世纪80年代末的研究生院大学作为专门培养日本顶尖人才的高等学府,更是在众多类型的学校中独树一帜。     

What English Counts as Writing Assessment? An Australian Move to Mainstream Critical Literacy

This article focuses on the demands of assessment tasks to address the question: What is entailed in becoming recognised as an accomplished student‐writer? In taking up this focus, the writers use authentic samples to show how tasks for assessing writing can be read as instantiations of particular approaches to English education. They investigate how the tasks inevitably draw on cultural knowledges as a primary resource and raise the issue of what is involved when writing assessment moves away, as it is doing in some Australian schools, from concerns with personal voice and individual growth to a socially critical, discourse‐oriented approach.     

Reflection on Foreign Languages Network Teaching Platform

This paper aims to build awareness in applying foreign languages network teaching platform to higher education.Teachers rationally use foreign languages network teaching platform to hold an initiative concept,establish a cooperative and interactive learning atmosphere,and a prompt and dynamic assessment system.     

When a “sloppy copy” is good enough: Results of a state writing assessment

Students in grades 5 and 8 completed a state writing assessment, and their first and final drafts on the extended writing portion of the test were copied and scored using the state writing rubric. The rubric consisted of three primary traits: Content and Organization, Style and Fluency, and Language Use. Scorers were blind to the study purpose and scored either a student's first or final draft. No significant difference was found between the first and final drafts written by students in special education at both grade levels. Likewise, no significant difference was found for the writing of general education students in grade 8. A significant difference was found, however, between first and final drafts written by fifth-grade students in general education. Cross tabulations conducted at grades 5 and 8 revealed that over 50% of the first drafts received the same score or a better score than what was earned on the final draft.     

Introducing teachers to new semiotic tools for writing instruction and writing assessment: consequences for students’ writing proficiency

This article reports consequences for student writing quality based on a long-term professional learning project. Project teachers, representing all school subjects in grades 3–7, were presented with a writing construct, ‘Wheel of Writing’, and norms of expectation for writing proficiency. Participating teachers used the writing construct and norms as a basis for writing instruction and writing assessment. The project was conducted in 24 schools across Norway. 3088 students from 20 project schools participated. Two hundred and thirty three students from 4 schools were used as a comparison group. The investigation showed that students in primary school improved their writing quality significantly. Students in lower secondary school did not. However, there was substantial variation in writing quality effects between schools, classes, and individual students. For instance at a number of schools, project students from lower secondary school improved their writing quality significantly. The article discusses potential explanations of the effects.     

What traits do academics value in student writing? Insights from a psychometric approach

The number of students studying for university qualifications through the medium of English and for whom English is not their first language has increased significantly in recent years. This, along with efforts to widen access to those under-represented in higher education, has brought into focus the question of what academics see as constituting a ‘good’ piece of student writing. In this small-scale pilot study, Thurstone’s method of paired comparisons was used to establish a scale for ranking six essays in terms of how favourably each was viewed by academic lecturers when compared with every other essay in the set. Kelly’s repertory grid technique was subsequently applied to interviews conducted with the same lecturers to establish which traits they associated with the upper and lower end of the scale. Findings suggest that this methodology represents a promising approach to establishing what academic tutors regard as the key elements of good writing and thus provides an indication of what English teachers might fruitfully focus on in their teaching of the skill.     

研究生教育质量监控要谨防“跑冒滴漏”———从2019年春天三起新闻事件谈起

2019年春天有三起有关研究生的新闻事件,引起社会舆论的广泛关注和热议。对此,政府教育行政部门及时回应了社会舆论关切,对相关涉事高校及其责任人给予了相应的处理。这种对新闻事件的应急处理是必要的,但更为重要的是应该深刻反省隐藏其后的研究生教育质量监控\"跑冒滴漏\"这一根本问题。研究生教育质量监控要坚守公平正义宗旨,研考选拔的严肃性和权威性、教育培养的科学性和严谨性、毕业审核的规范性和公正性是不容挑战的。新时代保障研究生教育质量的根本之策在于,完善研考选拔机制把好入口关,完善教育培养机制把好过程关,完善毕业审核机制把好出口关,让质量监控贯穿于研究生教育的全过程。     

The impact of formative peer feedback on higher education students’ academic writing: a Meta-Analysis

Peer feedback is frequently implemented with academic writing tasks in higher education. However, a quantitative synthesis is still lacking for the impact that peer feedback has on students’ writing performance. The current study conveyed two types of observations. First, regarding the impact of peer feedback on writing performance, this study synthesized the results of 24 quantitative studies reporting on higher education students’ academic writing performance after peer feedback. Engagement in peer feedback resulted in larger writing improvements compared to (no-feedback) controls (g?=?0.91 [0.41, 1.42]) and compared to self-assessment (g?=?0.33 [0.01, 0.64]). Peer feedback and teacher feedback resulted in similar writing improvements (g?=?0.46 [-0.44, 1.36]). The nature of the peer feedback significantly moderated the impact that peer feedback had on students’ writing improvement, whereas only a theoretically plausible, though non-significant moderating pattern was found for the number of peers that students engaged with. Second, this study shows that the number of well-controlled studies into the effects of peer feedback on writing is still low, indicating the need for more quantitative, methodologically sound research in this field. Findings and implications are discussed both for higher education teaching practice and future research approaches and directions.