Exploring the Impact of Rater Effects on Person Fit in Rater-Mediated Assessments

Researchers have documented the impact of rater effects, or raters’ tendencies to give different ratings than would be expected given examinee achievement levels, in performance assessments. However, the degree to which rater effects influence person fit, or the reasonableness of test-takers’ achievement estimates given their response patterns, has not been investigated. In rater-mediated assessments, person fit reflects the reasonableness of rater judgments of individual test-takers’ achievement over components of the assessment. This study illustrates an approach to visualizing and evaluating person fit in assessments that involve rater judgment using rater-mediated person response functions (rm-PRFs). The rm-PRF approach allows analysts to consider the impact of rater effects on person fit in order to identify individual test-takers for whom the assessment results may not have a straightforward interpretation. A simulation study is used to evaluate the impact of rater effects on person fit. Results indicate that rater effects can compromise the interpretation and use of performance assessment results for individual test-takers. Recommendations are presented that call researchers and practitioners to supplement routine psychometric analyses for performance assessments (e.g., rater reliability checks) with rm-PRFs to identify students whose ratings may have compromised interpretations as a result of rater effects, person misfit, or both.     

Peer assessment accounting for student agreement

Peer assessment has been gaining significant ground as a means of fostering an environment of accountability and responsibility for group projects in higher education. A popular peer assessment method assigns individual grades via a linear relationship to peer assessment scores. Using this method, a group member who receives his or her group’s average peer assessment score is assigned the instructor‐given group grade; the group member who receives a total peer assessment score of zero is assigned an individual grade of zero. Unfortunately, applying this method often requires time‐consuming exploratory data analysis because of the potential for the method to yield a high spread of grades, including grades over 100%. The present study proposes a modification, which accounts for the degree of student agreement. When the modified procedure results in a change from the group grade, the direction of change is always the same, but with smaller magnitude, requiring less subsequent data analysis and instructor time demands. Easy‐to‐modify SAS code for applying the method is presented in the Appendix.     

Gauging Item Alignment Through Online Systems While Controlling for Rater Effects

The alignment of test items to content standards is critical to the validity of decisions made from standards‐based tests. Generally, alignment is determined based on judgments made by a panel of content experts with either ratings averaged or via a consensus reached through discussion. When the pool of items to be reviewed is large, or the content‐matter experts are broadly distributed geographically, panel methods present significant challenges. This article illustrates the use of an online methodology for gauging item alignment that does not require that raters convene in person, reduces the overall cost of the study, increases time flexibility, and offers an efficient means for reviewing large item banks. Latent trait methods are applied to the data to control for between‐rater severity, evaluate intrarater consistency, and provide item‐level diagnostic statistics. Use of this methodology is illustrated with a large pool (1,345) of interim‐formative mathematics test items. Implications for the field and limitations of this approach are discussed.     

大学英语写作评分方法对评分者严厉程度的影响——整体评分法和分析评分法的对比分析

评分标准在写作测试中非常重要,使用不同的评分方法会影响评卷者的评分行为。研究显示,虽然整体法和分析法两种英语写作评分方法都可靠,但是在两种评分中,评卷者的严厉程度以及考生的写作成绩发生很大变化。总体上,整体法评分中,评卷者的严厉程度趋于一致,接近理想值;分析法评分中,考生的写作成绩更高,同时评卷者的严厉程度也存在显著差异。因而,在决定考生前途命运的重大考试中,整体评分法更受推崇。     

Cooperative methods and self-regulated strategies development for argumentative writing

The authors analyze the efficacy of a cooperative method associated with self-regulated strategy development (SRSD) for argumentative writing compared with a SRSD single approach. Seven teachers and 230 ninth-grade students from three middle schools participated in the present study in which teachers were part of a practice-based professional development program. Schools were assigned either to the experimental group or to the comparison group. Over a three-month period, all students were taught SRSD strategies. After this period, students from the experimental group participated in a cooperative setting, whereas the comparison group followed the SRSD only. The results indicated that students in the experimental group made statistically significant greater gains in structural argumentative elements and in the quality of writing texts immediately after the intervention and one month later. Thus, the study provides evidence for the benefits of adding a cooperative setting to SRSD in implementing argumentative writing skills.     

The shifting sands in the effects of source text summarizability on summary writing

This paper reports the effects of the properties of source texts on summarization. One hundred and fifty-seven undergraduates were asked to write summaries of one of three extended English texts of similar length and readability, but differing in other discoursal features such as lexical diversity and macro-organization. The effects of summarizability were examined from two perspectives: students’ summarization task performance and their perception of such effects as demonstrated in the post-summarization questionnaire and interviews. Source text had significant and relatively larger effects than the summarizers’ language abilities on summarization performance. These were more pronounced on Chinese than English summarization. Perceptual data were generally supportive of the performance data and further illuminated the dynamics of the summarizability of a source text and its potential effects on performance. Macro-organization, frequency of unfamiliar words, topic familiarity, length of source texts were among the most influential factors shaping text summarizability in the view of the participants. However, the effects of summarizability proved to be rather idiosyncratic. Implications of these findings are discussed in relation to source text selection when designing summarization tasks.     

The quality assurance of a national English writing assessment: Policy implications for quality improvement

The purpose of this study was to examine the quality assurance issues of a national English writing assessment in Chinese higher education. Specifically, using generalizability theory and rater interviews, this study examined how the current scoring policy of the TEM-4 (Test for English Majors – Band 4, a high-stakes national standardized EFL assessment in China) writing could impact its score variability and reliability. Eighteen argumentative essays written by nine English major undergraduate students were selected as the writing samples. Ten TEM-4 raters were first invited to use the authentic TEM-4 writing scoring rubric to score these essays holistically and analytically (with time intervals in between). They were then interviewed for their views on how the current scoring policy of the TEM-4 writing assessment could affect its overall quality. The quantitative generalizability theory results of this study suggested that the current scoring policy would not yield acceptable reliability coefficients. The qualitative results supported the generalizability theory findings. Policy implications for quality improvement of the TEM-4 writing assessment in China are discussed.     

基于话语的应用型写作教学模式探究

目前,教师职前教育中英语写作教学的根本问题是学生对英语写作及写作教学的认知严重欠缺。为改变当下英语写作教学现状,应采用基于话语的应用型写作教学模式。该模式能够有效增强学生的元认知意识和反思能力,增加学生写作练习时间,并能够以学生为中心展开教学。它能够较好地适应高校应用型教学的培养要求,为改善中学英语写作教学提供合格的师资力量。     

European standards for quality assurance and institutional practices of student assessment in the UK,the Netherlands and the Czech Republic

This article analyses the student assessment procedures of 12 universities in the UK, the Netherlands and the Czech Republic with respect to their alignment with the European standards and guidelines on the quality of assessing higher education students (European Standards and Guidelines for Quality Assurance [ESG] 1.3). Based on qualitative methodology combining document review with semi-structured interviews, the analysis has yielded three major results. First, the assessment procedures of the UK universities studied indicate the highest alignment with the ESG 1.3 when compared to the Czech institutions, with the Dutch universities occupying the middle ground. Second, a preference for summative assessment to account for pressures of massification can be observed at institutions in all three countries. Third, it is argued that contrary to some (ministerial) expectations, the influence of the ESG on institutions seems to be insignificant, due to unawareness of the ESG 1.3 at all of the UK and Dutch universities analysed and only moderate knowledge of ESG 1.3 in the Czech case. For this reason, there seems to be a need for communicating the ESG to higher education institutions rather than revising the scope of the ESG as envisaged in the 2012 Bologna policy document (Bucharest Communiqué).     

英汉思维差异对英语写作的影响及其对策

英汉思维差异的客观存在及其在语言上的表现对于中国学生的英语写作过程和写作水平产生的了极大的影响并综合地体现在措辞、造句和谋篇三个层面上。本文在具体分析英汉思维差异及其表现的基础上,试图从教与学两个方面,结合思维差异的文化根源,探讨可行的应对策略。     

服务外包英语写作需求分析和教学对策研究

随着中国服务外包离岸业务的快速发展,服务外包行业需要具有较高英语能力的人才。从服务外包企业和学生对英语写作的需求分析着手,提出服务外包英语写作的教学对策,同时为服务外包英语写作的课程目标设置、教学内容安排、教学方法采用等提供参考。     

Synchronous discussion between assessors and assessees in web-based peer assessment: impact on writing performance,feedback quality,meta-cognitive awareness and self-efficacy

Research on peer assessment in higher education has been conducted for decades. However, ways to improve the quality of peer feedback and optimise student’ benefits through peer assessment remain a puzzle. The present study aimed to examine the impacts of synchronous discussion between assessors and assessees on writing performance, qualitative feedback quality, meta-cognitive awareness and self-efficacy in web-based peer assessment. A total of 64 undergraduate students participated in the study and were randomly assigned into either the experimental or control group. Participants in the experimental group conducted synchronous discussion after the first round of peer assessment, while students in the control group did not conduct any synchronous discussion. The results revealed that synchronous discussion between the assessors and assessees significantly improved the students’ writing performance, especially content writing skills, affective and meta-cognitive feedback quality, meta-cognitive awareness and self-efficacy. Practical implications, limitations and future directions are discussed based on these findings.     

大学文科概率统计课程的教学策略

概率统计是文科学生进行现场统计和数据分析的重要工具,也是他们学习的难点。我们结合教学实践,提出了几点可行的教学策略。     

基于 PDCA 理论的开放大学信息化管理策略分析--以福建开放大学为例

作为一所以现代信息技术为支撑,没有围墙的远程开放大学,学校的信息化管理工作是一项十分重要的内容。针对学校信息化管理的现状和不足,从 PDCA 管理理论的角度出发,结合福建开放大学的实际工作,构建了基于 PDCA 理论的开放大学信息化管理模型,从创新管理理念和体制、建设信息化综合管理平台、提高支持服务水平和加强技术队伍建设等方面分析了该模型在实际工作中的管理策略,以期提高信息化管理工作的质量和效率,满足学校信息化管理的发展和需要。     

基于元认知策略培训的大学英语听力教学设计

通过对元认知策略使用情况的调研来探讨大学英语听力教学。元认知策略的培养能够使学生更好的把握听力思维过程,提高听力课的教学效果,加强学生自主学习的能力。调研结果显示,学生的策略使用情况处于中等水平,大部分只是偶尔或有时使用学习策略,针对这一结果,根据O’Malley和Charnot的论述设计了有针对性的学习策略课堂培训,并进行了讨论。学习策略培训在课堂上的应用将有利于大学英语教学“培养学生的英语综合应用能力,特别是听说能力”这一目标的实现。     

企业核心竞争力评估指标体系构建及评估模型

提出了企业核心竞争力的评估准则及指标体系设置构建原则，构建了一套由34个指标构成的评估企业核心竞争力的综合指标体系，该指标体系从技术能力、管理能力和集体教学能力三个企业核心竞争力构成要素出发，科学地、全面地反应了企业核心竞争力，为企业决策提供了重要依据，最后探讨了采用因子分析法诊断分析企业核心竞争力的评估方法。     

南阳市中小学素质教育存在问题的归因分析与策略探索

在用问卷调查、访谈、实地考察以及卷宗查阅等方式获取第一手资料的基础上，列述南阳市中小学实施素质教育中所存在的问题，分析了造成这些问题的原因，并提出目前实施素质教育应尝试政府介入、全面落实新课程改革纲要、加大教育投入、加强师资队伍建设、家庭学校社会相互配合等五个方面的策略。     

道歉策略使用差异的跨文化研究及对英语教学的启示

人们不仅用语言来传递信息,而且通过语言来完成各种行为如道歉,这样的行为被称为言语行为。文章从不同的角度对道歉进行定义和研究,研究发现不同文化下道歉策略的使用是不同的。造成这种不同的原因主要来自社会现实以及作为社会现实一部分的文化的不同。这就要求我们在进行外语教学的时候要注意对学生进行文化教学,并且注重学生语用能力的培养。     

数学问题解决的基本要素和策略研究

数学问题解决,是以思考为内涵,以问题目标为定向的心理活动及心理过程。数学问题的解决受到很多方面的影响,现就知识结构、技能因素和情感因素这三方面来谈它们对数学问题解决的影响并对其提出一些改进的策略.     

Keyboarding compared with handwriting on a high-stakes writing assessment: Student choice of composing medium, raters’ perceptions, and text quality

This article examines the influence of keyboarding versus handwriting in a high-stakes writing assessment. Conclusions are based on data collected from a pilot project to move Old Dominion University's Exit Exam of Writing Proficiency from a handwritten format into a dual-option format (i.e., the students may choose to handwrite or keyboard the exam). This test case is used to demonstrate the complex implications for a writing assessment tool when universities aim to accommodate students’ various methods of physically creating a text. Discussion includes speculation about the format of the writing component on the SAT. It is argued that by analyzing the differences in high-stakes exams when students keyboard and when they handwrite, researchers can better understand how the material conditions for composing influence students and raters. It is concluded that if handwriting and keyboarding represent even slightly different composing skills and cognitive processes, then high-stakes writing assessments should aim to accommodate students by providing methods of textual production with which students are most comfortable, even as this accommodation produces new challenges for the creators and managers of large-scale writing assessments.