Rater accuracy and training group effects in Expert- and Supervisor-based monitoring systems

This study evaluated rater accuracy with rater-monitoring data from high stakes examinations in England. Rater accuracy was estimated with cross-classified multilevel modelling. The data included face-to-face training and monitoring of 567 raters in 110 teams, across 22 examinations, giving a total of 5500 data points. Two rater-monitoring systems (Expert consensus scores and Supervisor judgement of correct scores) were utilised for all raters. Results showed significant group training (table leader) effects upon rater accuracy and these were greater in the expert consensus score monitoring system. When supervisor judgement methods of monitoring were used, differences between training teams (table leader effects) were underestimated. Supervisor-based judgements of raters’ accuracies were more widely dispersed than in the Expert consensus monitoring system. Supervisors not only influenced their teams’ scoring accuracies, they overestimated differences between raters’ accuracies, compared with the Expert consensus system. Systems using supervisor judgements of correct scores and face-to-face rater training are, therefore, likely to underestimate table leader effects and overestimate rater effects.     

口试评分规范化与信度研究

口语考试的效度较高,信度却比较低。但没有信度,效度也不可能真正得到保证。因此,如何提高口试的信度,是很多测试研究者普遍关注的问题。本文通过描述清华大学英语水平考试中口试部分的评分规范化与评分员培训,对如何规范评分以提高口试信度这一问题进行讨论。     

Rater Cognition: Implications for Validity

The scoring process is critical in the validation of tests that rely on constructed responses. Documenting that readers carry out the scoring in ways consistent with the construct and measurement goals is an important aspect of score validity. In this article, rater cognition is approached as a source of support for a validity argument for scores based on constructed responses, whether such scores are to be used on their own or as the basis for other scoring processes, for example, automated scoring.     

Legal and Psychometric Criteria for Evaluating Teacher Certification Tests

Will our teacher certification testing program stand up to a court challenge? What are the critical issues the courts consider in examining teacher certification testing? What do we need to do to strengthen our program legally and psychometrically?     

工程管理专业教育与执业资格认证一体化研究

随着我国职业资格的不断深入推行,高等学历教育与执业资格之间的衔接问题日渐突出。根据工程管理行业发展的实际情况,工程管理领域的高等学历教育与执业资格认证完全可以有机结合起来,为此政府、行业协会、学校三个层面都应认真思考实现学历教育与执业资格认证衔接的思路和对策。就学校层面而言,可以引入"本科学历教育+执业资格认证教育"培养模式,实现两证的一体化。     

教师评分员与非教师评分员对口语评分的对比研究

通过对比分析10位教师评分员和10位非教师评分员,对30位考生的口语故事复述进行评分,利用t-检验和FACETS分析发现:在任务简单的评分工作中,非教师评分员和教师评分员一样可信、有效。     

A Critical Review of Some Qualitative Research Methods Used to Explore Rater Cognition

Internationally, many assessment systems rely predominantly on human raters to score examinations. Arguably, this facilitates the assessment of multiple sophisticated educational constructs, strengthening assessment validity. It can introduce subjectivity into the scoring process, however, engendering threats to accuracy. The present objectives are to examine some key qualitative data collection methods used internationally to research this potential trade‐off, and to consider some theoretical contexts within which the methods are usable. Self‐report methods such as Kelly's Repertory Grid, think aloud, stimulated recall, and the NASA task load index have yielded important insights into the competencies needed for scoring expertise, as well as the sequences of mental activity that scoring typically involves. Examples of new data and of recent studies are used to illustrate these methods’ strengths and weaknesses. This investigation has significance for assessment designers, developers and administrators. It may inform decisions on the methods’ applicability in American and other rater cognition research contexts.     

“Digging for Gold” or “Sticking to the Criteria”: Teachers’ Rationales When Serving as Professional Raters

ABSTRACT

This paper reports findings from a project called “The National Panel of Raters” (NPR) that took place within a writing test programme in Norway (2010–2016). A recent research project found individual differences between the raters in the NPR. This paper reports results from an explorative follow up-study where 63 NPR members were surveyed with 23 items that were dilemma-like in the sense that deviating from the NPR rules would follow another—but socially acceptable—rationale. Four NPR members participated in a follow-up interview in which they motivated why they had agreed or disagreed with certain items. The results indicate two distinctly different stances toward rating work, with one stance threatening the validity of the scoring process.     

专项培训加认证的应用型人才培养模式改革与实践

为了合理兼顾通才培养和专才培养,培养应用型人才的新建本科院校提出了在通才教育平台上实施专项专才教育的培养模式,即将专项培训和认证纳入人才培养方案,形成理论教学、实验实习、专项培训相互支撑、协调促进的体系,增强学生的应用性色彩,提高学生的竞争能力。     

Rater strategies for reaching agreement on pupil text quality

Novice members of a Norwegian national rater panel tasked with assessing Year 8 pupils’ written texts were studied during three successive preparation sessions (2011–2012). The purpose was to investigate how the raters successfully make use of different decision-making strategies in an assessment situation where pre-set criteria and standards give a rather strict framework. The data sources were the raters’ pair assessment dialogues. The analysis shows that the raters use a ‘shared standards strategy’, but when reaching agreement on text quality they also seem to make very good use of assessment strategies related to their work as writing teachers. Moreover, asymmetries in knowledge and participation among raters contribute to creating an image of writing assessment as a challenging hermeneutic practice. It is suggested that future rater preparation would gain from being attentive to the internalised assessment practices teachers bring to the fore when working as raters.     

Exploring the Impact of Rater Effects on Person Fit in Rater-Mediated Assessments

Researchers have documented the impact of rater effects, or raters’ tendencies to give different ratings than would be expected given examinee achievement levels, in performance assessments. However, the degree to which rater effects influence person fit, or the reasonableness of test-takers’ achievement estimates given their response patterns, has not been investigated. In rater-mediated assessments, person fit reflects the reasonableness of rater judgments of individual test-takers’ achievement over components of the assessment. This study illustrates an approach to visualizing and evaluating person fit in assessments that involve rater judgment using rater-mediated person response functions (rm-PRFs). The rm-PRF approach allows analysts to consider the impact of rater effects on person fit in order to identify individual test-takers for whom the assessment results may not have a straightforward interpretation. A simulation study is used to evaluate the impact of rater effects on person fit. Results indicate that rater effects can compromise the interpretation and use of performance assessment results for individual test-takers. Recommendations are presented that call researchers and practitioners to supplement routine psychometric analyses for performance assessments (e.g., rater reliability checks) with rm-PRFs to identify students whose ratings may have compromised interpretations as a result of rater effects, person misfit, or both.     

论创新能力培养和工程教育专业认证之间的关系

文章详细论证了创新能力培养和工程教育专业认证之间的关系,指出专业认证中的各项要求将创新能力的培养具体化、可操作化。因此,经过工程教育认证的专业从客观上具备了全面、系统培养学生创新能力的条件。     

面向工程教育专业认证的通信工程一流专业建设

工程教育专业认证是工程教育质量的重要保障制度。以满足工程教育专业认证要求为目标,从修订人才培养方案、优化课程体系、推进实践教学改革、加强师资队伍建设和专业建设质量监控机制建设等方面推进通信工程一流专业建设。     

An Investigation of Rater Cognition in the Assessment of Projects

In the United Kingdom, the majority of national assessments involve human raters. The processes by which raters determine the scores to award are central to the assessment process and affect the extent to which valid inferences can be made from assessment outcomes. Thus, understanding rater cognition has become a growing area of research in the United Kingdom. This study investigated rater cognition in the context of the assessment of school‐based project work for high‐stakes purposes. Thirteen teachers across three subjects were asked to “think aloud” whilst scoring example projects. Teachers also completed an internal standardization exercise. Nine professional raters across the same three subjects standardized a set of project scores whilst thinking aloud. The behaviors and features attended to were coded. The data provided insights into aspects of rater cognition such as reading strategies, emotional and social influences, evaluations of features of student work (which aligned with scoring criteria), and how overall judgments are reached. The findings can be related to existing theories of judgment. Based on the evidence collected, the cognition of teacher raters did not appear to be substantially different from that of professional raters.     

实行中小学继续教育教师资格认定制度之我见

实行中小学继续教育教师资格认定制度是保证中小学教师培训质量、实现中小学教师继续教育培养目标的有效途径。10年来,我国教师资格制度的实施和对继续教育理论研究的深入为实施中小学继续教育教师资格认定制度提供了实践和理论支持。因此,应尽早建立独立的继续教育教师资格认证机构,设立科学合理的继续教育教师资格认定标准,建立开放的继续教育教师资格认定机制。     

轮机部船员适任证书培训实践性教学探究

轮机部船员适任证书培训在实践性教学方面(师资力量、培训设备等)存在不少问题,实践性教学的组织与实施有待改善。以教材的编写、因人施教、轮机英语听力的培训及考核评估方式等方面探讨了应对方法与措施。     

工程教育专业认证标准的特点

在高等教育国际化的趋势下,现代工程教育专业认证呈现出一些共同的特点:强调工程教育的发展应与社会的需求相适应;强调最低的准入标准;过程性与结果性评价标准相结合。在制定工程教育专业认证标准时应考虑高等教育质量观、高等教育传统及其管理体制、社会对工程教育的需求以及工程教育自身的发展水平等因素。     

全纳教育教师资格认定制度探微

我国全纳教育教师队伍建设存在的诸多问题限制了全纳教育的发展,其中全纳学校教师资格缺乏必要的认定,极大地影响了全纳教育的实施,但他国的成功经验和国内个别地区的试点说明,我国进行全纳教育教师资格认定是可行的。从国际上来看,全纳教育教师资格认定主要包括单证式和双证式两种模式,我国要促进全纳教育的发展,必须做好全纳教育教师资格认定的制度保障。     

数字证书技术的分析

结合多种网络验证方法(如基本验证、要验证、集成Windows验证和表单验证等)，提高计算机网络应用可靠性。     

试论生态旅游认证体系的建立和作用

在传统旅游业面临严峻的绿色环保批判的形势下,可持续旅游和生态旅游产品市场应运而生,认证体系作为一种对可持续旅游或生态旅游能清楚定位并提供有效运作的方法,具有提高企业形象和促进市场影响力的“双赢”性,已经被众多的旅游部门、组织、机构自愿或竞争性地接受。针对目前国内外生态旅游发展状况,本文对5个主要生态旅游认证机构（Green Globe21,Certificate for Sustainable Tourism,Green Deal,Smart Voyager,and Fair Trade Tourism in South Africa）在发展中国家的发展情况进行简单对比分析。