Measuring Variability in Proctor Decision Making on High-Stakes Assessments: Improving Test Security in the Digital Age

Remote proctoring, or monitoring test takers through internet-based, video-recording software, has become critical for maintaining test security on high-stakes assessments. The main role of remote proctors is to make judgments about test takers' behaviors and decide whether these behaviors constitute rule violations. Variability in proctor decision making, or the degree to which humans/proctors make different decisions about the same test-taking behaviors, can be problematic for both test takers and test users (e.g., universities). In this paper, we measure variability in proctor decision making over time on a high-stakes English language proficiency test. Our results show that (1) proctors systematically differ in their decision making and (2) these differences are trait-like (i.e., ranging from lenient to strict), but (3) systematic variability in decisions can be reduced. Based on these findings, we recommend that test security providers conduct regular measurements of proctors’ judgments and take actions to reduce variability in proctor decision making.     

博弈论视角下的可持续发展问题研究

可持续发展是属于外部不经济的公共地问题。从历史上看,造成目前一系列全球环境问题的主要责任在发达国家,所以,在可持续发展问题上,国际社会形成了共同但有区别的责任分摊原则。通过博弈模型分析发现,目前发达国家与发展中国家之所以谈判一再陷入僵局,是因为双方合作还缺乏适当的条件和机制,可持续发展问题的妥善解决,至少需要三个先决条件,但目前还没有一个条件已经满足。     

Assessment matters—Research and learning intensive: Bowling Green State University commits to both

Ten years ago, BGSU pledged to become the premier learning community in Ohio and one of the best in the nation. With campuswide agreement and widespread assessment, the institution has made gains and has a clear sense of what it needs to do next.     

Modeling student performance in diagnostic tasks: A decade of evolution

Since 1976 the authors have been involved in an evolving research and development program focusing on the application of computer-based simulation to diagnostic training in technical environments. Eight simulation-oriented training systems are used to describe the authors' understanding of and approach to student diagnosis. Each of the systems described capitalized on experimental and/or empirical findings from previous experiences. The article describes how design decisions were driven first by research findings and then by the reality of building effective computer-based training systems within the constraints of operational environments. The paper on which this article is based was presented at the Cognitive Skill Acquisition Workshop, Air Force Armstrong Laboratory Human Resources Directorate, Brooks Air Force Base, Texas, in July 1990. It is also published in Regian, W. J., & Shute, V. J. (Eds). (1992).Cognitive approaches to automated instruction. Hillsdale, NJ: Lawrence Erlbaum.