多元概化理论在儿科学临床技能考核中的应用

为明确多元概化理论对于儿科学临床技能考核的评价作用,该研究采用mG ENOVA2.1概化理论分析软件,对52名临床医学专业本科生的儿科学临床技能考核结果进行分析。结果显示,此次考核有较高的信度,考核内容设计最合理。多元概化理论能客观、科学地评价儿科学临床实践技能考核,值得进一步在临床教学中推广应用。     

基于概化理论构建自学考试信度分析框架

概化理论(GT)提出了新的测量信度观并逐渐应用于大规模考试领域。本文将概化理论思想与方法引入自学考试信度分析研究与实践,提出了自学考试信度分析的概化理论研究框架。本框架对自学考试信度分析的基本任务与流程进行了梳理与总结,对一元概化理论(UGT)模型与多元概化理论(MGT)模型的应用进行了整合,选择了更为合理的自学考试信度指标并探讨了及格线决策信度的考察思路,为自学考试研究者进行概化理论研究提供一定程度的参考。     

基于学生评教的多元概化理论分析

多元概化理论是现代教育与心理测量理论之一,对于分析学生评价教师教学水平具有一定的优势。本文采用一份非标准化评价量表,要求536名学生在20个题项上对16名教师进行评价,并采用多元嵌套不平衡设计对评价数据进行多元概化理论分析。结果发现:多元概化理论可以同时考察多种学生评价教师教学水平模式,进行评价质量分析更胜一筹,可对学生评价教师教学水平进行适度的"微观分析"。     

构建概化理论分析思路的新框架

本文对概化理论的分析思路进行程序化总结,介绍其主要步骤,即明确概化理论的研究设计、确定数据结构和概化模型、进行G研究、进行D研究;概述该框架的三个特点：对概化理论的基本概念和分析要素进行有效区分,对单变量概化分析和多元概化分析进行有机整合,对概化理论分析思路进行一定程度的创新性总结。     

用多元概化理论变革命题方法

目前测验编制中双向细目表存在明显的缺陷。三向细目表克服了双向细目表的缺陷。可从多元概化理论的角度提出和论证三向细目表的命题方法。该方法对我国心理与教育测验的研究和应用具有重要价值。     

教育与心理测评的新潮流——读《测评的概化理论及其应用》

“概化理论”(GeneralizabilityTheory,GT)是一种把误差变量作为模型参数来处理的测量理论。它的显著特点是,对于同一次测量资料,能够针对测量结果推论范围或使用目的的不同而提供多个不同的测量误差估计指标,使得人们的决策更为恰当和合理。这种现代测量理论能够同时达到区分考生与评估应考者真正实力之目的,较好地控制测评误差。由我国在该研究领域颇有建树的杨志明教授和张雷教授合著的《测评的概化理论及其应用》,是我国第一本系统研究概化理论及其应用的专著,可以说及时填补了国内此类研究的空白。该书通过将经典测验理论与概化理论…     

现代测验理论的发展及其在我国的应用

以概化理论和项目反应理论为代表的现代测验理论是在克服经典测验理论缺陷的基础上产生的。概化理论是在经典测验理论的基础上,引入实验设计和方差分析技术,对测评情境中的各类误差进行分解和控制的一种现代测量理论,其发展主要经历了一元概化理论和多元概化理论两个阶段。目前,其应用主要集中在评价、考试和评定量表编制三个领域。项目反应理论是在克服经典测验理论题目参数等指标的变异性基础上发展起来的一种现代测验理论,其发展经历了早期理论探索、理论初步形成和理论逐渐完善三个阶段。它主要用于处理分数等值和测验项目参数、测验和项目的质量的分析,剥离测验情境中评委特征对测验结果的影响,以及测查项目功能差异、编制适应性测验等。     

基于科学知识图谱的国内外概化理论研究可视化分析

研究对比国内外概化理论研究的异同,为国内概化理论研究发展提供参考。基于CNKI、Webof Science数据库,分别选取2001—2020年中文373篇、外文734篇文献,借助CiteSpace软件定量分析国内外研究的发文数量、主题内容和合作关系。结果表明,国内外概化理论研究呈现积极增长态势;美国与加拿大对概化理论研究关注度一直保持较高水平,与世界各国合作紧密;国内的概化理论研究深度略低于国外,作者及机构之间的合作较分散,应加强与国际的交流与合作。     

多元概化理论在教育教学能力测试中的应用

使用多元概化理论对教育教学能力测试实测数据进行了分析。结果表明:1.教育教学能力测试将说课、答辩和面试成绩进行合成是比较合理的;2.三个测评任务中说课和答辩的评分质量较好,面试评分的质量较差;3.总体测试结果较适合于相对决策,做绝对决策时要谨慎处理;4.影响教育教学能力测试质量的主要原因是评分者宽严程度不一;5.增加评分员可以提高测试的精度,但增幅递减,当评分员人数为5时能够较好满足测试要求。     

心理学专业基础综合考试的多元概化理论研究

全国硕士研究生入学心理学专业基础综合考试是为高等院校和科研院所招收心理学学科的硕士研究生而设置的具有选拔性质的统一入学考试科目。本文用多元概化理论评估2007年心理学专业基础综合考试的总体信度、试卷结构、二级学科比例分配的合理性,为今后改进测验质量提供参考依据。     

The Influence of Several Factors on Reliability for Complex Reading Comprehension Tests

The purpose of this study was to investigate the effects of items, passages, contents, themes, and types of passages on the reliability and standard errors of measurement for complex reading comprehension tests. Seven different generalizability theory models were used in the analyses. Results indicated that generalizability coefficients estimated using multivariate models incorporating content strata and types of passages were similar in size to reliability estimates based upon a model that did not include these factors. In contrast, incorporating passages and themes within univariate generalizability theory models produced non-negligible differences in the reliability estimates. This suggested that passages and themes be taken into account when evaluating the reliability of test scores for complex reading comprehension tests.     

A Multivariate Generalizability Analysis of Data from a Performance Assessment of Physicians' Clinical Skills

Although multivariate generalizability theory was developed more than 30 years ago, little published research utilizing this framework exists and most of what does exist examines tests built from tables of specifications. In this context, it is assumed that the universe scores from levels of the fixed multivariate facet will be correlated, but the error terms will be uncorrelated because subscores result from mutually exclusive sets of test items. This paper reports on an application in which multiple subscores are derived from each task completed by the examinee. In this context, both universe scores and errors may be correlated across levels of the fixed multivariate facet. The data described come from the United States Medical Licensing Examination^® Step 2 Clinical Skills Examination. In this test, each examinee interacts with a series of standardized patients and each interaction results in four component scores. The paper focuses on the application of multivariate generalizability theory in this context and on the practical interpretation of the resulting estimated variance and covariance components.     

Methodology for Examining the Reliability of Group Mean Difference Scores

This article treats various procedures for examining the reliability of group mean difference scores, with particular emphasis on procedures from univariate and multivariate generalizability theory. Attention is given to both traditional norm-referenced perspectives on reliability as well as criterion-referenced perspectives that focus on error-tolerance ratios and functions of them. The procedures discussed are illustrated using three cohorts of data for third- and fourth-grade students in Iowa who took the Iowa Tests of Basic Skills in recent years. For these data, estimates of reliability for norm-referenced decisions tend to be relatively low. By contrast, for criterion-referenced decisions, estimates of reliability-like coefficients based on error-tolerance ratios tend to be noticeably larger.     

Generalizability Theory

Generalizability theory consists of a conceptual framework and a methodology that enable an investigator to disentangle multiple sources of error in a measurement procedure. The roots of generalizability theory can be found in classical test theory and analysis of variance (ANOVA), but generalizability theory is not simply the conjunction of classical theory and ANOVA. In particular, the conceptual framework in generalizability theory is unique. This framework and the procedures of generalizability theory are introduced and illustrated in this instructional module using a hypothetical scenario involving writing proficiency.     

HSK高等考试信度的多元概化理论研究

本研究运用多元概化理论对HSK高等考试客观卷的信度、试卷结构、测验总分的合成、试题预测方式的改进等问题进行探讨,结果表明:(1)HSK高等考试客观卷的总体及各部分信度都较好,且测验总分的合成是合理的;(2)测验各部分内容对全域总分方差分量的贡献比例与预设的赋分比例基本一致,试卷结构较为合理;(3)在适当减少各部分题量的情况下测验的信度仍较高,今后可以考虑在正式考试中进行试题预测。     

The Generalizability of Scores for a Performance Assessment Scored with a Computer-Automated Scoring System

When performance assessments are delivered and scored by computer, the costs of scoring may be substantially lower than those of scoring the same assessment based on expert review of the individual performances. Computerized scoring algorithms also ensure that the scoring rules are implemented precisely and uniformly. Such computerized algorithms represent an effort to encode the scoring policies of experts. This raises the question, would a different group of experts have produced a meaningfully different algorithm? The research reported in this paper uses generalizability theory to assess the impact of using independent, randomly equivalent groups of experts to develop the scoring algorithms for a set of computer‐simulation tasks designed to measure physicians’ patient management skills. The results suggest that the impact of this “expert group” effect may be significant but that it can be controlled with appropriate test development strategies. The appendix presents multivariate generalizability analysis to examine the stability of the assessed proficiency across scores representing the scoring policies of different groups of experts.     

2012年全国硕士研究生入学中医综合考试的多元概化理论研究

全国硕士研究生入学中医综合考试,是为高等院校和科研院所招收中医药学专业硕士研究生而设置的、具有选拔性质的全国统一入学考试科目。本研究运用多元概化理论评估2012年中医综合考试的总体信度、试卷结构及二级学科分配比例的合理性。结果表明:(1)从考查的学科内容看,方剂学、中药学、针灸学、中医内科、中医诊断学的测量精度较高,而中医基础理论的测量精度相对偏低,可通过适当提高该学科试题的难度和区分度以增加测量精度;(2)从设置的题型看,各题型的测量精度均较高,各题型的分量分布较适当。     

(Mis) Conception About Generalizability Theory

In what sense is generalizability theory an extension of classical test theory? In what sense is generalizability theory an application of analysis of variance.     

Occasions and the Reliability of Classroom Observations: Alternative Conceptualizations and Methods of Analysis

Student–teacher interactions are dynamic relationships that change and evolve over the course of a school year. Measuring classroom quality through observations that focus on these interactions presents challenges when observations are conducted throughout the school year. Variability in observed scores could reflect true changes in the quality of student–teacher interaction or simply reflect measurement error. Classroom observation protocols should be designed to minimize measurement error while allowing measureable changes in the construct of interest. Treating occasions as fixed multivariate outcomes allows true changes to be separated from random measurement error. These outcomes may also be summarized through trend score composites to reflect different types of growth over the school year. We demonstrate the use of multivariate generalizability theory to estimate reliability for trend score composites, and we compare the results to traditional methods of analysis. Reliability estimates computed for average, linear, quadratic, and cubic trend scores from 118 classrooms participating in the MyTeachingPartner study indicate that universe scores account for between 57% and 88% of observed score variance.