Using Testlet Response Theory to Examine Local Dependence in C-Tests

C-tests are gap-filling tests widely used to assess general language proficiency for purposes of placement, screening, or provision of feedback to language learners. C-tests consist of several short texts in which parts of words are missing. We addressed the issue of local dependence in C-tests using an explicit modeling approach based on testlet response theory. Data were provided by a total sample of 4,708 participants working on eight C-test texts with 20 gaps each. The resulting parameter estimates were compared to those obtained from (a) a polytomous item response theory (IRT) model and (b) a standard IRT model that ignored the dependence structure. Testlet effects proved to be very small and correspondence between parameters obtained from the different modeling approaches was high. When local dependence was ignored reliability of the C-test was slightly overestimated. Implications for the analysis of testlets in general, and C-tests in particular are discussed.     

An Extension of IRT-Based Equating to the Dichotomous Testlet Response Theory Model

ABSTRACT

Current procedures for equating number-correct scores using traditional item response theory (IRT) methods assume local independence. However, when tests are constructed using testlets, one concern is the violation of the local item independence assumption. The testlet response theory (TRT) model is one way to accommodate local item dependence. This study proposes methods to extend IRT true score and observed score equating methods to the dichotomous TRT model. We also examine the impact of local item dependence on equating number-correct scores when a traditional IRT model is applied. Results of the study indicate that when local item dependence is at a low level, using the three-parameter logistic model does not substantially affect number-correct equating. However, when local item dependence is at a moderate or high level, using the three-parameter logistic model generates larger equating bias and standard errors of equating compared to the TRT model. However, observed score equating is more robust to the violation of the local item independence assumption than is true score equating.     

Estimation Methods for One‐Parameter Testlet Models

This study demonstrated the equivalence between the Rasch testlet model and the three‐level one‐parameter testlet model and explored the Markov Chain Monte Carlo (MCMC) method for model parameter estimation in WINBUGS. The estimation accuracy from the MCMC method was compared with those from the marginalized maximum likelihood estimation (MMLE) with the expectation‐maximization algorithm in ConQuest and the sixth‐order Laplace approximation estimation in HLM6. The results indicated that the estimation methods had significant effects on the bias of the testlet variance and ability variance estimation, the random error in the ability parameter estimation, and the bias in the item difficulty parameter estimation. The Laplace method best recovered the testlet variance while the MMLE best recovered the ability variance. The Laplace method resulted in the smallest random error in the ability parameter estimation while the MCMC method produced the smallest bias in item parameter estimates. Analyses of three real tests generally supported the findings from the simulation and indicated that the estimates for item difficulty and ability parameters were highly correlated across estimation methods.     

Dual Supervision: A Model for Counseling and Supervision

The counseling practicum is generally accepted as an essential part of a counselor-training program. Likewise, the training of future counselor educators necessitates experiences in supervision of counseling-practicum students. The dual supervision model described in this article is an approach that simultaneously meets the training needs of future counselors and counselor educators. The rationale for the dual supervision model, the results of an exploratory study of the counselor trainees' attitudes toward the dual supervision process, and the counseling-practicum supervisors' reactions to the model are described. Both the students enrolled in their second quarter of counseling practicum and the supervisors had very positive reactions to the dual supervision model.     

工程教育"双师型"教师培养的实践取向与路径

基于复杂性思维、实践哲学和缄默知识理论的启示,本文主张“双师型”教师培养坚持实践取向。“双师型”教师培养具体路径是:①要系统学习教育理论知识,提高教师的教育素养;②要立足当下的教师“教育生活”,进行教育实践与反思,提高教师的教育智慧;③要走向企业,参与企业的科学研究,培养教师的实践能力和创新能力。     

Item Response Models for Local Dependence Among Multiple Ratings

Ratings given to the same item response may have a stronger correlation than those given to different item responses, especially when raters interact with one another before giving ratings. The rater bundle model was developed to account for such local dependence by forming multiple ratings given to an item response as a bundle and assigning fixed‐effect parameters to describe response patterns in the bundle. Unfortunately, this model becomes difficult to manage when a polytomous item is graded by more than two raters. In this study, by adding random‐effect parameters to the facets model, we propose a class of generalized rater models to account for the local dependence among multiple ratings and intrarater variation in severity. A series of simulations was conducted with the freeware WinBUGS to evaluate parameter recovery of the new models and consequences of ignoring the local dependence or intrarater variation in severity. The results revealed a good parameter recovery when the data‐generating models were fit, and a poor estimation of parameters and test reliability when the local dependence or intrarater variation in severity was ignored. An empirical example is provided.     

Teacher Morale in Western Australia: A Multilevel Model

As part of a larger longitudinal study of Western Australian high schools, 212 teachers were surveyed about their perceptions of the school in which they teach and the management of that school. In particular, teachers responded to questions about the school environment, their morale and the organisational health of the school, and they were asked some self-concept and teaching efficacy questions. A multilevel model was used to investigate the effects of these and other school characteristics on teacher morale (the dependent variable), while accounting for school level differences. This study demonstrated that teacher morale varied between teachers and between schools, with school environment explaining 54% of the variability in morale. That is, when the school environment was positive, teacher morale was higher. The implications of this finding point to the importance of both improving the school environment and teacher morale in order to enhance the health of schools. This is only achieved when teachers believe that their school is improving and has a positive climate. This revised version was published online in July 2006 with corrections to the Cover Date.     

A More Flexible Bayesian Multilevel Bifactor Item Response Theory Model

Multilevel bifactor item response theory (IRT) models are commonly used to account for features of the data that are related to the sampling and measurement processes used to gather those data. These models conventionally make assumptions about the portions of the data structure that represent these features. Unfortunately, when data violate these models' assumptions but these models are used anyway, incorrect conclusions about the cluster effects could be made and potentially relevant dimensions could go undetected. To address the limitations of these conventional models, a more flexible multilevel bifactor IRT model that does not make these assumptions is presented, and this model is based on the generalized partial credit model. Details of a simulation study demonstrating this model outperforming competing models and showing the consequences of using conventional multilevel bifactor IRT models to analyze data that violate these models' assumptions are reported. Additionally, the model's usefulness is illustrated through the analysis of the Program for International Student Assessment data related to interest in science.     

Scaling Performance Assessments: Strategies for Managing Local Item Dependence

Performance assessments appear on a priori grounds to be likely to produce far more local item dependence (LID) than that produced in the use of traditional multiple-choice tests. This article (a) defines local item independence, (b) presents a compendium of causes of LID, (c) discusses some of LID's practical measurement implications, (d) details some empirical results for both performance assessments and multiple-choice tests, and (e) suggests some strategies for managing LID in order to avoid negative measurement consequences.     

论“土政策”的两种价值层面

“土政策”按照价值取向不同,可以分为私利层面和公利层面。两不能被混为一谈。其主要区别在于目的性及产生的原因不同。要解决“土政策”问题,既不能只抓政策的制定,又不能只抓政策执行,而只有将二有机结合起来考虑,才能根除“土政策”痼疾。     

局部对偶平坦且局部射影平坦的Finsler度量

研究了局部对偶平坦的Finsler度量,综合局部射影平坦,局部对偶平坦的性质,得到一个Finsler度量是局部对偶平坦且局部射影平坦的三个充要条件,并将其应用到(α,β)-度量,从而得到(α,β)-度量是局部对偶平坦且局部射影平坦的充要条件.     

基于“二元”管理模式的地方高校行业学院建设研究

行业学院是产教深度融合的一种教学组织模式,也是破解人才培养与企业需求相脱节的重要举措。针对应用型人才培养存在的问题,分析了行业学院建设的必要性,指出行业学院的建设应确保企业的主体地位,照顾企业的利益关切,同时兼顾学术性和应用性。基于这一观点,提出了一种新型的二元管理模式的行业学院组织结构。按照这种结构,行业学院和学部在高校并行存在,行业学院是培养学生的主体,负责专业规划和学生管理,学部是学术研究的主体,负责学科建设和教师管理。二者通过聘约关系和高校的考评机制发生联系,成为高校不可分割的部门,共同为人才培养服务,在保证学生培养符合产业需求的同时,又能促进高校学科建设围绕地方经济进行,真正推动学科与专业的一体化协调发展。     

Detecting Local Item Dependence in Polytomous Adaptive Data

A rapidly expanding arena for item response theory (IRT) is in attitudinal and health‐outcomes survey applications, often with polytomous items. In particular, there is interest in computer adaptive testing (CAT). Meeting model assumptions is necessary to realize the benefits of IRT in this setting, however. Although initial investigations of local item dependence have been studied both for polytomous items in fixed‐form settings and for dichotomous items in CAT settings, there have been no publications applying local item dependence detection methodology to polytomous items in CAT despite its central importance to these applications. The current research uses a simulation study to investigate the extension of widely used pairwise statistics, Yen's Q₃ Statistic and Pearson's Statistic X², in this context. The simulation design and results are contextualized throughout with a real item bank of this type from the Patient‐Reported Outcomes Measurement Information System (PROMIS).     

Parameter Recovery and Classification Accuracy Under Conditions of Testlet Dependency: A Comparison of the Traditional 2PL,Testlet, and Bi-Factor Models

ABSTRACT

Testlets, or groups of related items, are commonly included in educational assessments due to their many logistical and conceptual advantages. Despite their advantages, testlets introduce complications into the theory and practice of educational measurement. Responses to items within a testlet tend to be correlated even after controlling for latent ability, which violates the assumption of conditional independence made by traditional item response theory models. The present study used Monte Carlo simulation methods to evaluate the effects of testlet dependency on item and person parameter recovery and classification accuracy. Three calibration models were examined, including the traditional 2PL model with marginal maximum likelihood estimation, a testlet model with Bayesian estimation, and a bi-factor model with limited-information weighted least squares mean and variance adjusted estimation. Across testlet conditions, parameter types, and outcome criteria, the Bayesian testlet model outperformed, or performed equivalently to, the other approaches.     

Evaluating Model Fit and Structural Coefficient Bias: A Bayesian Approach to Multilevel Bifactor Model Misspecification

Appropriate model specification is fundamental to unbiased parameter estimates and accurate model interpretations in structural equation modeling. Thus detecting potential model misspecification has drawn the attention of many researchers. This simulation study evaluates the efficacy of the Bayesian approach (the posterior predictive checking, or PPC procedure) under multilevel bifactor model misspecification (i.e., ignoring a specific factor at the within level). The impact of model misspecification on structural coefficients was also examined in terms of bias and power. Results showed that the PPC procedure performed better in detecting multilevel bifactor model misspecification, when the misspecification became more severe and sample size was larger. Structural coefficients were increasingly negatively biased at the within level, as model misspecification became more severe. Model misspecification at the within level affected the between-level structural coefficient estimates more when data dependency was lower and the number of clusters was smaller. Implications for researchers are discussed.     

Formal Relations and an Empirical Comparison among the Bi-Factor,the Testlet,and a Second-Order Multidimensional IRT Model

Testlet effects can be taken into account by incorporating specific dimensions in addition to the general dimension into the item response theory model. Three such multidimensional models are described: the bi-factor model, the testlet model, and a second-order model. It is shown how the second-order model is formally equivalent to the testlet model. In turn, both models are constrained bi-factor models. Therefore, the efficient full maximum likelihood estimation method that has been established for the bi-factor model can be modified to estimate the parameters of the two other models. An application on a testlet-based international English assessment indicated that the bi-factor model was the preferred model for this particular data set.     

Assessing the Sensitivity of Weighted Least Squares Model Fit Indexes to Local Dependence in Item Response Theory Models

Given the relationships of item response theory (IRT) models to confirmatory factor analysis (CFA) models, IRT model misspecifications might be detectable through model fit indexes commonly used in categorical CFA. The purpose of this study is to investigate the sensitivity of weighted least squares with adjusted means and variance (WLSMV)-based root mean square error of approximation, comparative fit index, and Tucker–Lewis Index model fit indexes to IRT models that are misspecified due to local dependence (LD). It was found that WLSMV-based fit indexes have some functional relationships to parameter estimate bias in 2-parameter logistic models caused by violations of LD. Continued exploration into these functional relationships and development of LD-detection methods based on such relationships could hold much promise for providing IRT practitioners with global information on violations of local independence.     

Student Mobility in Multilevel Growth Modeling: A Multiple Membership Piecewise Growth Model

This study proposes a new model, termed the multiple membership piecewise growth model (MM-PGM), to handle individual mobility across clusters frequently encountered in longitudinal studies, especially in educational research wherein some students could attend multiple schools during the course of the study. A real data set containing some students who switched elementary schools was used to demonstrate and explain the MM-PGM. Parameter and model fit differences were compared between the MM-PGM and two other techniques for handling student mobility: the first school-PGM, which only used school membership at the first measurement occasion, and the delete-PGM, which removed mobile students from the analysis. Results indicated that the three approaches of handling mobile students led to different conclusions about the impact of school-level predictors of growth parameters and the school-level variability in the growth rates. Furthermore, deleting mobile students altered the impact of student-level predictors compared to the other two approaches.     

Estimation of MIMIC Model Parameters with Multilevel Data

The purpose of this simulation study was to assess the performance of latent variable models that take into account the complex sampling mechanism that often underlies data used in educational, psychological, and other social science research. Analyses were conducted using the multiple indicator multiple cause (MIMIC) model, which is a flexible and effective tool for relating observed and latent variables. The data were simulated in a hierarchical framework (e.g., individuals nested in schools) so that a multilevel modeling approach would be appropriate. Analyses were conducted accounting for and not accounting for the nested data to determine the impact of ignoring such multilevel data structures in full structural equation models. Results highlight the differences in modeling results when the analytic strategy is congruent with the data structure and what occurs when this congruency is absent. Type I error rates and power for the standard and multilevel methods were similar for within-cluster variables and for the multilevel model with between-cluster variables. However, Type I error rates were inflated for the standard approach when modeling between-cluster variables.     

依附与独立--试论《三国演义》中诸葛亮的双重文化人格

“鞠躬尽瘁，死而后已”既是对诸葛亮一生节操的高度评价，也是对他为实现理想而矢志不移、奋斗终生的一种概括。他的身上典型地体现了古代封建知识分子受儒道两家思想影响而形成的双重文化人格。这种双重文化人格，在其政治生涯中，既表现为对权势的选择与依附，又有对个体人格的保持与独立。这是封建时代杰出知识分子一种理想化的人生选择。