A Second-Order Longitudinal Model for Binary Outcomes: Item Response Theory Versus Structural Equation Modeling

Measuring academic growth, or change in aptitude, relies on longitudinal data collected across multiple measurements. The National Educational Longitudinal Study (NELS:88) is among the earliest, large-scale, educational surveys tracking students’ performance on cognitive batteries over 3 years. Notable features of the NELS:88 data set, and of almost all repeated measures educational assessments, are (a) the outcome variables are binary or at least categorical in nature; and (b) a set of different items is given at each measurement occasion with a few anchor items to fix the measurement scale. This study focuses on the challenges related to specifying and fitting a second-order longitudinal model for binary outcomes, within both the item response theory and structural equation modeling frameworks. The distinctions between and commonalities shared between these two frameworks are discussed. A real data analysis using the NELS:88 data set is presented for illustration purposes.     

Modeling Dynamic Functional Neuroimaging Data Using Structural Equation Modeling

The aims of this study were to present a method for developing a path analytic network model using data acquired from positron emission tomography. Regions of interest within the human brain were identified through quantitative activation likelihood estimation meta-analysis. Using this information, a “true” or population path model was then developed using Bayesian structural equation modeling. To evaluate the impact of sample size on parameter estimation bias, proportion of parameter replication coverage, and statistical power, a 2 group (clinical/control) × 6 (sample size: N = 10, N = 15, N = 20, N = 25, N = 50, N = 100) Markov chain Monte Carlo study was conducted. Results indicate that using a sample size of less than N = 15 per group will produce parameter estimates exhibiting bias greater than 5% and statistical power below .80.     

Comparing the Fit of Item Response Theory and Factor Analysis Models

Linear factor analysis (FA) models can be reliably tested using test statistics based on residual covariances. We show that the same statistics can be used to reliably test the fit of item response theory (IRT) models for ordinal data (under some conditions). Hence, the fit of an FA model and of an IRT model to the same data set can now be compared. When applied to a binary data set, our experience suggests that IRT and FA models yield similar fits. However, when the data are polytomous ordinal, IRT models yield a better fit because they involve a higher number of parameters. But when fit is assessed using the root mean square error of approximation (RMSEA), similar fits are obtained again. We explain why. These test statistics have little power to distinguish between FA and IRT models; they are unable to detect that linear FA is misspecified when applied to ordinal data generated under an IRT model.     

Effect Analysis Using Nonlinear Structural Equation Mixture Modeling

In this article, we present an approach for comprehensive analysis of the effectiveness of interventions based on nonlinear structural equation mixture models (NSEMM). We provide definitions of average and conditional effects and show how they can be computed. We extend the traditional moderated regression approach to include latent continous and discrete (mixture) variables as well as their higher order interactions, quadratic or more general nonlinear relationships. This new approach can be considered a combination of the recently proposed EffectLiteR approach and the NSEMM approach. A key advantage of this synthesis is that it gives applied researchers the opportunity to gain greater insight into the effectiveness of the intervention. For example, it makes it possible to consider structural equation models for situations where the treatment is noneffective for extreme values of a latent covariate but is effective for medium values, as we illustrate using an example from the educational sciences.     

Using Fixed Thresholds with Grouped Data in Structural Equation Modeling

Valuable methods have been developed for incorporating ordinal variables into structural equation models using a latent response variable formulation. However, some model parameters, such as the means and variances of latent factors, can be quite difficult to interpret because the latent response variables have an arbitrary metric. This limitation can be particularly problematic in growth models, where the means and variances of the latent growth parameters typically have important substantive meaning when continuous measures are used. However, these methods are often applied to grouped data, where the ordered categories actually represent an interval-level variable that has been measured on an ordinal scale for convenience. The method illustrated in this article shows how category threshold values can be incorporated into the model so that interpretation is more meaningful, with particular emphasis given to the application of this technique with latent growth models.     

Testing the Equivalence Among Different Item Response Formats in Personality Measurement: A Structural Equation Modeling Approach

This study discusses a procedure for testing the equivalence among different item response formats used in personality and attitude measurement. The procedure is based on the assumption that latent response variables underlie the observed item responses (underlying variables approach) and uses a nested series of confirmatory factor analysis models derived from Joreskog's (1971) method for estimating the dissatenuated correlation. The different stages of the procedure are illustrated using real data.     

A Bayesian Approach to Multilevel Structural Equation Modeling With Continuous and Dichotomous Outcomes

Multilevel Structural equation models are most often estimated from a frequentist framework via maximum likelihood. However, as shown in this article, frequentist results are not always accurate. Alternatively, one can apply a Bayesian approach using Markov chain Monte Carlo estimation methods. This simulation study compared estimation quality using Bayesian and frequentist approaches in the context of a multilevel latent covariate model. Continuous and dichotomous variables were examined because it is not yet known how different types of outcomes—most notably categorical—affect parameter recovery in this modeling context. Within the Bayesian estimation framework, the impact of diffuse, weakly informative, and informative prior distributions were compared. Findings indicated that Bayesian estimation may be used to overcome convergence problems and improve parameter estimate bias. Results highlight the differences in estimation quality between dichotomous and continuous variable models and the importance of prior distribution choice for cluster-level random effects.     

基于项目反应理论的公开招聘考试的最优题型选择

本研究基于项目反应理论,探索题目变动的公开招聘考试的最优题型。利用《北京市新进人员通用能力考试》专业技术岗位1 000名考生成绩,通过探索性因素分析保证仅包含一个维度的情况下,使用项目反应理论等级反应模型分析10个题型的性能。先将各个题型不同题目的得分加和,将不同得分的频数转换为等级,分别计算区分度、难度、类别反应曲线和信息函数。最优题型用两种方法确定,一是选取信息量占比高于均值的题型,二是排除各种参数达不到常用标准的题型。两种方法得到非常接近的结果,即逻辑推理、图表解读、短文加工、阅读理解四个题型最优。     

Modeling Latent Growth Curves With Incomplete Data Using Different Types of Structural Equation Modeling and Multilevel Software

This article offers different examples of how to fit latent growth curve (LGC) models to longitudinal data using a variety of different software programs (i.e., LISREL, Mx, Mplus, AMOS, SAS). The article shows how the same model can be fitted using both structural equation modeling and multilevel software, with nearly identical results, even in the case of models of latent growth fitted to incomplete data. The general purpose of this article is to provide a demonstration that integrates programming features from different software. The most immediate goal is to help researchers implement these LGC models as a useful way to test hypotheses of growth.     

Variance Estimation Using Replication Methods in Structural Equation Modeling With Complex Sample Data

This article discusses replication sampling variance estimation techniques that are often applied in analyses using data from complex sampling designs: jackknife repeated replication, balanced repeated replication, and bootstrapping. These techniques are used with traditional analyses such as regression, but are currently not used with structural equation modeling (SEM) analyses. This article provides an extension of these methods to SEM analyses, including a proposed adjustment to the likelihood ratio test, and presents the results from a simulation study suggesting replication estimates are robust. Finally, a demonstration of the application of these methods using data from the Early Childhood Longitudinal Study is included. Secondary analysts can undertake these more robust methods of sampling variance estimation if they have access to certain SEM software packages and data management packages such as SAS, as shown in the article.     

Sample Invariance of the Structural Equation Model and the Item Response Model: A Case Study

The sample invariance of item discrimination statistics is evaluated in this case study using real data. The hypothesized superiority of the item response model (IRM) is tested against structural equation modeling (SEM) for responses to the Center for Epidemiologic Studies-Depression (CES-D) scale. Responses from 10 random samples of 500 people were drawn from a base sample of 6,621 participants across gender, age, and different health groups. Hierarchical tests of multiple-group structural equation models indicated statistically significant differences exist in item regressions across contrast groups. Although the IRM item discrimination estimates were most stable in all conditions of this case study, additional research on the precision of individual scores and possible item bias is required to support the validity of either model for scoring the CES-D. The SEM approach to examining between-group differences holds promise for any field where heterogeneous populations are assessed and important consequences arise from score interpretations.     

Improving the Measurement of School Climate Using Item Response Theory

The U.S. government has become increasingly focused on school climate, as recently evidenced by its inclusion as an accountability indicator in the Every Student Succeeds Act. Yet, there remains considerable variability in both conceptualizing and measuring school climate. To better inform the research and practice related to school climate and its measurement, we leveraged item response theory (IRT), a commonly used psychometric approach for the design of achievement assessments, to create a parsimonious measure of school climate that operates across varying individual characteristics. Students (n = 69,513) in 111 secondary schools completed a school climate assessment focused on three domains of climate (i.e., safety, engagement, and environment), as defined by the U.S. Department of Education. Item and test characteristics were estimated using the mirt package in R using unidimensional IRT. Analyses revealed measurement difficulties that resulted in a greater ability to assess less favorable perspectives on school climate. Differential item functioning analyses indicated measurement differences based on student academic success. These findings support the development of a broad measure of school climate but also highlight the importance of work to ensure precision in measuring school climate, particularly when considering use as an accountability measure.     

Evaluating Interventions with Multimethod Data: A Structural Equation Modeling Approach

In many intervention and evaluation studies, outcome variables are assessed using a multimethod approach comparing multiple groups over time. In this article, we show how evaluation data obtained from a complex multitrait–multimethod–multioccasion–multigroup design can be analyzed with structural equation models. In particular, we show how the structural equation modeling approach can be used to (a) handle ordinal items as indicators, (b) test measurement invariance, and (c) test the means of the latent variables to examine treatment effects. We present an application to data from an evaluation study of an early childhood prevention program. A total of 659 children in intervention and control groups were rated by their parents and teachers on prosocial behavior and relational aggression before and after the program implementation. No mean change in relational aggression was found in either group, whereas an increase in prosocial behavior was found in both groups. Advantages and limitations of the proposed approach are highlighted.     

Estimating Non-Normal Latent Trait Distributions within Item Response Theory Using True and Estimated Item Parameters

Item response theory (IRT) procedures have been used extensively to study normal latent trait distributions and have been shown to perform well; however, less is known concerning the performance of IRT with non-normal latent trait distributions. This study investigated the degree of latent trait estimation error under normal and non-normal conditions using four latent trait estimation procedures and also evaluated whether the test composition, in terms of item difficulty level, reduces estimation error. Most importantly, both true and estimated item parameters were examined to disentangle the effects of latent trait estimation error from item parameter estimation error. Results revealed that non-normal latent trait distributions produced a considerably larger degree of latent trait estimation error than normal data. Estimated item parameters tended to have comparable precision to true item parameters, thus suggesting that increased latent trait estimation error results from latent trait estimation rather than item parameter estimation.     

Using Multidimensional Item Response Theory to Evaluate Educational and Psychological Tests

Many educational and psychological tests are inherently multidimensional, meaning these tests measure two or more dimensions or constructs. The purpose of this module is to illustrate how test practitioners and researchers can apply multidimensional item response theory (MIRT) to understand better what their tests are measuring, how accurately the different composites of ability are being assessed, and how this information can be cycled back into the test development process. Procedures for conducting MIRT analyses–from obtaining evidence that the test is multidimensional, to modeling the test as multidimensional, to illustrating the properties of multidimensional items graphically-are described from both a theoretical and a substantive basis. This module also illustrates these procedures using data from a ninth-grade mathematics achievement test. It concludes with a discussion of future directions in MIRT research.     

Classification Consistency and Accuracy for Complex Assessments Using Item Response Theory

In this article, procedures are described for estimating single-administration classification consistency and accuracy indices for complex assessments using item response theory (IRT). This IRT approach was applied to real test data comprising dichotomous and polytomous items. Several different IRT model combinations were considered. Comparisons were also made between the IRT approach and two non-IRT approaches including the Livingston-Lewis and compound multinomial procedures. Results for various IRT model combinations were not substantially different. The estimated classification consistency and accuracy indices for the non-IRT procedures were almost always lower than those for the IRT procedures.     

Comparisons of Structural Equation Modeling and Hierarchical Linear Modeling Approaches to Couples' Data

This article compares 2 statistical approaches for the analysis of data obtained from married couples. The article summarizes a current multilevel (or hierarchical) model that has demonstrated considerable utility in marital research; it also extends this formulation in several respects. This model is then respecified into a more familiar structural equation modeling (SEM) formulation, highlighting the similarities and the differences in the 2 approaches. Cross-sectional data on 348 American married couples is used to examine the influence of age, duration of marriage, and number of children on marital satisfaction. Results of the 2 sets of analyses yielded nearly identical findings. The strengths and possible extensions of the SEM approach are discussed.