Effects of sample size,estimation methods,and model specification on structural equation modeling fit indexes

A Monte Carlo simulation study was conducted to investigate the effects on structural equation modeling (SEM) fit indexes of sample size, estimation method, and model specification. Based on a balanced experimental design, samples were generated from a prespecified population covariance matrix and fitted to structural equation models with different degrees of model misspecification. Ten SEM fit indexes were studied. Two primary conclusions were suggested: (a) some fit indexes appear to be noncomparable in terms of the information they provide about model fit for misspecified models and (b) estimation method strongly influenced almost all the fit indexes examined, especially for misspecified models. These 2 issues do not seem to have drawn enough attention from SEM practitioners. Future research should study not only different models vis‐à‐vis model complexity, but a wider range of model specification conditions, including correctly specified models and models specified incorrectly to varying degrees.     

Bayesian Meta-Analytic SEM: A One-Stage Approach to Modeling Between-Studies Heterogeneity in Structural Parameters

Meta-analytic structural equation modeling (MASEM) refers to a set of meta-analysis techniques for combining and comparing structural equation modeling (SEM) results from multiple studies. Existing approaches to MASEM cannot appropriately model between-studies heterogeneity in structural parameters because of missing correlations, lack model fit assessment, and suffer from several theoretical limitations. In this study, we address the major shortcomings of existing approaches by proposing a novel Bayesian multilevel SEM approach. Simulation results showed that the proposed approach performed satisfactorily in terms of parameter estimation and model fit evaluation when the number of studies and the within-study sample size were sufficiently large and when correlations were missing completely at random. An empirical example about the structure of personality based on a subset of data was provided. Results favored the third factor structure over the hierarchical structure. We end the article with discussions and future directions.     

Structural equation modeling: Back to basics

Major technological advances incorporated into structural equation modeling (SEM) computer programs now make it possible for practitioners who are basically unfamiliar with the purposes and limitations of SEM to use this tool within their research contexts. The current move by program developers to market more user friendly software packages is a welcomed trend in the social and behavioral science research community. The quest to simplify the data analysis step in the research process has—at least with regard to SEM—created a situation that allows practitioners to apply SEM but forgetting, knowingly ignoring, or most dangerously, being ignorant of some basic philosophical and statistical issues that must be addressed before sound SEM analyses should be conducted. This article focuses on some of the almost forgotten topics taken here from each step in the SEM process: model conceptualization, identification and parameter estimation, and data‐model fit assessment and model modification. The main objective is to raise awareness among researchers new to SEM of a few basic but key philosophical and statistical issues. These should be addressed before launching into any one of the new generation of SEM software packages and being led astray by the seemingly irresistible temptation to prematurely start “playing” with the data.     

The Performance of ML,GLS, and WLS Estimation in Structural Equation Modeling Under Conditions of Misspecification and Nonnormality

This simulation study demonstrates how the choice of estimation method affects indexes of fit and parameter bias for different sample sizes when nested models vary in terms of specification error and the data demonstrate different levels of kurtosis. Using a fully crossed design, data were generated for 11 conditions of peakedness, 3 conditions of misspecification, and 5 different sample sizes. Three estimation methods (maximum likelihood [ML], generalized least squares [GLS], and weighted least squares [WLS]) were compared in terms of overall fit and the discrepancy between estimated parameter values and the true parameter values used to generate the data. Consistent with earlier findings, the results show that ML compared to GLS under conditions of misspecification provides more realistic indexes of overall fit and less biased parameter values for paths that overlap with the true model. However, despite recommendations found in the literature that WLS should be used when data are not normally distributed, we find that WLS under no conditions was preferable to the 2 other estimation procedures in terms of parameter bias and fit. In fact, only for large sample sizes (N = 1,000 and 2,000) and mildly misspecified models did WLS provide estimates and fit indexes close to the ones obtained for ML and GLS. For wrongly specified models WLS tended to give unreliable estimates and over-optimistic values of fit.     

The Impact of Moderate Priors For Bayesian Estimation and Testing of Item Factor Analysis Models When Maximum Likelihood is Unsuitable

In psychological research, available data are often insufficient to estimate item factor analysis (IFA) models using traditional estimation methods, such as maximum likelihood (ML) or limited information estimators. Bayesian estimation with common-sense, moderately informative priors can greatly improve efficiency of parameter estimates and stabilize estimation. There are a variety of methods available to evaluate model fit in a Bayesian framework; however, past work investigating Bayesian model fit assessment for IFA models has assumed flat priors, which have no advantage over ML in limited data settings. In this paper, we evaluated the impact of moderately informative priors on ability to detect model misfit for several candidate indices: posterior predictive checks based on the observed score distribution, leave-one-out cross-validation, and widely available information criterion (WAIC). We found that although Bayesian estimation with moderately informative priors is an excellent aid for estimating challenging IFA models, methods for testing model fit in these circumstances are inadequate.     

A Poor Person’s Posterior Predictive Checking of Structural Equation Models

Posterior predictive model checking (PPMC) is a Bayesian model checking method that compares the observed data to (plausible) future observations from the posterior predictive distribution. We propose an alternative to PPMC in the context of structural equation modeling, which we term the poor person’s PPMC (PP-PPMC), for the situation wherein one cannot afford (or is unwilling) to draw samples from the full posterior. Using only by-products of likelihood-based estimation (maximum likelihood estimate and information matrix), the PP-PPMC offers a natural method to handle parameter uncertainty in model fit assessment. In particular, a coupling relationship between the classical p values from the model fit chi-square test and the predictive p values from the PP-PPMC method is carefully examined, suggesting that PP-PPMC might offer an alternative, principled approach for model fit assessment. We also illustrate the flexibility of the PP-PPMC approach by applying it to case-influence diagnostics.     

A Simple Analytic Confidence Interval for CFI Given Nonnormal Data

The comparative fit index (CFI) is one of the most widely-used fit indices in structural equation modeling (SEM). When applying the CFI to model evaluation, although it is universally recognized that the focus should be the population fit, in practice one often considers only the CFI value within a sample and neglects the uncertainty in point estimation. Confidence interval (CI) methods for CFI appeared only recently, but these methods assume multivariate normality, which often fails to hold in practice. In addition, the current methods are applications of the bootstrap and are thus computationally intensive. To better handle nonnormal data and simplify CI construction, in this paper we propose an analytic CI method for CFI without assuming normality. We then carry out simulation studies to compare the new and current methods at various levels of model misfit and nonnormality. Simulation results verify the effectiveness and advantages of the new method.     

A Multilevel Testlet Model for Dual Local Dependence

The applications of item response theory (IRT) models assume local item independence and that examinees are independent of each other. When a representative sample for psychometric analysis is selected using a cluster sampling method in a testlet‐based assessment, both local item dependence and local person dependence are likely to be induced. This study proposed a four‐level IRT model to simultaneously account for dual local dependence due to item clustering and person clustering. Model parameter estimation was explored using the Markov Chain Monte Carlo method. Model parameter recovery was evaluated in a simulation study in comparison with three other related models: the Rasch model, the Rasch testlet model, and the three‐level Rasch model for person clustering. In general, the proposed model recovered the item difficulty and person ability parameters with the least total error. The bias in both item and person parameter estimation was not affected but the standard error (SE) was affected. In some simulation conditions, the difference in classification accuracy between models could go up to 11%. The illustration using the real data generally supported model performance observed in the simulation study.     

Standard Errors of IRT Parameter Scale Transformation Coefficients: Comparison of Bootstrap Method,Delta Method,and Multiple Imputation Method

The present study evaluated the multiple imputation method, a procedure that is similar to the one suggested by Li and Lissitz (2004), and compared the performance of this method with that of the bootstrap method and the delta method in obtaining the standard errors for the estimates of the parameter scale transformation coefficients in item response theory (IRT) equating in the context of the common‐item nonequivalent groups design. Two different estimation procedures for the variance‐covariance matrix of the IRT item parameter estimates, which were used in both the delta method and the multiple imputation method, were considered: empirical cross‐product (XPD) and supplemented expectation maximization (SEM). The results of the analyses with simulated and real data indicate that the multiple imputation method generally produced very similar results to the bootstrap method and the delta method in most of the conditions. The differences between the estimated standard errors obtained by the methods using the XPD matrices and the SEM matrices were very small when the sample size was reasonably large. When the sample size was small, the methods using the XPD matrices appeared to yield slight upward bias for the standard errors of the IRT parameter scale transformation coefficients.     

Effects of Missing Data Methods in Structural Equation Modeling With Nonnormal Longitudinal Data

The purpose of this study is to investigate the effects of missing data techniques in longitudinal studies under diverse conditions. A Monte Carlo simulation examined the performance of 3 missing data methods in latent growth modeling: listwise deletion (LD), maximum likelihood estimation using the expectation and maximization algorithm with a nonnormality correction (robust ML), and the pairwise asymptotically distribution-free method (pairwise ADF). The effects of 3 independent variables (sample size, missing data mechanism, and distribution shape) were investigated on convergence rate, parameter and standard error estimation, and model fit. The results favored robust ML over LD and pairwise ADF in almost all respects. The exceptions included convergence rates under the most severe nonnormality in the missing not at random (MNAR) condition and recovery of standard error estimates across sample sizes. The results also indicate that nonnormality, small sample size, MNAR, and multicollinearity might adversely affect convergence rate and the validity of statistical inferences concerning parameter estimates and model fit statistics.     

Bootstrapping Confidence Intervals for Fit Indexes in Structural Equation Modeling

Bootstrapping approximate fit indexes in structural equation modeling (SEM) is of great importance because most fit indexes do not have tractable analytic distributions. Model-based bootstrap, which has been proposed to obtain the distribution of the model chi-square statistic under the null hypothesis (Bollen & Stine, 1992), is not theoretically appropriate for obtaining confidence intervals (CIs) for fit indexes because it assumes the null is exactly true. On the other hand, naive bootstrap is not expected to work well for those fit indexes that are based on the chi-square statistic, such as the root mean square error of approximation (RMSEA) and the comparative fit index (CFI), because sample noncentrality is a biased estimate of the population noncentrality. In this article we argue that a recently proposed bootstrap approach due to Yuan, Hayashi, and Yanagihara (YHY; 2007) is ideal for bootstrapping fit indexes that are based on the chi-square. This method transforms the data so that the “parent” population has the population noncentrality parameter equal to the estimated noncentrality in the original sample. We conducted a simulation study to evaluate the performance of the YHY bootstrap and the naive bootstrap for 4 indexes: RMSEA, CFI, goodness-of-fit index (GFI), and standardized root mean square residual (SRMR). We found that for RMSEA and CFI, the CIs under the YHY bootstrap had relatively good coverage rates for all conditions, whereas the CIs under the naive bootstrap had very low coverage rates when the fitted model had large degrees of freedom. However, for GFI and SRMR, the CIs under both bootstrap methods had poor coverage rates in most conditions.     

Estimation Methods for One‐Parameter Testlet Models

This study demonstrated the equivalence between the Rasch testlet model and the three‐level one‐parameter testlet model and explored the Markov Chain Monte Carlo (MCMC) method for model parameter estimation in WINBUGS. The estimation accuracy from the MCMC method was compared with those from the marginalized maximum likelihood estimation (MMLE) with the expectation‐maximization algorithm in ConQuest and the sixth‐order Laplace approximation estimation in HLM6. The results indicated that the estimation methods had significant effects on the bias of the testlet variance and ability variance estimation, the random error in the ability parameter estimation, and the bias in the item difficulty parameter estimation. The Laplace method best recovered the testlet variance while the MMLE best recovered the ability variance. The Laplace method resulted in the smallest random error in the ability parameter estimation while the MCMC method produced the smallest bias in item parameter estimates. Analyses of three real tests generally supported the findings from the simulation and indicated that the estimates for item difficulty and ability parameters were highly correlated across estimation methods.     

The Use of Incorrect Informative Priors in the Estimation of MIMIC Model Parameters with Small Sample Sizes

Recently, advancements in Bayesian structural equation modeling (SEM), particularly software developments, have allowed researchers to more easily employ it in data analysis. With the potential for greater use, come opportunities to apply Bayesian SEM in a wider array of situations, including for small sample size problems. Effective use of Bayseian estimation hinges on selection of appropriate prior distributions for model parameters. Researchers have suggested that informative priors may be useful with small samples, presuming that the mean of the prior is accurate with respect to the population mean. The purpose of this simulation study was to examine model parameter estimation for the Multiple Indicator Multiple Cause model when an informative prior distribution had an incorrect mean. Results demonstrated that the use of incorrect informative priors with somewhat larger variance than is typical, yields more accurate parameter estimates than do naïve priors, or maximum likelihood estimation. Implications for practice are discussed.     

Semiparametric Item Response Functions in the Context of Guessing

We present a logistic function of a monotonic polynomial with a lower asymptote, allowing additional flexibility beyond the three‐parameter logistic model. We develop a maximum marginal likelihood‐based approach to estimate the item parameters. The new item response model is demonstrated on math assessment data from a state, and a computationally efficient strategy for choosing the order of the polynomial is demonstrated. Finally, our approach is tested through simulations and compared to response function estimation using smoothed isotonic regression. Results indicate that our approach can result in small gains in item response function recovery and latent trait estimation.     

An evaluation of the Satorra‐Bentler distributional misspecification correction applied to the McDonald fit index

McDonald goodness‐of‐fit indices based on maximum likelihood, asymptotic distribution free, and the Satorra‐Bentler scale correction estimation methods are investigated. Sampling experiments are conducted to assess the magnitude of error for each index under variations in distributional misspecification, structural misspecification, and sample size. The Satorra‐Bentler correction‐based index is shown to have the least error under each distributional misspecification level when the model has correct structural specification. The scaled index also performs adequately when there is minor structural misspecification and distributional misspecification. However, when a model has major structural misspecification with distributional misspecification, none of the estimation methods perform adequately.     

Digital Module 06: Bayesian Psychometrics—Posterior Predictive Model Checking https://ncme.elevate.commpartners.com

Drawing valid inferences from modern measurement models is contingent upon a good fit of the data to the model. Violations of model‐data fit have numerous consequences, limiting the usefulness and applicability of the model. As Bayesian estimation is becoming more common, understanding the Bayesian approaches for evaluating model‐data fit models is critical. In this instructional module, Allison Ames and Aaron Myers provide an overview of Posterior Predictive Model Checking (PPMC), the most common Bayesian model‐data fit approach. Specifically, they review the conceptual foundation of Bayesian inference as well as PPMC and walk through the computational steps of PPMC using real‐life data examples from simple linear regression and item response theory analysis. They provide guidance for how to interpret PPMC results and discuss how to implement PPMC for other model(s) and data. The digital module contains sample data, SAS code, diagnostic quiz questions, data‐based activities, curated resources, and a glossary.     

A Regularized GLS for Structural Equation Modeling

Ill conditioning of covariance and weight matrices used in structural equation modeling (SEM) is a possible source of inadequate performance of SEM statistics in nonasymptotic samples. A maximum a posteriori (MAP) covariance matrix is proposed for weight matrix regularization in normal theory generalized least squares (GLS) estimation. Maximum likelihood (ML), GLS, and regularized GLS test statistics (RGLS and rGLS) are studied by simulation in a 15-variable, 3-factor model with 15 levels of sample size varying from 60 to 100,000. A key result showed that in terms of nominal rejection rates, RGLS outperformed ML at all sample sizes below 500, and GLS at most sample sizes below 500. In larger samples, their performance was equivalent. The second regularization methodology (rGLS) performed well asymptotically, but poorly in small samples. Regularization in SEM deserves further study.     

A Comparison of Estimation Techniques for IRT Models With Small Samples

The usefulness of item response theory (IRT) models depends, in large part, on the accuracy of item and person parameter estimates. For the standard 3 parameter logistic model, for example, these parameters include the item parameters of difficulty, discrimination, and pseudo-chance, as well as the person ability parameter. Several factors impact traditional marginal maximum likelihood (ML) estimation of IRT model parameters, including sample size, with smaller samples generally being associated with lower parameter estimation accuracy, and inflated standard errors for the estimates. Given this deleterious impact of small samples on IRT model performance, use of these techniques with low-incidence populations, where it might prove to be particularly useful, estimation becomes difficult, especially with more complex models. Recently, a Pairwise estimation method for Rasch model parameters has been suggested for use with missing data, and may also hold promise for parameter estimation with small samples. This simulation study compared item difficulty parameter estimation accuracy of ML with the Pairwise approach to ascertain the benefits of this latter method. The results support the use of the Pairwise method with small samples, particularly for obtaining item location estimates.     

Why might relative fit indices differ between estimators?

Previous research indicates that relative fit indices in structural equation modeling may vary across estimation methods. Sugawara and MacCallum (1993) explained that the discrepancy arises from difference in the function values for the null model with no further derivation given. In this study, we derive explicit solutions for parameters of the null model. The null model specifies the variances of the observed variables as model parameters and fixes all the covariances to be zero. Three methods of estimation are considered: the maximum likelihood (ML) method, the ordinary least squares (OLS) method, and the generalized least squares (GLS) method. Results indicate that ML and LS yield an identical estimator, which is different from GLS. Function values and associated chi‐square statistics of the null model vary across estimation methods. Consequently, relative fit indices using the null model as the reference point in computation may yield different results depending on the estimation method chosen. An illustration example is given and implications of this study are discussed.     

Exploring the latent structure of the Luria model for the KABC‐II at school age: Further insights from confirmatory factor analysis

The present study examined the factor structure of the Luria interpretive model for the Kaufman Assessment Battery for Children‐Second Edition (KABC‐II) with normative sample participants aged 7–18 (N = 2,025) using confirmatory factor analysis with maximum‐likelihood estimation. For the eight subtest Luria configuration, an alternative higher‐order model with Pattern Reasoning being permitted to cross‐load on the Planning and Simultaneous Processing factors provided the best fit to the normative sample data. Variance apportionment suggests that additional consideration, beyond the omnibus Mental Processing Index, of the contribution of the first‐order factor‐based scores (i.e., SQ, SM, P, and L), and in some cases the individual subtests themselves, may be warranted. Implications for clinical interpretation and the anticipated normative update of the measurement instrument are discussed.