Using Fit Indexes to Select a Covariance Model for Longitudinal Data

This study investigated the performance of fit indexes in selecting a covariance structure for longitudinal data. Data were simulated to follow a compound symmetry, first-order autoregressive, first-order moving average, or random-coefficients covariance structure. We examined the ability of the likelihood ratio test (LRT), root mean square error of approximation (RMSEA), comparative fit index (CFI), and Tucker–Lewis Index (TLI) to reject misspecified models with varying degrees of misspecification. With a sample size of 20, RMSEA, CFI, and TLI are high in both Type I and Type II error rates, whereas LRT has a high Type II error rate. With a sample size of 100, these indexes generally have satisfactory performance, but CFI and TLI are affected by a confounding effect of their baseline model. Akaike's Information Criterion (AIC) and Bayesian Information Criterion (BIC) have high success rates in identifying the true model when sample size is 100. A comparison with the mixed model approach indicates that separately modeling the means and covariance structures in structural equation modeling dramatically improves the success rate of AIC and BIC.     

Performance of Nonrecursive Latent Variable Models under Misspecification

A problem central to structural equation modeling is measurement model specification error and its propagation into the structural part of nonrecursive latent variable models. Full-information estimation techniques such as maximum likelihood are consistent when the model is correctly specified and the sample size large enough; however, any misspecification within the model can affect parameter estimates in other parts of the model. The goals of this study included comparing the bias, efficiency, and accuracy of hypothesis tests in nonrecursive latent variable models with indirect and direct feedback loops. We compare the performance of maximum likelihood, two-stage least-squares and Bayesian estimators in nonrecursive latent variable models with indirect and direct feedback loops under various degrees of misspecification in small to moderate sample size conditions.     

Revisiting the Model Size Effect in Structural Equation Modeling

Fitting a large structural equation modeling (SEM) model with moderate to small sample sizes results in an inflated Type I error rate for the likelihood ratio test statistic under the chi-square reference distribution, known as the model size effect. In this article, we show that the number of observed variables (p) and the number of free parameters (q) have unique effects on the Type I error rate of the likelihood ratio test statistic. In addition, the effects of p and q cannot be fully explained using degrees of freedom (df). We also evaluated the performance of 4 correctional methods for the model size effect, including Bartlett’s (1950), Swain’s (1975), and Yuan’s (2005) corrected statistics, and Yuan, Tian, and Yanagihara’s (2015) empirically corrected statistic. We found that Yuan et al.’s (2015) empirically corrected statistic generally yields the best performance in controlling the Type I error rate when fitting large SEM models.     

Using Monte Carlo Normal Distributions to Evaluate Structural Models With Nonnormal Data

Statistical theories of goodness-of-fit tests in structural equation modeling are based on asymptotic distributions of test statistics. When the model includes a large number of variables or the population is not from a multivariate normal distribution, the asymptotic distributions do not approximate the distribution of the test statistics very well at small sample sizes. A variety of methods have been developed to improve the accuracy of hypothesis testing at small sample sizes. However, all these methods have their limitations, specially for nonnormal distributed data. We propose a Monte Carlo test that is able to control Type I error with more accuracy compared to existing approaches in both normal and nonnormally distributed data at small sample sizes. Extensive simulation studies show that the suggested Monte Carlo test has a more accurate observed significance level as compared to other tests with a reasonable power to reject misspecified models.     

Issues in applied structural equation modeling research

When using the popular structural equation modeling (SEM) methodology, the issues of sample size, method of parameter estimation, assessment of model fit, and capitalization on chance are of great importance in the process of evaluating the results of an empirical study. We focus first on implications of the large‐sample theory underlying applications of the methodology. The utility for applied contexts of the asymptotically distribution‐free parameter estimation and model testing method is discussed next. We then argue for wider use of a recently developed, non conventional model‐fit assessment strategy in SEM. We conclude by discussing the issue of capitalization on chance, primarily in situations in which exploratory and confirmatory analyses are conducted on the same data set.     

Performance of Bootstrapping Approaches to Model Test Statistics and Parameter Standard Error Estimation in Structural Equation Modeling

Though the common default maximum likelihood estimator used in structural equation modeling is predicated on the assumption of multivariate normality, applied researchers often find themselves with data clearly violating this assumption and without sufficient sample size to utilize distribution-free estimation methods. Fortunately, promising alternatives are being integrated into popular software packages. Bootstrap resampling, which is offered in AMOS (Arbuckle, 1997), is one potential solution for estimating model test statistic p values and parameter standard errors under nonnormal data conditions. This study is an evaluation of the bootstrap method under varied conditions of nonnormality, sample size, model specification, and number of bootstrap samples drawn from the resampling space. Accuracy of the test statistic p values is evaluated in terms of model rejection rates, whereas accuracy of bootstrap standard error estimates takes the form of bias and variability of the standard error estimates themselves.     

Assessing Structural Equation Models by Equivalence Testing With Adjusted Fit Indexes

Conventional null hypothesis testing (NHT) is a very important tool if the ultimate goal is to find a difference or to reject a model. However, the purpose of structural equation modeling (SEM) is to identify a model and use it to account for the relationship among substantive variables. With the setup of NHT, a nonsignificant test statistic does not necessarily imply that the model is correctly specified or the size of misspecification is properly controlled. To overcome this problem, this article proposes to replace NHT by equivalence testing, the goal of which is to endorse a model under a null hypothesis rather than to reject it. Differences and similarities between equivalence testing and NHT are discussed, and new “T-size” terminology is introduced to convey the goodness of the current model under equivalence testing. Adjusted cutoff values of root mean square error of approximation (RMSEA) and comparative fit index (CFI) corresponding to those conventionally used in the literature are obtained to facilitate the understanding of T-size RMSEA and CFI. The single most notable property of equivalence testing is that it allows a researcher to confidently claim that the size of misspecification in the current model is below the T-size RMSEA or CFI, which gives SEM a desirable property to be a scientific methodology. R code for conducting equivalence testing is provided in an appendix.     

Graphical Extension of Sample Size Planning With AIPE on RMSEA Using R

Kelley and Lai (2011) recently proposed the use of accuracy in parameter estimation (AIPE) for sample size planning in structural equation modeling. The sample size that reaches the desired width for the confidence interval of root mean square error of approximation (RMSEA) is suggested. This study proposes a graphical extension with the AIPE approach, abbreviated as GAIPE, on RMSEA to facilitate sample size planning in structural equation modeling. GAIPE simultaneously displays the expected width of a confidence interval of RMSEA, the necessary sample size to reach the desired width, and the RMSEA values covered in the confidence interval. Power analysis for hypothesis tests related to RMSEA can also be integrated into the GAIPE framework to allow for a concurrent consideration of accuracy in estimation and statistical power to plan sample sizes. A package written in R has been developed to implement GAIPE. Examples and instructions for using the GAIPE package are presented to help readers make use of this flexible framework. With the capacity of incorporating information on accuracy in RMSEA estimation, values of RMSEA, and power for hypothesis testing on RMSEA in a single graphical representation, the GAIPE extension offers an informative and practical approach for sample size planning in structural equation modeling.     

An evaluation of the Satorra‐Bentler distributional misspecification correction applied to the McDonald fit index

McDonald goodness‐of‐fit indices based on maximum likelihood, asymptotic distribution free, and the Satorra‐Bentler scale correction estimation methods are investigated. Sampling experiments are conducted to assess the magnitude of error for each index under variations in distributional misspecification, structural misspecification, and sample size. The Satorra‐Bentler correction‐based index is shown to have the least error under each distributional misspecification level when the model has correct structural specification. The scaled index also performs adequately when there is minor structural misspecification and distributional misspecification. However, when a model has major structural misspecification with distributional misspecification, none of the estimation methods perform adequately.     

A Note on Extending Steiger's (1998) Multiple Sample RMSEA Adjustment to Other Noncentrality Parameter-Based Statistics

This article considers the implications for other noncentrality parameter-based statistics from Steiger's (1998) multiple sample adjustment to the root mean square error of approximation (RMSEA) measure. When a structural equation model is fitted simultaneously in more than 1 sample, it is shown that the calculation of the noncentrality parameter used in tests of approximate fit and in point and interval estimators of other noncentral fit statistics (except the expected cross-validation index) also requires a likeminded adjustment. Furthermore, it is shown that an adjustment is needed in multiple sample models for correctly calculating MacCallum, Browne, and Sugawara's (1996) approach to power analysis. The accuracy of these proposals is investigated and demonstrated in a small Monte Carlo study in which particular attention is paid to using appropriately constructed covariance matrices that give specified nonzero population discrepancy values under maximum likelihood estimation.     

A Simulation Study of Polychoric Instrumental Variable Estimation in Structural Equation Models

Data collected from questionnaires are often in ordinal scale. Unweighted least squares (ULS), diagonally weighted least squares (DWLS) and normal-theory maximum likelihood (ML) are commonly used methods to fit structural equation models. Consistency of these estimators demands no structural misspecification. In this article, we conduct a simulation study to compare the equation-by-equation polychoric instrumental variable (PIV) estimation with ULS, DWLS, and ML. Accuracy of PIV for the correctly specified model and robustness of PIV for misspecified models are investigated through a confirmatory factor analysis (CFA) model and a structural equation model with ordinal indicators. The effects of sample size and nonnormality of the underlying continuous variables are also examined. The simulation results show that PIV produces robust factor loading estimates in the CFA model and in structural equation models. PIV also produces robust path coefficient estimates in the model where valid instruments are used. However, robustness highly depends on the validity of instruments.     

Effect of Sample Size Ratio and Model Misfit When Using the Difficulty Parameter Differences Procedure to Detect DIF

This study examined the effect of sample size ratio and model misfit on the Type I error rates and power of the Difficulty Parameter Differences procedure using Winsteps. A unidimensional 30-item test with responses from 130,000 examinees was simulated and four independent variables were manipulated: sample size ratio (20/100/250/500/1000); model fit/misfit (1 PL and 3PLc =. 15 models); impact (no difference/mean differences/variance differences/mean and variance differences); and percentage of items with uniform and nonuniform DIF (0%/10%/20%). In general, the results indicate the importance of ensuring model fit to achieve greater control of Type I error and adequate statistical power. The manipulated variables produced inflated Type I error rates, which were well controlled when a measure of DIF magnitude was applied. Sample size ratio also had an effect on the power of the procedure. The paper discusses the practical implications of these results.     

Detecting Mixtures From Structural Model Differences Using Latent Variable Mixture Modeling: A Comparison of Relative Model Fit Statistics

The accuracy of structural model parameter estimates in latent variable mixture modeling was explored with a 3 (sample size) × 3 (exogenous latent mean difference) × 3 (endogenous latent mean difference) × 3 (correlation between factors) × 3 (mixture proportions) factorial design. In addition, the efficacy of several likelihood-based statistics (Akaike's Information Criterion [AIC], Bayesian Information Ctriterion [BIC], the sample-size adjusted BIC [ssBIC], the consistent AIC [CAIC], the Vuong-Lo-Mendell-Rubin adjusted likelihood ratio test [aVLMR]), classification-based statistics (CLC [classification likelihood information criterion], ICL-BIC [integrated classification likelihood], normalized entropy criterion [NEC], entropy), and distributional statistics (multivariate skew and kurtosis test) were examined to determine which statistics best recover the correct number of components. Results indicate that the structural parameters were recovered, but the model fit statistics were not exceedingly accurate. The ssBIC statistic was the most accurate statistic, and the CLC, ICL-BIC, and aVLMR showed limited utility. However, none of these statistics were accurate for small samples (n = 500).     

tetrad: A Set of Stata Commands for Confirmatory Tetrad Analysis

This article provides a brief overview of confirmatory tetrad analysis (CTA) and presents a new set of Stata commands for conducting CTA. The tetrad command allows researchers to use model-implied vanishing tetrads to test the overall fit of structural equation models (SEMs) and the relative fit of two SEMs that are tetrad-nested. An extension of the command, tetrad_matrix, allows researchers to conduct CTA using a sample covariance matrix as input rather than relying on raw data. Researchers can also use the tetrad_matrix command to input a polychoric correlation matrix and conduct CTA for SEMs involving dichotomous, ordinal, or censored outcomes. Another extension of the command, tetrad_bootstrap, provides a bootstrapped p value for the chi-square test statistic. With Stata’s recently developed commands for structural equation modeling, researchers can integrate CTA with data preparation, likelihood ratio tests for model fit, and the estimation of model parameters in a single statistical software package.     

Measurement Bias Detection Through Factor Analysis

Assessing the correctness of a structural equation model is essential to avoid drawing incorrect conclusions from empirical research. In the past, the chi-square test was recommended for assessing the correctness of the model but this test has been criticized because of its sensitivity to sample size. As a reaction, an abundance of fit indexes have been developed. The result of these developments is that structural equation modeling packages are now producing a large list of fit measures. One would think that this progression has led to a clear understanding of evaluating models with respect to model misspecifications. In this article we question the validity of approaches for model evaluation based on overall goodness-of-fit indexes. The argument against such usage is that they do not provide an adequate indication of the “size” of the model's misspecification. That is, they vary dramatically with the values of incidental parameters that are unrelated with the misspecification in the model. This is illustrated using simple but fundamental models. As an alternative method of model evaluation, we suggest using the expected parameter change in combination with the modification index (MI) and the power of the MI test.     

Four New Corrected Statistics for SEM With Small Samples and Nonnormally Distributed Data

When the assumption of multivariate normality is violated and the sample sizes are relatively small, existing test statistics such as the likelihood ratio statistic and Satorra–Bentler’s rescaled and adjusted statistics often fail to provide reliable assessment of overall model fit. This article proposes four new corrected statistics, aiming for better model evaluation with nonnormally distributed data at small sample sizes. A Monte Carlo study is conducted to compare the performances of the four corrected statistics against those of existing statistics regarding Type I error rate. Results show that the performances of the four new statistics are relatively stable compared with those of existing statistics. In particular, Type I error rates of a new statistic are close to the nominal level across all sample sizes under a condition of asymptotic robustness. Other new statistics also exhibit improved Type I error control, especially with nonnormally distributed data at small sample sizes.     

Cognitive componential modelling of reading in ten- to twelve-year-old readers

The longitudinal project attempts to estimate the parameters of a priori postulates with their measurement errors, and the goodness-of-fit of model-implied population covariance sigma matrices with observed sample covariance matrices, using linear structural equation modelling (LISREL). The present report deals with Phase 2 of the project (see Leong 1988 for Phase 1 results) and involved 252 ten-to twelve-year-old readers. Three latent exogenous components were postulated: (a) orthographic/phonological component, (b) morphological component, and (c) sentence comprehension component. These constructs were subserved by 8 measurable indicators (5 lexical decision and 3 vocalization tasks). These 8 independent tasks were all administered on-line via the microcomputer under laboratory conditions with combined, scaled accuracy and latency measures as indices of efficiency in processing the reading related tasks. The latent endogenous or dependent component Reading Performance was subserved by standardized vocabulary and reading comprehension tests. Maximum likelihood analyses using LISREL VI to test the approximation of the hypothesized model show a good fit for the grade 4 data, a reasonable fit for grade 5, and a less unambiguous fit for grade 6. These results are generally confirmed with the simultaneous structural analyses of Phase 2 and Phase 1 (N=298 students) multisamples. Multi-components and multi-levels of reading for these age groups are emphasized.     

Conducting tests of hypotheses: The need for an adequate sample size

This article addresses the importance of obtaining a sample of an adequate size for the purpose of testing hypotheses. The logic underlying the requirement for a minimum sample size for hypothesis testing is discussed, as well as the criteria for determining it. Implications for researchers working with convenient samples of a fixed size are also considered, and suggestions are given about the steps that should be taken when they are not able to obtain a large enough sample. Finally, the implications of not having an adequate sample size for hypothesis testing are discussed to highlight the importance of determining sample size prior to conducting one’s study.     

Inferring Longitudinal Relationships Between Variables: Model Selection Between the Latent Change Score and Autoregressive Cross-Lagged Factor Models

This research focuses on the problem of model selection between the latent change score (LCS) model and the autoregressive cross-lagged (ARCL) model when the goal is to infer the longitudinal relationship between variables. We conducted a large-scale simulation study to (a) investigate the conditions under which these models return statistically (and substantively) different results concerning the presence of bivariate longitudinal relationships, and (b) ascertain the relative performance of an array of model selection procedures when such different results arise. The simulation results show that the primary sources of differences in parameter estimates across models are model parameters related to the slope factor scores in the LCS model (specifically, the correlation between the intercept factor and the slope factor scores) as well as the size of the data (specifically, the number of time points and sample size). Among several model selection procedures, correct selection rates were higher when using model fit indexes (i.e., comparative fit index, root mean square error of approximation) than when using a likelihood ratio test or any of several information criteria (i.e., Akaike’s information criterion, Bayesian information criterion, consistent AIC, and sample-size-adjusted BIC).     

The Relation Among Fit Indexes,Power, and Sample Size in Structural Equation Modeling

The relation among fit indexes, power, and sample size in structural equation modeling is examined. The noncentrality parameter is required to compute power. The 2 existing methods of computing power have estimated the noncentrality parameter by specifying an alternative hypothesis or alternative fit. These methods cannot be implemented easily and reliably. In this study, 4 fit indexes (RMSEA, CFI, McDonald's Fit Index, and Steiger's gamma) were used to compute the noncentrality parameter and sample size to achieve certain level of power. The resulting power and sample size varied as a function of (a) choice of fit index, (b) number of variables/degrees of freedom, (c) relation among the variables, and (d) value of the fit index. However, if the level of misspecification were held constant, then the resulting power and sample size would be identical.