The Poor Fit of Model Fit for Selecting Number of Factors in Exploratory Factor Analysis for Scale Evaluation

Model fit indices are being increasingly recommended and used to select the number of factors in an exploratory factor analysis. Growing evidence suggests that the recommended cutoff values for common model fit indices are not appropriate for use in an exploratory factor analysis context. A particularly prominent problem in scale evaluation is the ubiquity of correlated residuals and imperfect model specification. Our research focuses on a scale evaluation context and the performance of four standard model fit indices: root mean square error of approximate (RMSEA), standardized root mean square residual (SRMR), comparative fit index (CFI), and Tucker–Lewis index (TLI), and two equivalence test-based model fit indices: RMSEAt and CFIt. We use Monte Carlo simulation to generate and analyze data based on a substantive example using the positive and negative affective schedule (N = 1,000). We systematically vary the number and magnitude of correlated residuals as well as nonspecific misspecification, to evaluate the impact on model fit indices in fitting a two-factor exploratory factor analysis. Our results show that all fit indices, except SRMR, are overly sensitive to correlated residuals and nonspecific error, resulting in solutions that are overfactored. SRMR performed well, consistently selecting the correct number of factors; however, previous research suggests it does not perform well with categorical data. In general, we do not recommend using model fit indices to select number of factors in a scale evaluation framework.     

Comparing Exploratory Structural Equation Modeling and Existing Approaches for Multiple Regression with Latent Variables

Exploratory structural equation modeling (ESEM) is an approach for analysis of latent variables using exploratory factor analysis to evaluate the measurement model. This study compared ESEM with two dominant approaches for multiple regression with latent variables, structural equation modeling (SEM) and manifest regression analysis (MRA). Main findings included: (1) ESEM in general provided the least biased estimation of the regression coefficients; SEM was more biased than MRA given large cross-factor loadings. (2) MRA produced the most precise estimation, followed by ESEM and then SEM. (3) SEM was the least powerful in the significance tests; statistical power was lower for ESEM than MRA with relatively small target-factor loadings, but higher for ESEM than MRA with relatively large target-factor loadings. (4) ESEM showed difficulties in convergence and occasionally created an inflated type I error rate under some conditions. ESEM is recommended when non-ignorable cross-factor loadings exist.     

Exploratory and Confirmatory Factor Analysis of the School Level Environment Questionnaire (SLEQ)

The purpose of the present study was to validate an existing school environment instrument, the School Level Environment Questionnaire (SLEQ). The SLEQ consists of 56 items, with seven items in each of eight scales. One thousand, one hundred and six (1106) teachers in 59 elementary schools in a southwestern USA public school district completed the instrument. An exploratory factor analysis was undertaken for a random sample of half of the completed surveys. Using principal axis factoring with oblique rotation, this analysis suggested that 13 items should be dropped and that the remaining 43 items could best be represented by seven rather than eight factors. A confirmatory factor analysis was run with the other half of the original sample using structural equation modeling. Examination of the fit indices indicated that the model came close to fitting the data, with goodness-of-fit (GOF) coefficients just below recommended levels. A second model was then run with two of the seven factors, with their associated items removed. That left five factors with 35 items. Model fit was improved. A third model was tried, using the same five factors with 35 items but with correlated residuals between some of the items within a factor. This model seemed to fit the data well, with GOF coefficients in recommended ranges. These results led to a refined, more parsimonious version of the SLEQ that was then used in a larger study. Future research is needed to see if this model would fit other samples in different elementary schools and in secondary schools both in the USA and in other countries. This revised version was published online in July 2006 with corrections to the Cover Date.     

Exploratory Graph Analysis for Factor Retention: Simulation Results for Continuous and Binary Data

Exploratory graph analysis (EGA) is a commonly applied technique intended to help social scientists discover latent variables. Yet, the results can be influenced by the methodological decisions the researcher makes along the way. In this article, we focus on the choice regarding the number of factors to retain: We compare the performance of the recently developed EGA with various traditional factor retention criteria. We use both continuous and binary data, as evidence regarding the accuracy of such criteria in the latter case is scarce. Simulation results, based on scenarios resulting from varying sample size, communalities from major factors, interfactor correlations, skewness, and correlation measure, show that EGA outperforms the traditional factor retention criteria considered in most cases in terms of bias and accuracy. In addition, we show that factor retention decisions for binary data are preferably made using Pearson, instead of tetrachoric, correlations, which is contradictory to popular belief.     

Factor Retention in Exploratory Factor Analysis With Missing Data

Determining the number of factors in exploratory factor analysis is arguably the most crucial decision a researcher faces when conducting the analysis. While several simulation studies exist that compare various so-called factor retention criteria under different data conditions, little is known about the impact of missing data on this process. Hence, in this study, we evaluated the performance of different factor retention criteria—the Factor Forest, parallel analysis based on a principal component analysis as well as parallel analysis based on the common factor model and the comparison data approach—in combination with different missing data methods, namely an expectation-maximization algorithm called Amelia, predictive mean matching, and random forest imputation within the multiple imputations by chained equations (MICE) framework as well as pairwise deletion with regard to their accuracy in determining the number of factors when data are missing. Data were simulated for different sample sizes, numbers of factors, numbers of manifest variables (indicators), between-factor correlations, missing data mechanisms and proportions of missing values. In the majority of conditions and for all factor retention criteria except the comparison data approach, the missing data mechanism had little impact on the accuracy and pairwise deletion performed comparably well as the more sophisticated imputation methods. In some conditions, especially small-sample cases and when comparison data were used to determine the number of factors, random forest imputation was preferable to other missing data methods, though. Accordingly, depending on data characteristics and the selected factor retention criterion, choosing an appropriate missing data method is crucial to obtain a valid estimate of the number of factors to extract.     

Comparing Multilevel and Classical Confirmatory Factor Analysis Parameterizations of Multirater Data: A Monte Carlo Simulation Study

This simulation study assesses the statistical performance of two mathematically equivalent parameterizations for multitrait–multimethod data with interchangeable raters—a multilevel confirmatory factor analysis (CFA) and a classical CFA parameterization. The sample sizes of targets and raters, the factorial structure of the trait factors, and rater missingness are varied. The classical CFA approach yields a high proportion of improper solutions under conditions with small sample sizes and indicator-specific trait factors. In general, trait factor related parameters are more sensitive to bias than other types of parameters. For multilevel CFAs, there is a drastic bias in fit statistics under conditions with unidimensional trait factors on the between level, where root mean square error of approximation (RMSEA) and χ2 distributions reveal a downward bias, whereas the between standardized root mean square residual is biased upwards. In contrast, RMSEA and χ2 for classical CFA models are severely upwardly biased in conditions with a high number of raters and a small number of targets.     

Exploratory Factor Analysis (EFA) Programs in R

We provide a brief overview of two R packages that can conduct exploratory factor analysis (EFA): psych and EFAutilities. After introducing EFA and the exemplar data used in this paper we discuss best practices for EFA. Next, we describe the approaches used in the two packages for EFA. During this explanation, we provide sample code and discuss the usage and results of two empirical datasets. Finally, we highlight the similarities and distinctions of each package on modeling EFA.     

On Fitting a Multivariate Two-Part Latent Growth Model

The purpose of this study was to explore the influence of the number of targets specified on the quality of exploratory factor analysis solutions with a complex underlying structure and incomplete substantive measurement theory. Three Monte Carlo studies were performed based on the ratio of the number of observed variables to the number of underlying factors. Within each study, communality, sample size, and the number of targets were manipulated. Outcomes included a measure of congruence and a measure of variability with regard to the rotated pattern matrix. The magnitude of the main effect for the influence of the number of targets on congruence and variability ranged from moderate to large. The magnitude of the interaction between the number of targets and level of communality ranged from small to moderate with regard to congruence and variability. Consistent with theoretical expectations, the minimum number of targets to specify to be reasonably assured of obtaining an accurate solution varied across study conditions.     

The Asymptotic Covariance Matrix and its Use in Simulation Studies

The asymptotic performance of structural equation modeling tests and standard errors are influenced by two factors: the model and the asymptotic covariance matrix Γ of the sample covariances. Although most simulation studies clearly specify model conditions, specification of Γ is usually limited to values of univariate skewness and kurtosis. We illustrate that marginal skewness and kurtosis are not sufficient to adequately specify a nonnormal simulation condition by showing that asymptotic standard errors and test statistics vary substantially among distributions with skewness and kurtosis that are identical. We argue therefore that Γ should be reported when presenting the design of simulation studies. We show how Γ can be exactly calculated under the widely used Vale–Maurelli transform. We suggest plotting the elements of Γ and reporting the eigenvalues associated with the test statistic. R code is provided.     

Reporting Monte Carlo Studies in Structural Equation Modeling

In structural equation modeling, Monte Carlo simulations have been used increasingly over the last two decades, as an inventory from the journal Structural Equation Modeling illustrates. Reaching out to a broad audience, this article provides guidelines for reporting Monte Carlo studies in that field. The framework of discourse is set by a number of steps to be taken in such research, matching outlines of experimental design by Paxton, Curran, Bollen, Kirby, and Chen (2001) Chen, F., Bollen, K. A., Paxton, P., Curran, P. J. and Kirby, J. 2001. Improper solutions in structural equation modeling: Causes, consequences, and strategies. Sociological Methods & Research, 29: 468–508. [Crossref], [Web of Science ®] , [Google Scholar] and Skrondal (2000) Skrondal, A. 2000. Design and analysis of Monte Carlo experiments: Attacking the conventional wisdom. Multivariate Behavioral Research, 35: 137–167. [Taylor & Francis Online], [Web of Science ®] , [Google Scholar]. Throughout the article, reference is made to exemplary publications and, occasionally, to imperfect reporting.     

A Systematic Evaluation and Comparison Between Exploratory Structural Equation Modeling and Bayesian Structural Equation Modeling

In this study, we contrast two competing approaches, not previously compared, that balance the rigor of CFA/SEM with the flexibility to fit realistically complex data. Exploratory SEM (ESEM) is claimed to provide an optimal compromise between EFA and CFA/SEM. Alternatively, a family of three Bayesian SEMs (BSEMs) replace fixed-zero estimates with informative, small-variance priors for different subsets of parameters: cross-loadings (CL), residual covariances (RC), or CLs and RCs (CLRC). In Study 1, using three simulation studies, results showed that (1) BSEM-CL performed more closely to ESEM; (2) BSEM-CLRC did not provide more accurate model estimation compared with BSEM-CL; (3) BSEM-RC provided unstable estimation; and (4) different specifications of targeted values in ESEM and informative priors in BSEM have significant impacts on model estimation. The real data analysis (Study 2) showed that the differences in estimation between different models were largely consistent with those in Study1 but somewhat smaller.     

A Comparison of CFA,ESEM, and BSEM in Test Structure Analysis

Minor cross-loadings on non-targeted factors are often found in psychological or other instruments. Forcing them to zero in confirmatory factor analyses (CFA) leads to biased estimates and distorted structures. Alternatively, exploratory structural equation modeling (ESEM) and Bayesian structural equation modeling (BSEM) have been proposed. In this research, we compared the performance of the traditional independent-clusters-confirmatory-factor-analysis (ICM-CFA), the nonstandard CFA, ESEM with the Geomin- or Target-rotations, and BSEMs with different cross-loading priors (correct; small- or large-variance priors with zero mean) using simulated data with cross-loadings. Four factors were considered: the number of factors, the size of factor correlations, the cross-loading mean, and the loading variance. Results indicated that ICM-CFA performed the worst. ESEMs were generally superior to CFAs but inferior to BSEM with correct priors that provided the precise estimation. BSEM with large- or small-variance priors performed similarly while the prior mean for cross-loadings was more important than the prior variance.     

Factor structure of the WISC‐V in four standardization age groups: Exploratory and hierarchical factor analyses with the 16 primary and secondary subtests

This study examined the factor structure of the Wechsler Intelligence Scale for Children‐Fifth Edition (WISC‐V) with four standardization sample age groups (6–8, 9–11, 12–14, 15–16 years) using exploratory factor analysis (EFA), multiple factor extraction criteria, and hierarchical EFA not included in the WISC‐V Technical and Interpretation Manual. Factor extraction criteria suggested that one to four factors might be sufficient despite the publisher‐promoted, five‐factor solution. Forced extraction of five factors resulted in only one WISC‐V subtest obtaining a salient pattern coefficient on the fifth factor in all four groups, rendering it inadequate. Evidence did not support the publisher's desire to split Perceptual Reasoning into separate Visual Spatial and Fluid Reasoning dimensions. Results indicated that most WISC‐V subtests were properly associated with the four theoretically oriented first‐order factors resembling the WISC‐IV, the g factor accounted for large portions of total and common variance, and the four first‐order group factors accounted for small portions of total and common variance. Results were consistent with EFA of the WISC‐V total standardization sample.     

Estimating Latent Variable Interactions with the Unconstrained Approach: A Comparison of Methods to Form Product Indicators for Large,Unequal Numbers of Items

This Monte Carlo simulation study investigated methods of forming product indicators for the unconstrained approach for latent variable interaction estimation when the exogenous factors are measured by large and unequal numbers of indicators. Product indicators were created based on multiplying parcels of the larger scale by indicators of the smaller scale, multiplying the three most reliable indicators of each scale matched by reliability, and matching items by reliability to create as many product indicators as the number of indicators of the smallest scale. The unconstrained approach was compared with the latent moderated structural equations (LMS) approach. All methods considered provided unbiased parameter estimates. Unbiased standard errors were obtained in all conditions with the LMS approach and when the sample size was large with the unconstrained approach. Power levels to test the latent interaction and Type I error rates were similar for all methods but slightly better for the LMS approach.     

Testing Time-Invariance of Variable Specificity in Repeated Measure Designs Using Structural Equation Modeling

A structural equation modeling method for examining time-invariance of variable specificity in longitudinal studies with multiple measures is outlined, which is developed within a confirmatory factor-analytic framework. The approach represents a likelihood ratio test for the hypothesis of stability in the specificity part of the residual term associated with repeated administration of each measure. The procedure can be used in the search for parsimonious versions of multiwave multiple-indicator models, to test for variable specificity in them, and to examine assumptions underlying particular parameter estimation procedures in repeated measure designs. The outlined method is illustrated with empirical data.     

On Common Factor and Principal Component Representations of Data: Implications for Theory and for Confirmatory Replications

Common factor analysis (FA) and principal component analysis (PCA) are commonly used to obtain lower-dimensional representations of matrices of correlations among manifest variables. Whereas some experts argue that differences in results from use of FA and PCA are small and relatively unimportant in empirical studies, the fundamental rationales for the two methods are very different. Here, FA and PCA are contrasted on four key issues: the range of possible dimensional loadings, the range of potential correlations among dimensions, the structure of residual covariances and correlations, and the relation between population parameters and the correlational structures with which they are associated. For decades, experts have emphasized indeterminacies of the FA model, particularly indeterminacy of common factor scores. Determinate in most respects, a heretofore unacknowledged, pernicious indeterminacy of PCA is demonstrated: the indeterminacy between PCA structural representations and the correlational structures from which they are derived. Researchers are often advised to use either FA or PCA in exploratory rounds of data analysis to understand and refine the dimensional structure of a domain before moving to Structural Equation Modeling in later theory-testing, confirmatory, replication studies. Results from the current study suggest that PCA is an unreliable method to use for such purposes and may lead to serious misrepresentation of the structure of a domain. Hence, PCA should never be used if the goal is to understand and represent the latent structure of a domain; only FA techniques should be used for this purpose, as only FA provides reliable structural representations as the basis for confirmatory tests in future studies.     

Effects of Missing Data Methods in SEM Under Conditions of Incomplete and Nonnormal Data

Using Monte Carlo simulations, this research examined the performance of four missing data methods in SEM under different multivariate distributional conditions. The effects of four independent variables (sample size, missing proportion, distribution shape, and factor loading magnitude) were investigated on six outcome variables: convergence rate, parameter estimate bias, MSE of parameter estimates, standard error coverage, model rejection rate, and model goodness of fit—RMSEA. A three-factor CFA model was used. Findings indicated that FIML outperformed the other methods in MCAR, and MI should be used to increase the plausibility of MAR. SRPI was not comparable to the other three methods in either MCAR or MAR.     

The Misspecification of the Covariance Structures in Multilevel Models for Single-Case Data: A Monte Carlo Simulation Study

The impact of misspecifying covariance matrices at the second and third levels of the three-level model is evaluated. Results indicate that ignoring existing covariance has no effect on the treatment effect estimate. In addition, the between-case variance estimates are unbiased when covariance is either modeled or ignored. If the research interest lies in the between-study variance estimate, including at least 30 studies is warranted. Modeling covariance does not result in less biased between-study variance estimates as the between-study covariance estimate is biased. When the research interest lies in the between-case covariance, the model including covariance results in unbiased between-case variance estimates. The three-level model appears to be less appropriate for estimating between-study variance if fewer than 30 studies are included.