Investigating the Sensitivity of Goodness-of-Fit Indices to Detect Measurement Invariance in a Bifactor Model

A Monte Carlo simulation study was conducted to evaluate the sensitivities of the likelihood ratio test and five commonly used delta goodness-of-fit (ΔGOF) indices (i.e., ΔGamma, ΔMcDonald’s, ΔCFI, ΔRMSEA, and ΔSRMR) to detect a lack of metric invariance in a bifactor model. Experimental conditions included factor loading differences, location and number of noninvariant items, and sample size. The results indicated all ΔGOF indices held Type I error to a minimum and overall had adequate power for the study. For detecting the violation of metric invariance, only ΔGamma and ΔCFI, in addition to Δχ², are recommended to use in the bifactor model with values of ?.016 to ?.023 and ?.003 to ?.004, respectively. Moreover, in the variance component analysis, the magnitude of the factor loading differences contributed the most variation to all ΔGOF indices, whereas sample size affected Δχ² the most.     

Random Permutation Testing Applied to Measurement Invariance Testing with Ordered-Categorical Indicators

We describe and evaluate a random permutation test of measurement invariance with ordered-categorical data. To calculate a p-value for the observed (?)χ², an empirical reference distribution is built by repeatedly shuffling the grouping variable, then saving the χ² from a configural model, or the ?χ² between configural and scalar-invariance models, fitted to each permuted dataset. The current gold standard in this context is a robust mean- and variance-adjusted ?χ² test proposed by Satorra (2000), which yields inflated Type I errors, particularly when thresholds are asymmetric, unless samples sizes are quite large (Bandalos, 2014; Sass et al., 2014). In a Monte Carlo simulation, we compare permutation to three implementations of Satorra’s robust χ² across a variety of conditions evaluating configural and scalar invariance. Results suggest permutation can better control Type I error rates while providing comparable power under conditions that the standard robust test yields inflated errors.     

Sensitivity of Goodness of Fit Indexes to Lack of Measurement Invariance

Two Monte Carlo studies were conducted to examine the sensitivity of goodness of fit indexes to lack of measurement invariance at 3 commonly tested levels: factor loadings, intercepts, and residual variances. Standardized root mean square residual (SRMR) appears to be more sensitive to lack of invariance in factor loadings than in intercepts or residual variances. Comparative fit index (CFI) and root mean square error of approximation (RMSEA) appear to be equally sensitive to all 3 types of lack of invariance. The most intriguing finding is that changes in fit statistics are affected by the interaction between the pattern of invariance and the proportion of invariant items: when the pattern of lack of invariance is uniform, the relation is nonmonotonic, whereas when the pattern of lack of invariance is mixed, the relation is monotonic. Unequal sample sizes affect changes across all 3 levels of invariance: Changes are bigger when sample sizes are equal rather than when they are unequal. Cutoff points for testing invariance at different levels are recommended.     

Teacher's Corner: Testing Measurement Invariance of Second-Order Factor Models

We illustrate testing measurement invariance in a second-order factor model using a quality of life dataset (n = 924). Measurement invariance was tested across 2 groups at a set of hierarchically structured levels: (a) configural invariance, (b) first-order factor loadings, (c) second-order factor loadings, (d) intercepts of measured variables, (e) intercepts of first-order factors, (f) disturbances of first-order factors, and (g) residual variances of observed variables. Given that measurement invariance at the factor loading and intercept levels was achieved, the latent factor mean difference on the higher order factor between the groups was also estimated. The analyses were performed on the mean and covariance structures within the framework of the confirmatory factor analysis using the LISREL 8.51 program. Implications of second-order factor models and measurement invariance in psychological research were discussed.     

教育技术学实验研究工具的跨时测量恒等检测

基于跨时测量恒等视角与知识图谱分析,文章对我国教育技术学较常探讨的变量"自我效能"量表进行了工具检测,并以四川省某小学三年级的197名学生为被试,前后测时间间隔为6个月。文章采用结构方程模型的跨时测量恒等检验程序,依序针对不同恒等程度的模型进行比较,结果发现:数学自我效能量表不符合完全的度量恒等,放宽两道题项的参数限制后可达到部分的纯量恒等,但仍不及严格恒等的要求;跨时测量恒等性的结果会影响配对样本t检验的结论。基于此,文章提出建议:为了提升实验的内在效度,较长时间的实验研究应纳入工具的跨时测量恒等性检验。     

Testing Measurement Invariance in the Target Rotated Multigroup Exploratory Factor Model

We propose a method to investigate measurement invariance in the multigroup exploratory factor model, subject to target rotation. We consider both oblique and orthogonal target rotation. This method has clear advantages over other approaches, such as the use of congruence measures. We demonstrate that the model can be implemented readily in the freely available Mx program. We present the results of 2 illustrative analyses, one based on artificial data, and the other on real data relating to personality in male and female psychology students.     

Measurement Invariance Testing with Many Groups: A Comparison of Five Approaches

With the increasing use of international survey data especially in cross-cultural and multinational studies, establishing measurement invariance (MI) across a large number of groups in a study is essential. Testing MI over many groups is methodologically challenging, however. We identified 5 methods for MI testing across many groups (multiple group confirmatory factor analysis, multilevel confirmatory factor analysis, multilevel factor mixture modeling, Bayesian approximate MI testing, and alignment optimization) and explicated the similarities and differences of these approaches in terms of their conceptual models and statistical procedures. A Monte Carlo study was conducted to investigate the efficacy of the 5 methods in detecting measurement noninvariance across many groups using various fit criteria. Generally, the 5 methods showed reasonable performance in identifying the level of invariance if an appropriate fit criterion was used (e.g., Bayesian information criteron with multilevel factor mixture modeling). Finally, general guidelines in selecting an appropriate method are provided.     

Measurement Invariance Testing Across Between-Level Latent Classes Using Multilevel Factor Mixture Modeling

This simulation study examines the efficacy of multilevel factor mixture modeling (ML FMM) for measurement invariance testing across unobserved groups when the groups are at the between level of multilevel data. To this end, latent classes are generated with class-specific item parameters (i.e., factor loading and intercept) across the between-level classes. The efficacy of ML FMM is evaluated in terms of class enumeration, class assignment, and the detection of noninvariance. Various classification criteria such as Akaike’s information criterion, Bayesian information criterion, and bootstrap likelihood ratio tests are examined for the correct enumeration of between-level latent classes. For the detection of measurement noninvariance, free and constrained baseline approaches are compared with respect to true positive and false positive rates. This study evidences the adequacy of ML FMM. However, its performance heavily depends on the simulation factors such as the classification criteria, sample size, and the magnitude of noninvariance. Practical guidelines for applied researchers are provided.     

Testing Measurement Invariance Across Groups in Longitudinal Data: Multigroup Second-Order Latent Growth Model

In latent growth modeling, measurement invariance across groups has received little attention. Considering that a group difference is commonly of interest in social science, a Monte Carlo study explored the performance of multigroup second-order latent growth modeling (MSLGM) in testing measurement invariance. True positive and false positive rates in detecting noninvariance across groups in addition to bias estimates of major MSLGM parameters were investigated. Simulation results support the suitability of MSLGM for measurement invariance testing when either forward or iterative likelihood ratio procedure is applied.     

Testing Factorial Invariance With Unbalanced Samples

In testing the factorial invariance of a measure across groups, the groups are often of different sizes. Large imbalances in group size might affect the results of factorial invariance studies and lead to incorrect conclusions of invariance because the fit function in multiple-group factor analysis includes a weighting by group sample size. The implication is that violations of invariance might not be detected if the sample sizes of the 2 groups are severely unbalanced. In this study, we examined the effects of group size differences on results of factorial invariance tests, proposed a subsampling method to address unbalanced sample size issue in factorial invariance studies, and evaluated the proposed approach in various simulation conditions. Our findings confirm that violations of invariance might be masked in the case of severely unbalanced group size conditions and support the use of the proposed subsampling method to obtain accurate results for invariance studies.     

Testing Measurement Invariance Across Unobserved Groups: The Role of Covariates in Factor Mixture Modeling

Factor mixture modeling (FMM) has been increasingly used to investigate unobserved population heterogeneity. This study examined the issue of covariate effects with FMM in the context of measurement invariance testing. Specifically, the impact of excluding and misspecifying covariate effects on measurement invariance testing and class enumeration was investigated via Monte Carlo simulations. Data were generated based on FMM models with (1) a zero covariate effect, (2) a covariate effect on the latent class variable, and (3) covariate effects on both the latent class variable and the factor. For each population model, different analysis models that excluded or misspecified covariate effects were fitted. Results highlighted the importance of including proper covariates in measurement invariance testing and evidenced the utility of a model comparison approach in searching for the correct specification of covariate effects and the level of measurement invariance. This approach was demonstrated using an empirical data set. Implications for methodological and applied research are discussed.     

Collapsing Categorical Variables and Measurement Invariance

Cross-cultural comparisons of latent variable means demands equivalent loadings and intercepts or thresholds. Although equivalence generally emphasizes items as originally designed, researchers sometimes modify response options in categorical items. For example, substantive research interests drive decisions to reduce the number of item categories. Further, categorical multiple-group confirmatory factor analysis (MG-CFA) methods generally require that the number of indicator categories is equal across groups; however, categories with few observations in at least one group can cause challenges. In the current paper, we examine the impact of collapsing ordinal response categories in MG-CFA. An empirical analysis and a complementary simulation study suggested meaningful impacts on model fit due to collapsing categories. We also found reduced scale reliability, measured as a function of Fisher’s information. Our findings further illustrated artifactual fit improvement, pointing to the possibility of data dredging for improved model-data consistency in challenging invariance contexts with large numbers of groups.     

Evaluating Model Fit With Ordered Categorical Data Within a Measurement Invariance Framework: A Comparison of Estimators

A paucity of research has compared estimation methods within a measurement invariance (MI) framework and determined if research conclusions using normal-theory maximum likelihood (ML) generalizes to the robust ML (MLR) and weighted least squares means and variance adjusted (WLSMV) estimators. Using ordered categorical data, this simulation study aimed to address these queries by investigating 342 conditions. When testing for metric and scalar invariance, Δχ² results revealed that Type I error rates varied across estimators (ML, MLR, and WLSMV) with symmetric and asymmetric data. The Δχ² power varied substantially based on the estimator selected, type of noninvariant indicator, number of noninvariant indicators, and sample size. Although some the changes in approximate fit indexes (ΔAFI) are relatively sample size independent, researchers who use the ΔAFI with WLSMV should use caution, as these statistics do not perform well with misspecified models. As a supplemental analysis, our results evaluate and suggest cutoff values based on previous research.     

Assessing Measurement Invariance in Multiple-Group Latent Profile Analysis

The study of measurement invariance in latent profile analysis (LPA) indicates whether the latent profiles differ across known subgroups (e.g., gender). The purpose of the present study was to examine the impact of noninvariance on the relative bias of LPA parameter estimates and on the ability of the likelihood ratio test (LRT) and information criteria statistics to reject the hypothesis of invariance. A Monte Carlo simulation study was conducted in which noninvariance was defined as known group differences in the indicator means in each profile. Results indicated that parameter estimates were biased in conditions with medium and large noninvariance. The LRT and AIC detected noninvariance in most conditions with small sample sizes, while the BIC and adjusted BIC needed larger sample sizes to detect noninvariance. Implications of the results are discussed along with recommendations for future research.     

Measurement Invariance of Socioeconomic Status Across Migrational Background

Socioeconomic status (SES) is often used as control variable when relations between academic outcomes and students' migrational background are investigated. When measuring SES, indicators used must have the same meaning across groups. This study aims to examine the measurement invariance of SES, using data from TIMSS, 2003. The study shows that a latent SES variable has the same meaning across sub-populations with Swedish and non-Swedish background. However, the assumption of scalar invariance was rejected, which is essential for estimation of differences in latent means between groups. Comparisons between models assuming different degrees of scalar invariance indicated that models allowing partial scalar invariance should not be used when comparing latent variable means across groups of students with different migrational backgrounds.     

Measurement Invariance Across Groups in Latent Growth Modeling

This Monte Carlo study investigated the impacts of measurement noninvariance across groups on major parameter estimates in latent growth modeling when researchers test group differences in initial status and latent growth. The average initial status and latent growth and the group effects on initial status and latent growth were investigated in terms of Type I error and bias. The location and magnitude of noninvariance across groups was related to the location and magnitude of bias and Type I error in the parameter estimates. That is, noninvariance in factor loadings and intercepts was associated with the Type I error inflation and bias in the parameter estimates of the slope factor (or latent growth) and the intercept factor (or initial status), respectively. As noninvariance became large, the degree of Type I error and bias also increased. On the other hand, a correctly specified second-order latent growth model yielded unbiased parameter estimates and correct statistical inferences. Other findings and implications on future studies were discussed.     

Model Misspecification and Invariance Testing Using Confirmatory Factor Analytic Procedures

Confirmatory factor analytic procedures are routinely implemented to provide evidence of measurement invariance. Current lines of research focus on the accuracy of common analytic steps used in confirmatory factor analysis for invariance testing. However, the few studies that have examined this procedure have done so with perfectly or near perfectly fitting models. In the present study, the authors examined procedures for detecting simulated test structure differences across groups under model misspecification conditions. In particular, they manipulated sample size, number of factors, number of indicators per factor, percentage of a lack of invariance, and model misspecification. Model misspecification was introduced at the factor loading level. They evaluated three criteria for detection of invariance, including the chi-square difference test, the difference in comparative fit index values, and the combination of the two. Results indicate that misspecification was associated with elevated Type I error rates in measurement invariance testing.     

Measurement Invariance of a School Readiness Screener for Use in Preschool and Kindergarten

Research Findings: Public policy has increasingly focused on expansion of preschool access for underserved students and systematic evaluation of preschool quality and students’ readiness for school. However, such evaluation is limited by a lack of thoroughly validated assessments for use with preschool populations. The present study examined the measurement and structural invariance of the Kindergarten Student Entrance Profile (KSEP) across kindergarten and prekindergarten groups to evaluate its potential use across developmental groups. Participants included 522 kindergarten and 548 prekindergarten students in central California. Invariance was tested by fitting a series of multiple-groups confirmatory factor analysis models with parameter constraints across groups. Results indicated that measurement and structural parameters of the KSEP were invariant across kindergarten and prekindergarten groups. Prekindergarten means on both Social–Emotional Readiness and Cognitive Readiness were significantly lower than kindergarten means. Practice or Policy: These results suggest that the KSEP may potentially be used with prekindergarten students to assess school readiness and inform intervention before kindergarten entry.     

Perceived Competence Scale for Children: Testing for Factorial Validity and Invariance Across Age and Ability

Given the policy imperative of using multiple measures for state education accountability under the No Child Left Behind Act (NCLB), this study examines similarities and discrepancies between the National Assessment of Educational Progress (NAEP) and the states' own math assessment results in Kentucky and Maine, with a focus on 3 major academic performance indicators: proficiency level, achievement gap, and achievement gain. Using meta-analytic techniques, the study synthesizes multiple measures from the two states over the periods of 1992–1996 and 2000–2003. It pinpoints the areas and degrees of the discrepancies and explores contributing factors. It also reports emerging convergence of the NAEP and state assessments under the NCLB.