A Test for Cluster Bias: Detecting Violations of Measurement Invariance Across Clusters in Multilevel Data

We present a test for cluster bias, which can be used to detect violations of measurement invariance across clusters in 2-level data. We show how measurement invariance assumptions across clusters imply measurement invariance across levels in a 2-level factor model. Cluster bias is investigated by testing whether the within-level factor loadings are equal to the between-level factor loadings, and whether the between-level residual variances are zero. The test is illustrated with an example from school research. In a simulation study, we show that the cluster bias test has sufficient power, and the proportions of false positives are close to the chosen levels of significance.     

Multilevel Factorial Invariance in n-Level Structural Equation Modeling (nSEM)

We present a multigroup multilevel confirmatory factor analysis (CFA) model and a procedure for testing multilevel factorial invariance in n-level structural equation modeling (nSEM). Multigroup multilevel CFA introduces a complexity when the group membership at the lower level intersects the clustered structure, because the observations in different groups but in the same cluster are not independent of one another. nSEM provides a framework in which the multigroup multilevel data structure is represented with the dependency between groups at the lower level properly taken into account. The procedure for testing multilevel factorial invariance is illustrated with an empirical example using an R package xxm2.     

Evaluating Model Fit and Structural Coefficient Bias: A Bayesian Approach to Multilevel Bifactor Model Misspecification

Appropriate model specification is fundamental to unbiased parameter estimates and accurate model interpretations in structural equation modeling. Thus detecting potential model misspecification has drawn the attention of many researchers. This simulation study evaluates the efficacy of the Bayesian approach (the posterior predictive checking, or PPC procedure) under multilevel bifactor model misspecification (i.e., ignoring a specific factor at the within level). The impact of model misspecification on structural coefficients was also examined in terms of bias and power. Results showed that the PPC procedure performed better in detecting multilevel bifactor model misspecification, when the misspecification became more severe and sample size was larger. Structural coefficients were increasingly negatively biased at the within level, as model misspecification became more severe. Model misspecification at the within level affected the between-level structural coefficient estimates more when data dependency was lower and the number of clusters was smaller. Implications for researchers are discussed.     

Alternative Methods for Assessing Mediation in Multilevel Data: The Advantages of Multilevel SEM

Multilevel modeling (MLM) is a popular way of assessing mediation effects with clustered data. Two important limitations of this approach have been identified in prior research and a theoretical rationale has been provided for why multilevel structural equation modeling (MSEM) should be preferred. However, to date, no empirical evidence of MSEM's advantages relative to MLM approaches for multilevel mediation analysis has been provided. Nor has it been demonstrated that MSEM performs adequately for mediation analysis in an absolute sense. This study addresses these gaps and finds that the MSEM method outperforms 2 MLM-based techniques in 2-level models in terms of bias and confidence interval coverage while displaying adequate efficiency, convergence rates, and power under a variety of conditions. Simulation results support prior theoretical work regarding the advantages of MSEM over MLM for mediation in clustered data.     

Testing Factorial Invariance in Multilevel Data: A Monte Carlo Study

Testing factorial invariance has recently gained more attention in different social science disciplines. Nevertheless, when examining factorial invariance, it is generally assumed that the observations are independent of each other, which might not be always true. In this study, we examined the impact of testing factorial invariance in multilevel data, especially when the dependency issue is not taken into account. We considered a set of design factors, including number of clusters, cluster size, and intraclass correlation (ICC) at different levels. The simulation results showed that the test of factorial invariance became more liberal (or had inflated Type I error rate) in terms of rejecting the null hypothesis of invariance held between groups when the dependency was not considered in the analysis. Additionally, the magnitude of the inflation in the Type I error rate was a function of both ICC and cluster size. Implications of the findings and limitations are discussed.     

Measurement Invariance Testing Across Between-Level Latent Classes Using Multilevel Factor Mixture Modeling

This simulation study examines the efficacy of multilevel factor mixture modeling (ML FMM) for measurement invariance testing across unobserved groups when the groups are at the between level of multilevel data. To this end, latent classes are generated with class-specific item parameters (i.e., factor loading and intercept) across the between-level classes. The efficacy of ML FMM is evaluated in terms of class enumeration, class assignment, and the detection of noninvariance. Various classification criteria such as Akaike’s information criterion, Bayesian information criterion, and bootstrap likelihood ratio tests are examined for the correct enumeration of between-level latent classes. For the detection of measurement noninvariance, free and constrained baseline approaches are compared with respect to true positive and false positive rates. This study evidences the adequacy of ML FMM. However, its performance heavily depends on the simulation factors such as the classification criteria, sample size, and the magnitude of noninvariance. Practical guidelines for applied researchers are provided.     

Measurement Bias Detection Through Factor Analysis

Assessing the correctness of a structural equation model is essential to avoid drawing incorrect conclusions from empirical research. In the past, the chi-square test was recommended for assessing the correctness of the model but this test has been criticized because of its sensitivity to sample size. As a reaction, an abundance of fit indexes have been developed. The result of these developments is that structural equation modeling packages are now producing a large list of fit measures. One would think that this progression has led to a clear understanding of evaluating models with respect to model misspecifications. In this article we question the validity of approaches for model evaluation based on overall goodness-of-fit indexes. The argument against such usage is that they do not provide an adequate indication of the “size” of the model's misspecification. That is, they vary dramatically with the values of incidental parameters that are unrelated with the misspecification in the model. This is illustrated using simple but fundamental models. As an alternative method of model evaluation, we suggest using the expected parameter change in combination with the modification index (MI) and the power of the MI test.     

A Bayesian Approach to Multilevel Structural Equation Modeling With Continuous and Dichotomous Outcomes

Multilevel Structural equation models are most often estimated from a frequentist framework via maximum likelihood. However, as shown in this article, frequentist results are not always accurate. Alternatively, one can apply a Bayesian approach using Markov chain Monte Carlo estimation methods. This simulation study compared estimation quality using Bayesian and frequentist approaches in the context of a multilevel latent covariate model. Continuous and dichotomous variables were examined because it is not yet known how different types of outcomes—most notably categorical—affect parameter recovery in this modeling context. Within the Bayesian estimation framework, the impact of diffuse, weakly informative, and informative prior distributions were compared. Findings indicated that Bayesian estimation may be used to overcome convergence problems and improve parameter estimate bias. Results highlight the differences in estimation quality between dichotomous and continuous variable models and the importance of prior distribution choice for cluster-level random effects.     

A Bayesian Approach for Estimating Multilevel Latent Contextual Models

In many applications of multilevel modeling, group-level (L2) variables for assessing group-level effects are generated by aggregating variables from a lower level (L1). However, the observed group mean might not be a reliable measure of the unobserved true group mean. In this article, we propose a Bayesian approach for estimating a multilevel latent contextual model that corrects for measurement error and sampling error (i.e., sampling only a small number of L1 units from a L2 unit) when estimating group-level effects of aggregated L1 variables. Two simulation studies were conducted to compare the Bayesian approach with the maximum likelihood approach implemented in Mplus. The Bayesian approach showed fewer estimation problems (e.g., inadmissible solutions) and more accurate estimates of the group-level effect than the maximum likelihood approach under problematic conditions (i.e., small number of groups, predictor variable with a small intraclass correlation). An application from educational psychology is used to illustrate the different estimation approaches.     

Measurement Invariance Testing with Many Groups: A Comparison of Five Approaches

With the increasing use of international survey data especially in cross-cultural and multinational studies, establishing measurement invariance (MI) across a large number of groups in a study is essential. Testing MI over many groups is methodologically challenging, however. We identified 5 methods for MI testing across many groups (multiple group confirmatory factor analysis, multilevel confirmatory factor analysis, multilevel factor mixture modeling, Bayesian approximate MI testing, and alignment optimization) and explicated the similarities and differences of these approaches in terms of their conceptual models and statistical procedures. A Monte Carlo study was conducted to investigate the efficacy of the 5 methods in detecting measurement noninvariance across many groups using various fit criteria. Generally, the 5 methods showed reasonable performance in identifying the level of invariance if an appropriate fit criterion was used (e.g., Bayesian information criteron with multilevel factor mixture modeling). Finally, general guidelines in selecting an appropriate method are provided.     

Cross-Level Invariance in Multilevel Factor Models

When modeling latent variables at multiple levels, it is important to consider the meaning of the latent variables at the different levels. If a higher-level common factor represents the aggregated version of a lower-level factor, the associated factor loadings will be equal across levels. However, many researchers do not consider cross-level invariance constraints in their research. Not applying these constraints when in fact they are appropriate leads to overparameterized models, and associated convergence and estimation problems. This simulation study used a two-level mediation model on common factors to show that when factor loadings are equal in the population, not applying cross-level invariance constraints leads to more estimation problems and smaller true positive rates. Some directions for future research on cross-level invariance in MLSEM are discussed.     

Analyzing Faculty Workload Data Using Multilevel Modeling

Research on faculty productivity fails to account for the hierarchical nature of the data. Faculty members within an academic discipline more closely resemble one another than faculty in other disciplines, resulting in dependent observations and thus inaccurate statistical results. Unlike ordinary least squares, multilevel modeling takes into account this grouping effect. This article analyzes the research productivity of 1,104 tenured/tenure-track faculty from the 1993 NSOPF survey to compare traditional regression models with a random coefficients model. The results indicate a large grouping effect on research productivity, and the statistical as well as the substantive results of the random coefficients model differ significantly from the regression approach.     

Comparing Multilevel and Classical Confirmatory Factor Analysis Parameterizations of Multirater Data: A Monte Carlo Simulation Study

This simulation study assesses the statistical performance of two mathematically equivalent parameterizations for multitrait–multimethod data with interchangeable raters—a multilevel confirmatory factor analysis (CFA) and a classical CFA parameterization. The sample sizes of targets and raters, the factorial structure of the trait factors, and rater missingness are varied. The classical CFA approach yields a high proportion of improper solutions under conditions with small sample sizes and indicator-specific trait factors. In general, trait factor related parameters are more sensitive to bias than other types of parameters. For multilevel CFAs, there is a drastic bias in fit statistics under conditions with unidimensional trait factors on the between level, where root mean square error of approximation (RMSEA) and χ2 distributions reveal a downward bias, whereas the between standardized root mean square residual is biased upwards. In contrast, RMSEA and χ2 for classical CFA models are severely upwardly biased in conditions with a high number of raters and a small number of targets.     

Evaluation of Structural Relationships in Autoregressive Cross-Lagged Models Under Longitudinal Approximate Invariance:A Bayesian Analysis

To infer longitudinal relationships among latent factors, traditional analyses assume that the measurement model is invariant across measurement occasions. Alternative to placing cross-occasion equality constraints on parameters, approximate measurement invariance (MI) can be analyzed by specifying informative priors on parameter differences between occasions. This study evaluated the estimation of structural coefficients in multiple-indicator autoregressive cross-lagged models under various conditions of approximate MI using Bayesian structural equation modeling. Design factors included factor structures, conditions of non-invariance, sizes of structural coefficients, and sample sizes. Models were analyzed using two sets of small-variance priors on select model parameters. Results showed that autoregressive coefficient estimates were more accurate for the mixed pattern than the decreasing pattern of non-invariance. When a model included cross-loadings, an interaction was found between the cross-lagged estimates and the non-invariance conditions. Implications of findings and future research directions are discussed.     

Using students’ feedback for teacher education: measurement invariance across pre-service teacher-rated and student-rated aspects of quality of teaching

Comparing self-perceived quality of teaching to students’ perception can be used in higher education to improve the quality of teaching of pre-service teachers in teacher education. However, comparing these measurements from different perspectives is only meaningful if the same constructs are being measured. To shed light on this comparison’s meaningfulness, we scrutinised whether aspects of quality of teaching are measured in the same way across pre-service teachers and their students by means of measurement invariance analyses. To do so, 272 pre-service teachers in teacher education rated aspects of their quality of teaching, and were rated by their 4851 students. Measurement invariance across these perspectives was tested in multilevel structural equation models. Strong measurement invariance held for two aspects of quality of teaching; for the third, one item lacked weak measurement invariance. Pre-service teachers perceived their quality of teaching lower than their students. In conclusion, aspects of quality of teaching can be compared across perspectives, and teacher education should encourage pre-service teachers to use students’ feedback as a valuable resource for improving their quality of teaching.     

Growth Curve Modeling to Studying Change: A Comparison of Approaches Using Longitudinal Dyadic Data With Distinguishable Dyads

Although methodology articles have increasingly emphasized the need to analyze data from two members of a dyad simultaneously, the most popular method in substantive applications is to examine dyad members separately. This might be due to the underappreciation of the extra information simultaneous modeling strategies can provide. Therefore, the goal of this study was to compare multiple growth curve modeling approaches for longitudinal dyadic data (LDD) in both structural equation modeling and multilevel modeling frameworks. Models separately assessing change over time for distinguishable dyad members are compared to simultaneous models fitted to LDD from both dyad members. Furthermore, we compared the simultaneous default versus dependent approaches (whether dyad pairs’ Level 1 [or unique] residuals are allowed to covary and differ in variance). Results indicated that estimates of variance and covariance components led to conflicting results. We recommend the simultaneous dependent approach for inferring differences in change over time within a dyad.     

The Actor–Partner Interdependence Model for Longitudinal Dyadic Data: An Implementation in the SEM Framework

In dyadic research, the actor–partner interdependence model (APIM) is widely used to model the effect of a predictor measured across dyad members on one’s own and one’s partner outcome. When such dyadic data are measured repeatedly over time, both the non-independence within couples and the non-independence over time need to be accounted for. In this paper, we present a longitudinal extension of the APIM, the L-APIM, that allows for both stable and time-varying sources of non-independence. Its implementation is readily available in multilevel software, such as proc mixed in SAS, but is lacking in the structural equation modeling (SEM) framework. We tackle the computational challenges associated with its SEM-implementation and propose a user-friendly free application for the L-APIM, which can be found at http://fgisteli.shinyapps.io/Shiny_LDD. As an illustration, we explore the actor and partner effects of positive relationship feelings on next day’s intimacy using 3-week diary data of 66 heterosexual couples.     

Assessing statistical aspects of test fairness with structural equation modelling

Test fairness and test bias are not synonymous concepts. Test bias refers to statistical evidence that the psychometrics or interpretation of test scores depend on group membership, such as gender or race, when such differences are not expected. A test that is grossly biased may be judged to be unfair, but test fairness concerns the broader, more subjective evaluation of assessment outcomes from perspectives of social justice. Thus, the determination of test fairness is not solely a matter of statistics, but statistical evidence is important when evaluating test fairness. This work introduces the use of the structural equation modelling technique of multiple-group confirmatory factor analysis (MGCFA) to evaluate hypotheses of measurement invariance, or whether a set of observed variables measures the same factors with the same precision over different populations. An example of testing for measurement invariance with MGCFA in an actual, downloadable data set is also demonstrated.     

Alternatives to Multilevel Modeling for the Analysis of Clustered Data

Multilevel modeling has grown in use over the years as a way to deal with the nonindependent nature of observations found in clustered data. However, other alternatives to multilevel modeling are available that can account for observations nested within clusters, including the use of Taylor series linearization for variance estimation, the design effect adjusted standard errors approach, and fixed effects modeling. Using 1,000 replications of 12 conditions with varied Level 1 and Level 2 sample sizes, the author compared parameter estimates, standard errors, and statistical significance using various alternative procedures. Results indicate that several acceptable procedures can be used in lieu of or together with multilevel modeling, depending on the type of research question asked and the number of clusters under investigation. Guidelines for applied researchers are discussed.     

Implementing Restricted Maximum Likelihood Estimation in Structural Equation Models

Structural equation modeling (SEM) is now a generic modeling framework for many multivariate techniques applied in the social and behavioral sciences. Many statistical models can be considered either as special cases of SEM or as part of the latent variable modeling framework. One popular extension is the use of SEM to conduct linear mixed-effects modeling (LMM) such as cross-sectional multilevel modeling and latent growth modeling. It is well known that LMM can be formulated as structural equation models. However, one main difference between the implementations in SEM and LMM is that maximum likelihood (ML) estimation is usually used in SEM, whereas restricted (or residual) maximum likelihood (REML) estimation is the default method in most LMM packages. This article shows how REML estimation can be implemented in SEM. Two empirical examples on latent growth model and meta-analysis are used to illustrate the procedures implemented in OpenMx. Issues related to implementing REML in SEM are discussed.