Multilevel Mediation With Small Samples: A Cautionary Note on the Multilevel Structural Equation Modeling Framework

Multilevel structural equation modeling (ML-SEM) for multilevel mediation is noted for its flexibility over a system of multilevel models (MLMs). Sample size requirements are an overlooked limitation of ML-SEM (100 clusters is recommended). We find that 89% of ML-SEM studies have fewer than 100 clusters and the median number is 44. Furthermore, 75% of ML-SEM studies implement 2–1–1 or 1–1–1 models, which can be equivalently fit with MLMs. MLMs theoretically have lower sample size requirements, although studies have yet to assess small sample performance for multilevel mediation. We conduct a simulation to address this pervasive problem. We find that MLMs have more desirable small sample performance and can be trustworthy with 10 clusters. Importantly, many studies lack the sample size and model complexity to necessitate ML-SEM. Although ML-SEM is undeniably more flexible and uniquely positioned for difficult problems, small samples often can be more effectively and simply addressed with MLMs.     

Mediation Effects In 2-1-1 Multilevel Model: Evaluation Of Alternative Estimation Methods

We compared six common methods in estimating the 2-1-1 (level-2 independent, level-1 mediator, level-1 dependent) multilevel mediation model with a random slope. They were the Bayesian with informative priors, the Bayesian with non-informative priors, the Monte-Carlo, the distribution of the product, the bias-corrected, and the bias-uncorrected parametric percentile residual bootstrap. The Bayesian method with informative priors was superior in relative mean square error (RMSE), power, interval width, and interval imbalance. The prior variance and prior mean were also varied and examined. Decreasing the prior variance increased the power, reduced RMSE and interval width when the prior mean was the true value, but decreasing the prior variance reduced the power when the prior mean was set incorrectly. The influence of misspecification of prior information of the b coefficient on multilevel mediation analysis was greater than that on coefficient a. An illustrate example with the Bayesian multilevel mediation was provided.     

Measurement Bias in Multilevel Data

Measurement bias can be detected using structural equation modeling (SEM), by testing measurement invariance with multigroup factor analysis (Jöreskog, 1971;Meredith, 1993;Sörbom, 1974) MIMIC modeling (Muthén, 1989) or restricted factor analysis (Oort, 1992,1998). In educational research, data often have a nested, multilevel structure, for example when data are collected from children in classrooms. Multilevel structures might complicate measurement bias research. In 2-level data, the potentially “biasing trait” or “violator” can be a Level 1 variable (e.g., pupil sex), or a Level 2 variable (e.g., teacher sex). One can also test measurement invariance with respect to the clustering variable (e.g., classroom). This article provides a stepwise approach for the detection of measurement bias with respect to these 3 types of violators. This approach works from Level 1 upward, so the final model accounts for all bias and substantive findings at both levels. The 5 proposed steps are illustrated with data of teacher–child relationships.     

A Test for Cluster Bias: Detecting Violations of Measurement Invariance Across Clusters in Multilevel Data

We present a test for cluster bias, which can be used to detect violations of measurement invariance across clusters in 2-level data. We show how measurement invariance assumptions across clusters imply measurement invariance across levels in a 2-level factor model. Cluster bias is investigated by testing whether the within-level factor loadings are equal to the between-level factor loadings, and whether the between-level residual variances are zero. The test is illustrated with an example from school research. In a simulation study, we show that the cluster bias test has sufficient power, and the proportions of false positives are close to the chosen levels of significance.     

Bayesian Meta-Analytic SEM: A One-Stage Approach to Modeling Between-Studies Heterogeneity in Structural Parameters

Meta-analytic structural equation modeling (MASEM) refers to a set of meta-analysis techniques for combining and comparing structural equation modeling (SEM) results from multiple studies. Existing approaches to MASEM cannot appropriately model between-studies heterogeneity in structural parameters because of missing correlations, lack model fit assessment, and suffer from several theoretical limitations. In this study, we address the major shortcomings of existing approaches by proposing a novel Bayesian multilevel SEM approach. Simulation results showed that the proposed approach performed satisfactorily in terms of parameter estimation and model fit evaluation when the number of studies and the within-study sample size were sufficiently large and when correlations were missing completely at random. An empirical example about the structure of personality based on a subset of data was provided. Results favored the third factor structure over the hierarchical structure. We end the article with discussions and future directions.     

The Actor–Partner Interdependence Model for Longitudinal Dyadic Data: An Implementation in the SEM Framework

In dyadic research, the actor–partner interdependence model (APIM) is widely used to model the effect of a predictor measured across dyad members on one’s own and one’s partner outcome. When such dyadic data are measured repeatedly over time, both the non-independence within couples and the non-independence over time need to be accounted for. In this paper, we present a longitudinal extension of the APIM, the L-APIM, that allows for both stable and time-varying sources of non-independence. Its implementation is readily available in multilevel software, such as proc mixed in SAS, but is lacking in the structural equation modeling (SEM) framework. We tackle the computational challenges associated with its SEM-implementation and propose a user-friendly free application for the L-APIM, which can be found at http://fgisteli.shinyapps.io/Shiny_LDD. As an illustration, we explore the actor and partner effects of positive relationship feelings on next day’s intimacy using 3-week diary data of 66 heterosexual couples.     

Effects of Missing Data Methods in SEM Under Conditions of Incomplete and Nonnormal Data

Using Monte Carlo simulations, this research examined the performance of four missing data methods in SEM under different multivariate distributional conditions. The effects of four independent variables (sample size, missing proportion, distribution shape, and factor loading magnitude) were investigated on six outcome variables: convergence rate, parameter estimate bias, MSE of parameter estimates, standard error coverage, model rejection rate, and model goodness of fit—RMSEA. A three-factor CFA model was used. Findings indicated that FIML outperformed the other methods in MCAR, and MI should be used to increase the plausibility of MAR. SRPI was not comparable to the other three methods in either MCAR or MAR.     

Multilevel Factorial Invariance in n-Level Structural Equation Modeling (nSEM)

We present a multigroup multilevel confirmatory factor analysis (CFA) model and a procedure for testing multilevel factorial invariance in n-level structural equation modeling (nSEM). Multigroup multilevel CFA introduces a complexity when the group membership at the lower level intersects the clustered structure, because the observations in different groups but in the same cluster are not independent of one another. nSEM provides a framework in which the multigroup multilevel data structure is represented with the dependency between groups at the lower level properly taken into account. The procedure for testing multilevel factorial invariance is illustrated with an empirical example using an R package xxm2.     

A Bayesian Approach for Estimating Multilevel Latent Contextual Models

In many applications of multilevel modeling, group-level (L2) variables for assessing group-level effects are generated by aggregating variables from a lower level (L1). However, the observed group mean might not be a reliable measure of the unobserved true group mean. In this article, we propose a Bayesian approach for estimating a multilevel latent contextual model that corrects for measurement error and sampling error (i.e., sampling only a small number of L1 units from a L2 unit) when estimating group-level effects of aggregated L1 variables. Two simulation studies were conducted to compare the Bayesian approach with the maximum likelihood approach implemented in Mplus. The Bayesian approach showed fewer estimation problems (e.g., inadmissible solutions) and more accurate estimates of the group-level effect than the maximum likelihood approach under problematic conditions (i.e., small number of groups, predictor variable with a small intraclass correlation). An application from educational psychology is used to illustrate the different estimation approaches.     

A Bayesian Approach to Multilevel Structural Equation Modeling With Continuous and Dichotomous Outcomes

Multilevel Structural equation models are most often estimated from a frequentist framework via maximum likelihood. However, as shown in this article, frequentist results are not always accurate. Alternatively, one can apply a Bayesian approach using Markov chain Monte Carlo estimation methods. This simulation study compared estimation quality using Bayesian and frequentist approaches in the context of a multilevel latent covariate model. Continuous and dichotomous variables were examined because it is not yet known how different types of outcomes—most notably categorical—affect parameter recovery in this modeling context. Within the Bayesian estimation framework, the impact of diffuse, weakly informative, and informative prior distributions were compared. Findings indicated that Bayesian estimation may be used to overcome convergence problems and improve parameter estimate bias. Results highlight the differences in estimation quality between dichotomous and continuous variable models and the importance of prior distribution choice for cluster-level random effects.     

Evaluating Model Fit and Structural Coefficient Bias: A Bayesian Approach to Multilevel Bifactor Model Misspecification

Appropriate model specification is fundamental to unbiased parameter estimates and accurate model interpretations in structural equation modeling. Thus detecting potential model misspecification has drawn the attention of many researchers. This simulation study evaluates the efficacy of the Bayesian approach (the posterior predictive checking, or PPC procedure) under multilevel bifactor model misspecification (i.e., ignoring a specific factor at the within level). The impact of model misspecification on structural coefficients was also examined in terms of bias and power. Results showed that the PPC procedure performed better in detecting multilevel bifactor model misspecification, when the misspecification became more severe and sample size was larger. Structural coefficients were increasingly negatively biased at the within level, as model misspecification became more severe. Model misspecification at the within level affected the between-level structural coefficient estimates more when data dependency was lower and the number of clusters was smaller. Implications for researchers are discussed.     

An Expansion of the Trait-State-Occasion Model: Accounting for Shared Method Variance

This article examines 4 approaches for explaining shared method variance, each applied to a longitudinal trait–state–occasion (TSO) model. Many approaches have been developed to account for shared method variance in multitrait-multimethod (MTMM) data. Some of these MTMM approaches (correlated method, orthogonal method, correlated method minus one, correlated uniqueness) were therefore borrowed in these analyses such that their effectiveness could be evaluated in conjunction with a TSO model. To this end, datasets were generated according to 4 different covariance matrices (each created according to specifications of a model built with 1 of the 4 approaches) and each model was crossed with each type of data. Whereas the correlated method and correlated method minus one approaches encountered many difficulties in convergence, fit, or parameter estimates, the correlated uniqueness and orthogonal method approaches proved to be quite versatile.     

Using SEM to Analyze Complex Survey Data: A Comparison between Design-Based Single-Level and Model-Based Multilevel Approaches

Both ad-hoc robust sandwich standard error estimators (design-based approach) and multilevel analysis (model-based approach) are commonly used for analyzing complex survey data with nonindependent observations. Although these 2 approaches perform equally well on analyzing complex survey data with equal between- and within-level model structures (B. O. Muthén & Satorra, 1995), the performances of these 2 approaches for analyzing multilevel data with unequal between- and within-level structures have not yet been systematically examined. In this study, we extended B. O. Muthén and Satorra's (1995) Muthén, B. O. and Satorra, A. 1995. Complex sample data in structural equation modeling. Sociological Methodology, 25: 267–316. [Crossref], [Web of Science ®] , [Google Scholar] study by comparing these 2 approaches and an additional model-based maximum model for analyzing multilevel data considering number of clusters, cluster size, intraclass correlation, and the equality of different level structures. The simulation results showed the model-based maximum model generally performed well across conditions. This model is also recommended as an alternative for analyzing nonindependent survey data, especially when the information of the higher level model structure is not known.     

Comparing Multilevel and Classical Confirmatory Factor Analysis Parameterizations of Multirater Data: A Monte Carlo Simulation Study

This simulation study assesses the statistical performance of two mathematically equivalent parameterizations for multitrait–multimethod data with interchangeable raters—a multilevel confirmatory factor analysis (CFA) and a classical CFA parameterization. The sample sizes of targets and raters, the factorial structure of the trait factors, and rater missingness are varied. The classical CFA approach yields a high proportion of improper solutions under conditions with small sample sizes and indicator-specific trait factors. In general, trait factor related parameters are more sensitive to bias than other types of parameters. For multilevel CFAs, there is a drastic bias in fit statistics under conditions with unidimensional trait factors on the between level, where root mean square error of approximation (RMSEA) and χ2 distributions reveal a downward bias, whereas the between standardized root mean square residual is biased upwards. In contrast, RMSEA and χ2 for classical CFA models are severely upwardly biased in conditions with a high number of raters and a small number of targets.     

Assessing Mediation in Dyadic Data Using the Actor-Partner Interdependence Model

The assessment of mediation in dyadic data is an important issue if researchers are to test process models. Using an extended version of the actor–partner interdependence model the estimation and testing of mediation is complex, especially when dyad members are distinguishable (e.g., heterosexual couples). We show how the complexity of the model can be reduced by assuming specific dyadic patterns. Using structural equation modeling, we demonstrate how specific mediating effects and contrasts among effects can be tested by phantom models that permit point and bootstrap interval estimates. We illustrate the assessment of mediation and the strategies to simplify the model using data from heterosexual couples.     

Inference and Interval Estimation Methods for Indirect Effects With Latent Variable Models

Although much is known about the performance of recent methods for inference and interval estimation for indirect or mediated effects with observed variables, little is known about their performance in latent variable models. This article presents an extensive Monte Carlo study of 11 different leading or popular methods adapted to structural equation models with latent variables. Manipulated variables included sample size, number of indicators per latent variable, internal consistency per set of indicators, and 16 different path combinations between latent variables. Results indicate that some popular or previously recommended methods, such as the bias-corrected bootstrap and asymptotic standard errors had poorly calibrated Type I error and coverage rates in some conditions. Likelihood-based confidence intervals, the distribution of the product method, and the percentile bootstrap emerged as leading methods for both interval estimation and inference, whereas joint significance tests and the partial posterior method performed well for inference.     

Growth Curve Modeling to Studying Change: A Comparison of Approaches Using Longitudinal Dyadic Data With Distinguishable Dyads

Although methodology articles have increasingly emphasized the need to analyze data from two members of a dyad simultaneously, the most popular method in substantive applications is to examine dyad members separately. This might be due to the underappreciation of the extra information simultaneous modeling strategies can provide. Therefore, the goal of this study was to compare multiple growth curve modeling approaches for longitudinal dyadic data (LDD) in both structural equation modeling and multilevel modeling frameworks. Models separately assessing change over time for distinguishable dyad members are compared to simultaneous models fitted to LDD from both dyad members. Furthermore, we compared the simultaneous default versus dependent approaches (whether dyad pairs’ Level 1 [or unique] residuals are allowed to covary and differ in variance). Results indicated that estimates of variance and covariance components led to conflicting results. We recommend the simultaneous dependent approach for inferring differences in change over time within a dyad.     

Causal Effects in Mediation Modeling: An Introduction With Applications to Latent Variables

Causal inference in mediation analysis offers counterfactually based causal definitions of direct and indirect effects, drawing on research by Robins, Greenland, Pearl, VanderWeele, Vansteelandt, Imai, and others. This type of mediation effect estimation is little known and seldom used among analysts using structural equation modeling (SEM). The aim of this article is to describe the new analysis opportunities in a way that is accessible to SEM analysts and show examples of how to perform the analyses. An application is presented with an extension to a latent mediator measured with multiple indicators.     

The Role of Centering for Interaction of Level 1 Variables in Multilevel Structural Equation Models

This article examined the role of centering in estimating interaction effects in multilevel structural equation models. Interactions are typically represented by product term of 2 variables that are hypothesized to interact. In multilevel structural equation modeling (MSEM), the product term involving Level 1 variables is decomposed into within-cluster and between-cluster random components. The choice of centering affects the decomposition of the product term, and therefore affects the sample variance and covariance associated with the product term used in the maximum likelihood fitting function. The simulation study showed that for an interaction between a Level 1 variable and a Level 2 variable, the product term of uncentered variables or the product term of grand mean centered variables produced unbiased estimates in both Level 1 and Level 2 models. The product term of cluster mean centered variables produced biased estimates in the Level 1 model. For an interaction between 2 Level 1 variables, the product term of cluster mean centered variables produced unbiased estimates in the Level 1 model, whereas the product term of grand mean centered variables produced unbiased estimates for the Level 1 model. Recommendations for researchers who wish to estimate interactions in MSEM are provided.     

The Misspecification of the Covariance Structures in Multilevel Models for Single-Case Data: A Monte Carlo Simulation Study

The impact of misspecifying covariance matrices at the second and third levels of the three-level model is evaluated. Results indicate that ignoring existing covariance has no effect on the treatment effect estimate. In addition, the between-case variance estimates are unbiased when covariance is either modeled or ignored. If the research interest lies in the between-study variance estimate, including at least 30 studies is warranted. Modeling covariance does not result in less biased between-study variance estimates as the between-study covariance estimate is biased. When the research interest lies in the between-case covariance, the model including covariance results in unbiased between-case variance estimates. The three-level model appears to be less appropriate for estimating between-study variance if fewer than 30 studies are included.