Multilevel Mediation With Small Samples: A Cautionary Note on the Multilevel Structural Equation Modeling Framework

Multilevel structural equation modeling (ML-SEM) for multilevel mediation is noted for its flexibility over a system of multilevel models (MLMs). Sample size requirements are an overlooked limitation of ML-SEM (100 clusters is recommended). We find that 89% of ML-SEM studies have fewer than 100 clusters and the median number is 44. Furthermore, 75% of ML-SEM studies implement 2–1–1 or 1–1–1 models, which can be equivalently fit with MLMs. MLMs theoretically have lower sample size requirements, although studies have yet to assess small sample performance for multilevel mediation. We conduct a simulation to address this pervasive problem. We find that MLMs have more desirable small sample performance and can be trustworthy with 10 clusters. Importantly, many studies lack the sample size and model complexity to necessitate ML-SEM. Although ML-SEM is undeniably more flexible and uniquely positioned for difficult problems, small samples often can be more effectively and simply addressed with MLMs.     

Multilevel Factorial Invariance in n-Level Structural Equation Modeling (nSEM)

We present a multigroup multilevel confirmatory factor analysis (CFA) model and a procedure for testing multilevel factorial invariance in n-level structural equation modeling (nSEM). Multigroup multilevel CFA introduces a complexity when the group membership at the lower level intersects the clustered structure, because the observations in different groups but in the same cluster are not independent of one another. nSEM provides a framework in which the multigroup multilevel data structure is represented with the dependency between groups at the lower level properly taken into account. The procedure for testing multilevel factorial invariance is illustrated with an empirical example using an R package xxm2.     

A Bayesian Approach to Multilevel Structural Equation Modeling With Continuous and Dichotomous Outcomes

Multilevel Structural equation models are most often estimated from a frequentist framework via maximum likelihood. However, as shown in this article, frequentist results are not always accurate. Alternatively, one can apply a Bayesian approach using Markov chain Monte Carlo estimation methods. This simulation study compared estimation quality using Bayesian and frequentist approaches in the context of a multilevel latent covariate model. Continuous and dichotomous variables were examined because it is not yet known how different types of outcomes—most notably categorical—affect parameter recovery in this modeling context. Within the Bayesian estimation framework, the impact of diffuse, weakly informative, and informative prior distributions were compared. Findings indicated that Bayesian estimation may be used to overcome convergence problems and improve parameter estimate bias. Results highlight the differences in estimation quality between dichotomous and continuous variable models and the importance of prior distribution choice for cluster-level random effects.     

The Role of Centering for Interaction of Level 1 Variables in Multilevel Structural Equation Models

This article examined the role of centering in estimating interaction effects in multilevel structural equation models. Interactions are typically represented by product term of 2 variables that are hypothesized to interact. In multilevel structural equation modeling (MSEM), the product term involving Level 1 variables is decomposed into within-cluster and between-cluster random components. The choice of centering affects the decomposition of the product term, and therefore affects the sample variance and covariance associated with the product term used in the maximum likelihood fitting function. The simulation study showed that for an interaction between a Level 1 variable and a Level 2 variable, the product term of uncentered variables or the product term of grand mean centered variables produced unbiased estimates in both Level 1 and Level 2 models. The product term of cluster mean centered variables produced biased estimates in the Level 1 model. For an interaction between 2 Level 1 variables, the product term of cluster mean centered variables produced unbiased estimates in the Level 1 model, whereas the product term of grand mean centered variables produced unbiased estimates for the Level 1 model. Recommendations for researchers who wish to estimate interactions in MSEM are provided.     

Estimating First-Year Student Attrition Rates: An Application of Multilevel Modeling Using Categorical Variables

There have recently been significant theoretical developments in multilevel statistical modeling, and improved software is readily available. This study demonstrates the application of multilevel modeling to one of the most common issues that confront institutional researchers: that of student attrition, where the response variable is typically binary rather than continuous. Comparisons are made with a traditional logistic regression approach. The data pertain to one large university. The techniques illustrated may be extended to the analysis of data sets encompassing many institutions, making meaningful interinstitutional comparisons of performance feasible even when there is hierarchical clustering present in the data.     

Evaluation of Test Statistics for Robust Structural Equation Modeling With Nonnormal Missing Data

A 2-stage robust procedure as well as an R package, rsem, were recently developed for structural equation modeling with nonnormal missing data by Yuan and Zhang (2012). Several test statistics that have been used for complete data analysis are employed to evaluate model fit in the 2-stage robust method. However, properties of these statistics under robust procedures for incomplete nonnormal data analysis have never been studied. This study aims to systematically evaluate and compare 5 test statistics, including a test statistic derived from normal-distribution-based maximum likelihood, a rescaled chi-square statistic, an adjusted chi-square statistic, a corrected residual-based asymptotical distribution-free chi-square statistic, and a residual-based F statistic. These statistics are evaluated under a linear growth curve model by varying 8 factors: population distribution, missing data mechanism, missing data rate, sample size, number of measurement occasions, covariance between the latent intercept and slope, variance of measurement errors, and downweighting rate of the 2-stage robust method. The performance of the test statistics varies and the one derived from the 2-stage normal-distribution-based maximum likelihood performs much worse than the other four. Application of the 2-stage robust method and of the test statistics is illustrated through growth curve analysis of mathematical ability development, using data on the Peabody Individual Achievement Test mathematics assessment from the National Longitudinal Survey of Youth 1997 Cohort.     

Fit for a Bayesian: An Evaluation of PPP and DIC for Structural Equation Modeling

Despite its importance to structural equation modeling, model evaluation remains underdeveloped in the Bayesian SEM framework. Posterior predictive p-values (PPP) and deviance information criteria (DIC) are now available in popular software for Bayesian model evaluation, but they remain underutilized. This is largely due to the lack of recommendations for their use. To address this problem, PPP and DIC were evaluated in a series of Monte Carlo simulation studies. The results show that both PPP and DIC are influenced by severity of model misspecification, sample size, model size, and choice of prior. The cutoffs PPP < 0.10 and ?DIC > 7 work best in the conditions and models tested here to maintain low false detection rates and misspecified model selection rates, respectively. The recommendations provided in this study will help researchers evaluate their models in a Bayesian SEM analysis and set the stage for future development and evaluation of Bayesian SEM fit indices.     

SEMDS: An R Package for Structural Equation Multidimensional Scaling

The aim of this article is to introduce the R package semds for structural equation multidimensional scaling. This methodology combines multidimensional scaling with latent variable features from structural equation modeling and is applicable to asymmetric and three-way input dissimilarity data. This key idea of this approach is that the input data are assumed to be imperfect measurements of a latent symmetric dissimilarity matrix. The parameter estimation is performed via an alternating least squares multidimensional scaling procedure that minimizes the stress. The latent dissimilarities are estimated as factor scores within a structural equation modeling framework. Applications shown in the article involve data associated with the banking crisis and data from avalanche research. The models fitted with the semds package are compared to related methods from multidimensional scaling. The R code to reproduce all the computations is provided in the supplementary materials.     

Standards-Based Evaluation and Teacher Career Satisfaction: A Structural Equation Modeling Analysis

Structural equation modeling was used to assess the plausibility of a conceptual model specifying hypothesized linkages among perceptions of characteristics of standards-based evaluation, work environment mediators, and career satisfaction and other outcomes. Four comprehensive high schools located in two neighboring counties in southern California provided the data for this study. The schools’ districts had recently developed and implemented evaluation systems based on six California Standards for the Teaching Profession generated in 1997. One hundred and seventy-eight teachers responded to survey questions designed to capture the following constructs: understandable/relevant standards, satisfactory/helpful evaluation, role ambiguity, effort performance-rating linkage, work criteria autonomy, career satisfaction, organizational commitment, and perceptions of the effectiveness of the evaluation system. Confirmatory factor analysis was used to assess whether the items measuring evaluation fit two hypothesized constructs. Structural equation modeling results indicated that there are two mediators in the evaluation-career satisfaction relationship: role ambiguity and work criteria autonomy.

A Second-Order Longitudinal Model for Binary Outcomes: Item Response Theory Versus Structural Equation Modeling

Measuring academic growth, or change in aptitude, relies on longitudinal data collected across multiple measurements. The National Educational Longitudinal Study (NELS:88) is among the earliest, large-scale, educational surveys tracking students’ performance on cognitive batteries over 3 years. Notable features of the NELS:88 data set, and of almost all repeated measures educational assessments, are (a) the outcome variables are binary or at least categorical in nature; and (b) a set of different items is given at each measurement occasion with a few anchor items to fix the measurement scale. This study focuses on the challenges related to specifying and fitting a second-order longitudinal model for binary outcomes, within both the item response theory and structural equation modeling frameworks. The distinctions between and commonalities shared between these two frameworks are discussed. A real data analysis using the NELS:88 data set is presented for illustration purposes.     

Number of Subjects and Time Points Needed for Multilevel Time-Series Analysis: A Simulation Study of Dynamic Structural Equation Modeling

Dynamic structural equation modeling (DSEM) is a novel, intensive longitudinal data (ILD) analysis framework. DSEM models intraindividual changes over time on Level 1 and allows the parameters of these processes to vary across individuals on Level 2 using random effects. DSEM merges time series, structural equation, multilevel, and time-varying effects models. Despite the well-known properties of these analysis areas by themselves, it is unclear how their sample size requirements and recommendations transfer to the DSEM framework. This article presents the results of a simulation study that examines the estimation quality of univariate 2-level autoregressive models of order 1, AR(1), using Bayesian analysis in Mplus Version 8. Three features are varied in the simulations: complexity of the model, number of subjects, and number of time points per subject. Samples with many subjects and few time points are shown to perform substantially better than samples with few subjects and many time points.     

The Theory of Planned Behavior Within the Stages of the Transtheoretical Model: Latent Structural Modeling of Stage-Specific Prediction Patterns in Physical Activity

This is the first study to test whether the stages of change of the transtheoretical model are qualitatively different through exploring discontinuity patterns in theory of planned behavior (TPB) variables using latent multigroup structural equation modeling (MSEM) with AMOS. Discontinuity patterns in terms of latent means and prediction patterns for the different stage groups were examined. Adults (n = 3,462) were assessed on their physical activity stages of change and TPB variables. The TPB was separately examined within the five stage groups. The TPB measurement model fit was acceptable. Latent mean analyses with post-hoc contrast and MSEM indicated discontinuity patterns. Results underscore the qualitative differences between the stages that may guide further research and the design of interventions integrating the approaches.     

Digital Module 16: Longitudinal Data Analysis

In this digital ITEMS module, Dr. Jeffrey Harring and Ms. Tessa Johnson introduce the linear mixed effects (LME) model as a flexible general framework for simultaneously modeling continuous repeated measures data with a scientifically defensible function that adequately summarizes both individual change as well as the average response. The module begins with a nontechnical overview of longitudinal data analyses drawing distinctions with cross-sectional analyses in terms of research questions to be addressed. Nuances of longitudinal designs, timing of measurements, and the real possibility of missing data are then discussed. The three interconnected components of the LME model—(1) a model for individual and mean response profiles, (2) a model to characterize the covariation among the time-specific residuals, and (3) a set of models that summarize the extent that individual coefficients vary—are discussed in the context of the set of activities comprising an analysis. Finally, they demonstrate how to estimate the linear mixed effects model within an open-source environment (R). The digital module contains sample R code, diagnostic quiz questions, hands-on activities in R, curated resources, and a glossary.     

高校双语教学满意度与忠诚度影响因素研究--基于结构方程模型的实证分析

高校双语教学满意度与忠诚度是衡量双语教学质量的重要指标,对于培养国际化复合型人才有着积极意义。本文构建了高校双语教学满意度与忠诚度模型,分析了教学水平、教师素养、学习潜能、学习自主性、外语氛围和激励保障六个前因变量,以及学生抱怨和双语教学忠诚度两个结果变量对双语教学满意度的影响。从华南农业大学采集了540个有效样本,采用结构方程技术对模型进行了实证分析。研究结果表明,教学水平、教师素养、学习自主性和激励保障在不同程度上对双语教学满意度产生显著正向影响;双语教学满意度对双语教学忠诚度有正向显著影响,其中双语教学满意度对学生抱怨有负向显著影响,而学生抱怨对双语教学忠诚度有负向显著影响。     

Fitting Structural Equation Model Trees and Latent Growth Curve Mixture Models in Longitudinal Designs: The Influence of Model Misspecification

When conducting longitudinal research, the investigation of between-individual differences in patterns of within-individual change can provide important insights. In this article, we use simulation methods to investigate the performance of a model-based exploratory data mining technique—structural equation model trees (SEM trees; Brandmaier, Oertzen, McArdle, & Lindenberger, 2013)—as a tool for detecting population heterogeneity. We use a latent-change score model as a data generation model and manipulate the precision of the information provided by a covariate about the true latent profile as well as other factors, including sample size, under the possible influences of model misspecifications. Simulation results show that, compared with latent growth curve mixture models, SEM trees might be very sensitive to model misspecification in estimating the number of classes. This can be attributed to the lower statistical power in identifying classes, resulting from smaller differences of parameters prescribed by the template model between classes.     

The Relationship Between Learner Characteristics and Student Outcomes in a Middle School Computing Course: An Exploratory Analysis Using Structural Equation Modeling

This study, which took place during a 7-week middle school computing course, used structural equation modeling to examine the overall cumulative relationship between self-efficacy, interest, and prior computing experience and students’ computer science learning outcomes. The findings indicated that 52% of the variance of student success, measured by a computational thinking quiz and rubric-based evaluations of participants’ computing artifacts, was related to the aforementioned learner characteristics. These findings have implications for theory and practice and suggest that future research and instructional design practice in K–12 computing education should take these learner characteristics into account.     

Digital Module 11: Bayesian Psychometric Modeling https://ncme.elevate.commpartners.com

In this digital ITEMS module, Dr. Roy Levy describes Bayesian approaches to psychometric modeling. He discusses how Bayesian inference is a mechanism for reasoning in a probability-modeling framework and is well-suited to core problems in educational measurement: reasoning from student performances on an assessment to make inferences about their capabilities more broadly conceived, as well as fitting models to characterize the psychometric properties of tasks. The approach is first developed in the context of estimating a mean and variance of a normal distribution before turning to the context of unidimensional item response theory (IRT) models for dichotomously scored data. Dr. Levy illustrates the process of fitting Bayesian models using the JAGS software facilitated through the R statistical environment. The module is designed to be relevant for students, researchers, and data scientists in various disciplines such as education, psychology, sociology, political science, business, health, and other social sciences. It contains audio-narrated slides, diagnostic quiz questions, and data-based activities with video solutions as well as curated resources and a glossary.     

学校文化认同、组织文化氛围与教师满意度对学校效能的影响效应:基于结构方程模型(SEM)的实证研究

基于结构方程模型的研究在回顾相关文献的基础上建立了学校文化认同、组织文化氛围与教师满意度对学校效能影响效应的假设理论模型,运用结构方程模型对假设模型进行了实证分析,结果发现:第一,学校文化认同、人本导向的组织文化氛围对学校效能具有显著影响效应;第二,学校文化认同、人本导向的组织文化氛围对教师的内在满意度具有显著影响效应;第三,教师满意度在学校文化认同、人本导向的组织文化氛围到学校效能之间未发现起中介作用.这一研究从实证的角度验证了学校文化对于学校效能的影响效应,对提高学校的文化领导力具有较好的参考价值.