Alternative Methods for Assessing Mediation in Multilevel Data: The Advantages of Multilevel SEM

Multilevel modeling (MLM) is a popular way of assessing mediation effects with clustered data. Two important limitations of this approach have been identified in prior research and a theoretical rationale has been provided for why multilevel structural equation modeling (MSEM) should be preferred. However, to date, no empirical evidence of MSEM's advantages relative to MLM approaches for multilevel mediation analysis has been provided. Nor has it been demonstrated that MSEM performs adequately for mediation analysis in an absolute sense. This study addresses these gaps and finds that the MSEM method outperforms 2 MLM-based techniques in 2-level models in terms of bias and confidence interval coverage while displaying adequate efficiency, convergence rates, and power under a variety of conditions. Simulation results support prior theoretical work regarding the advantages of MSEM over MLM for mediation in clustered data.     

Using Monte Carlo Normal Distributions to Evaluate Structural Models With Nonnormal Data

Statistical theories of goodness-of-fit tests in structural equation modeling are based on asymptotic distributions of test statistics. When the model includes a large number of variables or the population is not from a multivariate normal distribution, the asymptotic distributions do not approximate the distribution of the test statistics very well at small sample sizes. A variety of methods have been developed to improve the accuracy of hypothesis testing at small sample sizes. However, all these methods have their limitations, specially for nonnormal distributed data. We propose a Monte Carlo test that is able to control Type I error with more accuracy compared to existing approaches in both normal and nonnormally distributed data at small sample sizes. Extensive simulation studies show that the suggested Monte Carlo test has a more accurate observed significance level as compared to other tests with a reasonable power to reject misspecified models.     

Automated Selection of Robust Individual-Level Structural Equation Models for Time Series Data

In order to analyze intensive longitudinal data collected across multiple individuals, researchers frequently have to decide between aggregating all individuals or analyzing each individual separately. This paper presents an R package, gimme, which allows for the automatic specification of individual-level structural equation models that combine group-, subgroup-, and individual-level information. This R package is a complement of the GIMME program currently available via a combination of MATLAB and LISREL. By capitalizing on the flexibility of R and the capabilities of the existing structural equation modeling package lavaan, gimme allows for the automated specification and estimation of group-, subgroup-, and individual-level relations in time series data from within a structural equation modeling framework. Applications include daily diary data as well as functional magnetic resonance imaging data.     

The Theory of Planned Behavior Within the Stages of the Transtheoretical Model: Latent Structural Modeling of Stage-Specific Prediction Patterns in Physical Activity

This is the first study to test whether the stages of change of the transtheoretical model are qualitatively different through exploring discontinuity patterns in theory of planned behavior (TPB) variables using latent multigroup structural equation modeling (MSEM) with AMOS. Discontinuity patterns in terms of latent means and prediction patterns for the different stage groups were examined. Adults (n = 3,462) were assessed on their physical activity stages of change and TPB variables. The TPB was separately examined within the five stage groups. The TPB measurement model fit was acceptable. Latent mean analyses with post-hoc contrast and MSEM indicated discontinuity patterns. Results underscore the qualitative differences between the stages that may guide further research and the design of interventions integrating the approaches.     

The Role of Centering for Interaction of Level 1 Variables in Multilevel Structural Equation Models

This article examined the role of centering in estimating interaction effects in multilevel structural equation models. Interactions are typically represented by product term of 2 variables that are hypothesized to interact. In multilevel structural equation modeling (MSEM), the product term involving Level 1 variables is decomposed into within-cluster and between-cluster random components. The choice of centering affects the decomposition of the product term, and therefore affects the sample variance and covariance associated with the product term used in the maximum likelihood fitting function. The simulation study showed that for an interaction between a Level 1 variable and a Level 2 variable, the product term of uncentered variables or the product term of grand mean centered variables produced unbiased estimates in both Level 1 and Level 2 models. The product term of cluster mean centered variables produced biased estimates in the Level 1 model. For an interaction between 2 Level 1 variables, the product term of cluster mean centered variables produced unbiased estimates in the Level 1 model, whereas the product term of grand mean centered variables produced unbiased estimates for the Level 1 model. Recommendations for researchers who wish to estimate interactions in MSEM are provided.     

A review of using partial least square structural equation modeling in e-learning research

Partial least squares structural equation modeling (PLS-SEM) has become a key multivariate statistical modeling technique that educational researchers frequently use. This paper reviews the uses of PLS-SEM in 16 major e-learning journals, and provides guidelines for improving the use of PLS-SEM as well as recommendations for future applications in e-learning research. A total of 53 articles using PLS-SEM published in January 2009–August 2019 are reviewed. We assess these published applications in terms of the following key criteria: reasons for using PLS-SEM, model characteristics, sample characteristics, model evaluations and reporting. Our results reveal that small sample size and nonnormal data are the first two major reasons for using PLS-SEM. Moreover, we have identified how to extend the applications of PLS-SEM in the e-learning research field.     

Graphical Extension of Sample Size Planning With AIPE on RMSEA Using R

Kelley and Lai (2011) recently proposed the use of accuracy in parameter estimation (AIPE) for sample size planning in structural equation modeling. The sample size that reaches the desired width for the confidence interval of root mean square error of approximation (RMSEA) is suggested. This study proposes a graphical extension with the AIPE approach, abbreviated as GAIPE, on RMSEA to facilitate sample size planning in structural equation modeling. GAIPE simultaneously displays the expected width of a confidence interval of RMSEA, the necessary sample size to reach the desired width, and the RMSEA values covered in the confidence interval. Power analysis for hypothesis tests related to RMSEA can also be integrated into the GAIPE framework to allow for a concurrent consideration of accuracy in estimation and statistical power to plan sample sizes. A package written in R has been developed to implement GAIPE. Examples and instructions for using the GAIPE package are presented to help readers make use of this flexible framework. With the capacity of incorporating information on accuracy in RMSEA estimation, values of RMSEA, and power for hypothesis testing on RMSEA in a single graphical representation, the GAIPE extension offers an informative and practical approach for sample size planning in structural equation modeling.     

Using Mx to Analyze Cross-Level Effects in Two-Level Structural Equation Models

Two-level data sets are frequently encountered in social and behavioral science research. They arise when observations are drawn from a known hierarchical structure, such as when individuals are randomly drawn from groups that are randomly drawn from a target population. Although 2-level data analysis in the context of structural equation modeling can be conducted by easily accessible software such as LISREL, the group- and individual-level effects are usually treated as though they are uncorrelated. When extra group variables are measured and their relationships with individual-level variables are studied, the analysis of cross-level covariance structures is of interest. In this article, we propose a model setup framework in Mx that allows the analysis of cross-level covariance structures. An illustrative example is given and a small-scale simulation study is conducted to examine the performance of the proposed procedure. The results show that the proposed method can produce reliable parameter and standard error estimates, and the goodness-of-fit statistics also follow the chi-square distribution in large samples.     

Mathematical equivalence assessment: Measurement invariance across six countries

Within mathematics education research, there has been a strong focus on students’ understanding of mathematical equivalence because of its key role in the development of mathematics skills. One of the most frequently used tools to assess students’ understanding of equivalence has been the Mathematical Equivalence Assessment (MEA) (Rittle-Johnson et al., 2011). In this study, we investigate for the first time the cross-cultural measurement invariance of an adaptation of the MEA. This included a sample (N = 2760) of students aged 8–12 years old from China, England, New Zealand, South Korea, Turkey, and US to examine whether the same construct is being measured across all countries. Configural and partial scalar invariance was established for a two-factor, 11-item version of the adapted MEA. There were significant mean differences across countries, with students from China performing better and students from New Zealand performing worse than the rest of the sample.     

A Regularized GLS for Structural Equation Modeling

Ill conditioning of covariance and weight matrices used in structural equation modeling (SEM) is a possible source of inadequate performance of SEM statistics in nonasymptotic samples. A maximum a posteriori (MAP) covariance matrix is proposed for weight matrix regularization in normal theory generalized least squares (GLS) estimation. Maximum likelihood (ML), GLS, and regularized GLS test statistics (RGLS and rGLS) are studied by simulation in a 15-variable, 3-factor model with 15 levels of sample size varying from 60 to 100,000. A key result showed that in terms of nominal rejection rates, RGLS outperformed ML at all sample sizes below 500, and GLS at most sample sizes below 500. In larger samples, their performance was equivalent. The second regularization methodology (rGLS) performed well asymptotically, but poorly in small samples. Regularization in SEM deserves further study.     

Selecting Path Models in SEM: A Comparison of Model Selection Criteria

Model comparison is one useful approach in applications of structural equation modeling. Akaike’s information criterion (AIC) and the Bayesian information criterion (BIC) are commonly used for selecting an optimal model from the alternatives. We conducted a comprehensive evaluation of various model selection criteria, including AIC, BIC, and their extensions, in selecting an optimal path model under a wide range of conditions over different compositions of candidate set, distinct values of misspecified parameters, and diverse sample sizes. The chance of selecting an optimal model rose as the values of misspecified parameters and sample sizes increased. The relative performance of AIC and BIC type criteria depended on the magnitudes of the parameter misspecified. The BIC family in general outperformed AIC counterparts unless under small values of omitted parameters and sample sizes, where AIC performed better. Scaled unit information prior BIC (SPBIC) and Haughton's BIC (HBIC) demonstrated the highest accuracy ratios across most of the conditions investigated in this simulation.     

Discriminating Between Measurement Scales Using Nonnested Tests and 2SLS: Monte Carlo Evidence

How should researchers choose between competing scales in predicting a criterion variable? This article proposes the use of nonnested tests for the 2SLS estimator of latent variable models to discriminate between scales. The finite sample performance of these tests is compared to structural equation modeling information-based criteria such as root mean squared error of approximation (RMSEA) and Akaike's Information Criterion (AIC). The Cox and encompassing tests and augmented versions of these tests are compared to the inconsistent ordinary least squares (OLS) J test. An augmented version of the encompassing test performs best for sample sizes of 100 or more and can be recommended for use on scales with high reliability (0.9) and sample sizes of 200 or more, under varying regressor and error distributions. The OLS J test performs best for small samples of N = 50 and can be recommended for use in small samples when scales have high reliability (0.9). Relative to the nonnested tests, the information-based criteria perform poorly.     

Developing a shared cluster construct of instructional leadership in TALIS

In the Teaching and Learning International Survey (TALIS), instructional leadership is measured by the self-reports of principals on three items only. When this measure is investigated together with teacher satisfaction with current work environment, no significant associations were found in the Nordic countries participating in the TALIS 2013 round. This paper argues that a potential reason for this might be the severely underrepresented construct of instructional leadership. As an alternative approach, teacher data from the same study are used to establish two important dimensions of instructional leadership at the school level: 1) managing the instructional program and 2) developing the school learning climate. Applying multilevel structural equation modelling (MSEM), we establish two shared cluster constructs at the school level and observe significant modest relationships between these constructs and teacher job satisfaction with current work environment. The paper brings to our attention the different approaches for interpreting, exploring, and making sense of instructional leadership in international large-scale studies, such as TALIS, from the joint perspective of teachers.     

Power of Latent Growth Modeling for Detecting Group Differences in Linear Growth Trajectory Parameters

This simulation study focused on the power for detecting group differences in linear growth trajectory parameters within the framework of structural equation modeling (SEM) and compared the latent growth modeling (LGM) approach to the more traditional repeated-measures analysis of variance (ANOVA) approach. Several patterns of group differences in linear growth trajectories were considered. SEM growth modeling consistently showed higher statistical power for detecting group differences in the linear growth slope than repeated-measures ANOVA. For small group differences in the growth trajectories, large sample size (e.g., N > 500) would be required for adequate statistical power. For medium or large group differences, moderate or small sample size would be sufficient for adequate power. Some future research directions are discussed.     

Two-Level Dynamic Structural Equation Models with Small Samples

Advances in data collection have made intensive longitudinal data easier to collect, unlocking potential for methodological innovations to model such data. Dynamic structural equation modeling (DSEM) is one such methodology but recent studies have suggested that its small N performance is poor. This is problematic because small N data are omnipresent in empirical applications due to logistical and financial concerns associated with gathering many measurements on many people. In this paper, we discuss how previous studies considering small samples have focused on Bayesian methods with diffuse priors. The small sample literature has shown that diffuse priors may cause problems because they become unintentionally informative. Instead, we outline how researchers can create weakly informative admissible-range-restricted priors, even in the absence of previous studies. A simulation study shows that metrics like relative bias and non-null detection rates with these admissible-range-restricted priors improve small N properties of DSEM compared to diffuse priors.     

Measurement Bias Detection Through Factor Analysis

Assessing the correctness of a structural equation model is essential to avoid drawing incorrect conclusions from empirical research. In the past, the chi-square test was recommended for assessing the correctness of the model but this test has been criticized because of its sensitivity to sample size. As a reaction, an abundance of fit indexes have been developed. The result of these developments is that structural equation modeling packages are now producing a large list of fit measures. One would think that this progression has led to a clear understanding of evaluating models with respect to model misspecifications. In this article we question the validity of approaches for model evaluation based on overall goodness-of-fit indexes. The argument against such usage is that they do not provide an adequate indication of the “size” of the model's misspecification. That is, they vary dramatically with the values of incidental parameters that are unrelated with the misspecification in the model. This is illustrated using simple but fundamental models. As an alternative method of model evaluation, we suggest using the expected parameter change in combination with the modification index (MI) and the power of the MI test.     

The Incorporation of Sample Weights Into Multilevel Structural Equation Models

Over the past decade and a half, methodologists working with structural equation modeling (SEM) have developed approaches for accommodating multilevel data. These approaches are particularly helpful when modeling data that come from complex sampling designs. However, most data sets that are associated with complex sampling designs also include observation weights, and methods to incorporate these sampling weights into multilevel SEM analyses have not been addressed. This article investigates the use of different weighting techniques and finds, through a simulation study, that the use of an effective sample size weight provides unbiased estimates of key parameters and their sampling variances. Also, a popular normalization technique of scaling weights to reflect the actual sample size is shown to produce negatively biased sampling variance estimates, as well as negatively biased within-group variance parameter estimates in the small group size case.     

The relationship between epistemological beliefs,learning conceptions,and approaches to study: a cross-cultural structural model?

Recent research has shown interest in studying the relationship between epistemological beliefs and numerous aspects of learning. A new question interests us: Is this kind of relationship homogeneous across cultures? This study focuses on the relationship between epistemological beliefs, learning conceptions, and approaches to study. A sample of Chinese (n = 299) and Flemish (n = 324) first-year university students in Beijing, China and Flanders were involved in the study. A structural equation model (SEM) relating the three concepts was applied to the sample data, largely confirming the theoretical assumptions. The results validated the postulation that epistemological beliefs predict students' conceptions of learning, which in turn are related to specific approaches to study. Multiple group analysis using SEM was applied and the structural weights model was confirmed across the two cultural groups. Mean level variations of the three main concepts were detected between the Chinese and Flemish groups. The results identified in the study offer valuable contributions to a deeper understanding of the interplay between epistemological beliefs and student learning from a cross-cultural perspective. Implications for learning and instruction are discussed.     

The negative effect of school-average ability on science self-concept in the UK,the UK countries and the world: the Big-Fish-Little-Pond-Effect for PISA 2006

Research on the relation between students’ achievement (ACH) and their academic self-concept (ASC) has consistently shown a Big-Fish-Little-Pond-Effect (BFLPE); ASC is positively affected by individual ACH, but negatively affected by school-average ACH. Surprisingly, however, there are few good UK studies of the BFLPE and few anywhere in the world based on science self-concept (S-ASC). Addressing this substantive limitation in existing research with data from PISA 2006, we extend new multigroup doubly-latent multilevel structural equation models – a substantive-methodological synergy. BFLPE predictions for S-ASC are supported for: the total international sample; the total UK sample; each of the four UK countries considered separately. The BFLPE was marginally larger in the UK than the international sample. However, consistent with the selective nature of school systems in the UK, the BFLPE was larger in Northern Ireland and, to a lesser extent, England than in Scotland or Wales.     

Teacher feedback,disciplinary climate,student self-concept,and reading achievement:A multilevel moderated mediation model

Research has confirmed the importance of teacher feedback on student learning. The mechanism of how they are related, however, is not clear enough. In this study, we explored this relation with 60,501 fifteen-year-old students from collectivistic and individualistic cultures in PISA 2018. Importantly, we examined the possible mediating role of reading self-concept and the moderating role of disciplinary climate at both student level and school level in multi-level structural equation models (MSEM). Results demonstrated that the association between teacher feedback and reading achievement was significantly mediated via reading self-concept at student level across cultures, and this indirect effect was significant irrespective of the disciplinary climate level. Moreover, results showed that a positive disciplinary climate would facilitate the building of students’ reading self-concept which subsequently would enhance their reading achievement at school level. This study has important theoretical, practical, cross-cultural, and methodological implications for teacher feedback research and student learning.