Multilevel Factorial Invariance in n-Level Structural Equation Modeling (nSEM)

We present a multigroup multilevel confirmatory factor analysis (CFA) model and a procedure for testing multilevel factorial invariance in n-level structural equation modeling (nSEM). Multigroup multilevel CFA introduces a complexity when the group membership at the lower level intersects the clustered structure, because the observations in different groups but in the same cluster are not independent of one another. nSEM provides a framework in which the multigroup multilevel data structure is represented with the dependency between groups at the lower level properly taken into account. The procedure for testing multilevel factorial invariance is illustrated with an empirical example using an R package xxm2.     

Estimation of MIMIC Model Parameters with Multilevel Data

The purpose of this simulation study was to assess the performance of latent variable models that take into account the complex sampling mechanism that often underlies data used in educational, psychological, and other social science research. Analyses were conducted using the multiple indicator multiple cause (MIMIC) model, which is a flexible and effective tool for relating observed and latent variables. The data were simulated in a hierarchical framework (e.g., individuals nested in schools) so that a multilevel modeling approach would be appropriate. Analyses were conducted accounting for and not accounting for the nested data to determine the impact of ignoring such multilevel data structures in full structural equation models. Results highlight the differences in modeling results when the analytic strategy is congruent with the data structure and what occurs when this congruency is absent. Type I error rates and power for the standard and multilevel methods were similar for within-cluster variables and for the multilevel model with between-cluster variables. However, Type I error rates were inflated for the standard approach when modeling between-cluster variables.     

A multilevel examination of an instrument measuring school renewal via teachers

The purpose of the present study was to assess an instrument that uses responses from teachers to measure principal behaviors on the critical educational issue of school renewal. To account for the data hierarchy of teachers nested within schools, we employed a multilevel confirmatory factor analysis and estimated multilevel reliabilities at both teacher and school levels to examine the instrument. With 2765 teachers from 148 schools, the instrument showed acceptable structural validity. Specifically, we found that when teachers provide responses as indicators of school renewal, the instrument can generate a valid and highly reliable measure or estimate of school renewal at the school level. However, data analysis at the teacher level attempting to use teacher perceptions of school renewal as either dependent variables or independent variables should be avoided because of the low reliability at the teacher level.     

Applications of Multilevel IRT Modeling

The recent development of multilevel IRT models (Fox & Glas, 2001, 2003) has been shown to be very useful for analyzing relationships between observed variables on different levels containing measurement error. Model parameter estimates and their standard deviations are concurrently estimated taking account of measurement error in observed variables. The multilevel IRT models are, in particular, useful in the analysis of school effectiveness research data since hierarchical structured educational data are subject to error. By re-examining some school effectiveness studies, the basic aspects of this new model and consequences of measurement error are shown.     

The Effects of Ignoring a Level in Multilevel Analysis

Ignoring a level can have a substantial impact on the conclusions of a multilevel analysis. For intercept-only models and for balanced data, we derive these effects analytically. For more complex random intercept models or for unbalanced data, a simulation study is performed. Most important effects concern estimates and corresponding standard errors of the variance parameters at adjacent levels and of the coefficients of the predictors at the ignored and bordering levels. Therefore, we conclude that if the researcher is interested in a specific level, she/he should account for both the upper and lower level. Conclusions are illustrated using empirical data from educational research.     

Measurement and evaluation issues in education: The value of multivariate techniques in evaluating an innovative high school reform program

The purpose of this chapter is twofold. The first is to explore uses of new methods for the analysis of multilevel data in school effectiveness research with limited observed data, where the word “limited” is used because more often than not interesting information about changes in teaching styles and the influence of teachers on students is limited, and sometimes incomplete. The statistical and mathematical models we use to analyze our data should allow us to deal with these facts. Secondly a contribution is made to school effectiveness research by establishing that multilevel causes determine student achievement. The combination of student ability, a new curriculum and teacher satisfaction predict student achievement in a multilevel multivariate analysis. The rich data set contains 342 students, 20 teachers and more than 50 variables for each level, the student level and the teacher level. Half of the teachers were in an experimental condition, teaching a new type of class, called Humanitas, the other half taught history in a traditional way, and served as the control group in twelve schools in Los Angeles county.The statistical models used in this chapter can be described as (1) a data reduction model, where several observed variables are combined into a single scale, (2) a regression model for multilevel data, with a single dependent variable, and (3) a model for multilevel multivariate analysis. One of the conclusions of this chapter is that researchers should have a certain amount of knowledge of how the data are generated regarding the important relationships between the variables, before choosing a complicated model. It is advisable to explore the data, guided by a theoretical model, before fitting complicated hierarchical bilevel path models with latent variables.Above statements are in support of Rubin's statement (1991, p. 306): “that complicated analyses are often necessary in complicated problems, but unless their conclusions can be represented in clear summaries that can be supported by the real data and the observations in the field, they should not be trusted. Too often, the results of complicated analyses are summarized without making clear how the conclusions are consistent with the raw data”. The multilevel analyses in this chapter are such complicated analysis tools. The data are experimental data, collected under close observation by the researchers. The knowledge gained from the complicated data analysis is supported by observations, interviews and simple exploratory data analysis. The positive effect, on students and teachers, of an experimental curriculum in ten secondary high schools is revealed by multilevel path analysis with latent variables.     

Optimal Design for Two-Level Random Assignment and Regression Discontinuity Studies

An important concern when planning research studies is to obtain maximum precision of an estimate of a treatment effect given a budget constraint. When research designs have a multilevel or hierarchical structure changes in sample size at different levels of the design will impact precision differently. Furthermore, there will typically be differential costs of enrolling additional units at different levels of the hierarchy. The optimal design problem in multilevel research studies involves determining the optimal sample size at each level of the design given specified design parameters and a specified marginal cost of recruitment at each level. The current work extends existing results by considering optimal design for (a) unbalanced random assignment designs and (b) regression discontinuity designs.     

Alternatives to Multilevel Modeling for the Analysis of Clustered Data

Multilevel modeling has grown in use over the years as a way to deal with the nonindependent nature of observations found in clustered data. However, other alternatives to multilevel modeling are available that can account for observations nested within clusters, including the use of Taylor series linearization for variance estimation, the design effect adjusted standard errors approach, and fixed effects modeling. Using 1,000 replications of 12 conditions with varied Level 1 and Level 2 sample sizes, the author compared parameter estimates, standard errors, and statistical significance using various alternative procedures. Results indicate that several acceptable procedures can be used in lieu of or together with multilevel modeling, depending on the type of research question asked and the number of clusters under investigation. Guidelines for applied researchers are discussed.     

Analysis of the Effects of School Variables Using Multilevel Models

Multilevel models allow data to be analysed which are hierarchical in nature; in particular, data which have been collected on pupils grouped into schools. Some of the associated variables may be measured at the pupil level, and others at the school level. The use of multilevel models produces estimates of variances between schools and pupils, as well as the effects of background variables in reducing or explaining these variances. One data set which has been analysed relates to the national surveys of mathematics carried out in England, Wales and Northern Ireland. In this case the basic unit of analysis was a pupil's performance in a group of items within one of 12 sub‐categories of maths. Each pupil tackled two such item groups (or sub‐tests) and thus a three‐level model was required, with the levels representing sub‐tests, pupils and schools. A number of background variables at both pupil and school levels were also measured, and interesting results were obtained when a multilevel model was fitted. The program used was a version of one developed by Professor H. Goldstein. A quite different data set related to pupils’ responses to a questionnaire survey about their reactions to their current course of study. The dependent variable was a measure of pupils’ satisfaction with the course derived from their responses, and other pupil level variables were also derived, relating to their school experiences and personal attributes. School level variables such as size and type of school were obtained from a schools data base. The program Hierarchical Linear Model (HLM) was used to model these data, using only two levels. The two multilevel program used have different strengths and capabilities, but are related in terms of the kinds of models that can be fitted. Such models can lead to greater insights into the relationships between school and pupil level variables, and their influence on pupil results or attitudes.     

Investigating the relationship between students’ attentive–inattentive behaviors in the classroom and their literacy progress: Chapter 3 Model specifications,results and comments: Fitting conditional models

The fitting of simple, three-level, variance-components models to estimate the proportion of variance in each of the continuous variables due to the inherent multilevel structure of the data (i.e., students within class/teacher groups clustered within schools), identified substantial residual variation at the class/teacher-level for several variables. To explain variation in students’ literacy progress, and their attentive–inattentive behaviors, respectively, two traditional, unidirectional models are then specified and fitted to each of the data sets. The results of fitting both single-level and multi-level versions of these models are compared and evaluated critically in terms of their explanatory power and their links with the research literature. As a basis for subsequently specifying and fitting three multilevel, non-recursive, structural equation models to the data in Chapter 4, the findings from fitting multivariate, multilevel models to each data set are presented. This is done to estimate the variances and covariances among the variables — both at the student-level and at the class/teacher-level.     

A More Powerful Test in Three-Level Cluster Randomized Designs

Abstract

Field experiments that involve nested structures frequently assign treatment conditions to entire groups (such as schools). A key aspect of the design of such experiments includes knowledge of the clustering effects that are often expressed via intraclass correlation. This study provides methods for constructing a more powerful test for the treatment effect in three-level cluster randomized designs with two levels of nesting (at the second and third levels). When the intraclass correlation structure at the second and third level is assumed to be known, the proposed test provides higher estimates of power than those obtained from the typical test based on level-3 unit means, because it preserves the degrees of freedom associated with the number of level-2 and level-1 units. The advantage in power estimates is more pronounced when the number of level-3 units (e.g., schools) is small and the samples are homogeneous (e.g., low-achieving schools).     

The Power of the Test for Treatment Effects in Three-Level Block Randomized Designs

Abstract

Experiments that involve nested structures may assign treatment conditions either to subgroups (such as classrooms) or individuals within subgroups (such as students). The design of such experiments requires knowledge of the intraclass correlation structure to compute the sample sizes necessary to achieve adequate power to detect the treatment effect. This study provides methods for computing power in three-level block randomized balanced designs (with two levels of nesting) where, for example, students are nested within classrooms and classrooms are nested within schools. The power computations take into account nesting effects at the second (classroom) and at the third (school) level, sample size effects (e.g., number of level-1, level-2, and level-3 units), and covariate effects (e.g., pretreatment measures). The methods are generalizable to quasi-experimental studies that examine group differences on an outcome.     

A Multilevel CFA–MTMM Approach for Multisource Feedback Instruments: Presentation and Application of a New Statistical Model

Multisource feedback instruments are a widely used tool in human resource management. However, comprehensive validation studies remain scarce and there is a lack of statistical models that account appropriately for the complex data structure. Because both peers and subordinates are nested within the target but stem from different populations, the assumption of traditional multilevel structural equation models that the sample on a lower level stems from the same population is violated. We present a multilevel confirmatory factor analysis multitrait–multimethod (ML–CFA–MTMM) model that considers this peculiarity of multisource feedback instruments. The model is applied to 2 scales of the Benchmarks® instrument and it is demonstrated how measures of reliability and of convergent and discriminant validity can be obtained using multilevel structural equation modeling software. We discuss the results as well as some implications and guidelines for the use of the model.     

Scientific Reasoning,School Achievement and Gender: a Multilevel Study of between and within School Effects in Finland

The relationships between reasoning and school achievement were studied taking into account the multilevel nature (school- and class-levels) of the data. We gathered data from 51 classes at seven schools in metropolitan and Eastern Finland (N = 769, 395 males, 15-year-old students). To study scientific reasoning, we used a modified version of Science Reasoning Tasks, tapping control-of-variable schemata. Analyses were conducted by MLwiN2.10 multilevel modelling. The present results showed that the intra-class correlation coefficient (ICC) of schools for scientific reasoning is 7% and that the corresponding ICC of classes is 10%. Whereas the first finding confirms earlier PISA results, the second finding provides new insights into class variation within schools. In practice, class composition seems to be an efficient solution to meeting the differing needs of individual students.     

Using Multilevel Regression Mixture Models to Identify Level-1 Heterogeneity in Level-2 Effects

This article proposes a novel exploratory approach for assessing how the effects of Level-2 predictors differ across Level-1 units. Multilevel regression mixture models are used to identify latent classes at Level 1 that differ in the effect of 1 or more Level-2 predictors. Monte Carlo simulations are used to demonstrate the approach with different sample sizes and to demonstrate the consequences of constraining 1 of the random effects to 0. An application of the method to evaluate heterogeneity in the effects of classroom practices on students is used to show the types of research questions that can be answered with this method and the issues faced when estimating multilevel regression mixtures.     

A More Flexible Bayesian Multilevel Bifactor Item Response Theory Model

Multilevel bifactor item response theory (IRT) models are commonly used to account for features of the data that are related to the sampling and measurement processes used to gather those data. These models conventionally make assumptions about the portions of the data structure that represent these features. Unfortunately, when data violate these models' assumptions but these models are used anyway, incorrect conclusions about the cluster effects could be made and potentially relevant dimensions could go undetected. To address the limitations of these conventional models, a more flexible multilevel bifactor IRT model that does not make these assumptions is presented, and this model is based on the generalized partial credit model. Details of a simulation study demonstrating this model outperforming competing models and showing the consequences of using conventional multilevel bifactor IRT models to analyze data that violate these models' assumptions are reported. Additionally, the model's usefulness is illustrated through the analysis of the Program for International Student Assessment data related to interest in science.     

Measurement Bias in Multilevel Data

Measurement bias can be detected using structural equation modeling (SEM), by testing measurement invariance with multigroup factor analysis (Jöreskog, 1971;Meredith, 1993;Sörbom, 1974) MIMIC modeling (Muthén, 1989) or restricted factor analysis (Oort, 1992,1998). In educational research, data often have a nested, multilevel structure, for example when data are collected from children in classrooms. Multilevel structures might complicate measurement bias research. In 2-level data, the potentially “biasing trait” or “violator” can be a Level 1 variable (e.g., pupil sex), or a Level 2 variable (e.g., teacher sex). One can also test measurement invariance with respect to the clustering variable (e.g., classroom). This article provides a stepwise approach for the detection of measurement bias with respect to these 3 types of violators. This approach works from Level 1 upward, so the final model accounts for all bias and substantive findings at both levels. The 5 proposed steps are illustrated with data of teacher–child relationships.     

Use of Standard Deviations as Predictors in Models Using Large-Scale International Data Sets

Measures of variability are successfully used in predictive modeling in research areas outside of education. This study examined how standard deviations can be used to address research questions not easily addressed using traditional measures such as group means based on index variables. Student survey data were obtained from the Organisation for Economic Co-operation and Development to examine standard deviation predictors in multilevel models. These predictors and interactions explained additional variation in the dependent variable beyond the control variables. Models using biased and unbiased standard deviations were compared. Meaningful differences were found between the models. Findings supported how standard deviation predictors may increase explanatory power and accuracy of models commonly used in educational research.     

The Performance of Multilevel Models When Outcome Data Are Incomplete

When data for multiple outcomes are collected in a multilevel design, researchers can select a univariate or multivariate analysis to examine group-mean differences. When correlated outcomes are incomplete, a multivariate multilevel model (MVMM) may provide greater power than univariate multilevel models (MLMs). For a two-group multilevel design with two correlated outcomes, a simulation study was conducted to compare the performance of MVMM to MLMs. The results showed that MVMM and MLM performed similarly when data were complete or missing completely at random. However, when outcome data were missing at random, MVMM continued to provide unbiased estimates, whereas MLM produced grossly biased estimates and severely inflated Type I error rates. As such, this study provides further support for using MVMM rather than univariate analyses, particularly when outcome data are incomplete.     

The Innovative School: Organization and Instruction

Researchers in the area of educational effectiveness should attempt to develop a new theoretical framework. A critical analysis of the current models of educational effectiveness research is provided and reveals that a dynamic model of effectiveness must: (a) be multilevel in nature, (b) be based on the assumption that the relation of some effectiveness factors with achievement may be curvilinear, (c) illustrate the dimensions upon which the measurement of each effectiveness factor should be based, and (d) define relations among the effectiveness factors. In principle, each factor that refers to the classroom, school, and system, can be measured by taking into account five dimensions: frequency, focus, stage, quality, and differentiation. Examples of measuring effectiveness factors operating at different levels using these 5 dimensions are given. More attention in describing in detail factors associated with teacher behaviour in the classroom is given, since this is seen as the starting point for the development and the testing of the dynamic model. Finally, suggestions for the next steps in the development of other parts of the model are provided.