Using SEM to Analyze Complex Survey Data: A Comparison between Design-Based Single-Level and Model-Based Multilevel Approaches

Both ad-hoc robust sandwich standard error estimators (design-based approach) and multilevel analysis (model-based approach) are commonly used for analyzing complex survey data with nonindependent observations. Although these 2 approaches perform equally well on analyzing complex survey data with equal between- and within-level model structures (B. O. Muthén & Satorra, 1995), the performances of these 2 approaches for analyzing multilevel data with unequal between- and within-level structures have not yet been systematically examined. In this study, we extended B. O. Muthén and Satorra's (1995) Muthén, B. O. and Satorra, A. 1995. Complex sample data in structural equation modeling. Sociological Methodology, 25: 267–316. [Crossref], [Web of Science ®] , [Google Scholar] study by comparing these 2 approaches and an additional model-based maximum model for analyzing multilevel data considering number of clusters, cluster size, intraclass correlation, and the equality of different level structures. The simulation results showed the model-based maximum model generally performed well across conditions. This model is also recommended as an alternative for analyzing nonindependent survey data, especially when the information of the higher level model structure is not known.     

Association between schoolwide positive behavioural interventions and supports and academic achievement: a 9-year evaluation

This study evaluated the long-term impact of schoolwide positive behavioural interventions and supports (PBIS) on student academic achievement. In this quasi-experimental study, academic achievement data were collected over 9 years. The 21 elementary, middle, and high schools that achieved moderate to high fidelity to the Save & Civil Schools’ PBIS model were matched with 28 control schools to assess academic gains. There were 5 years of baseline data (no intervention in treatment schools) and 4 years of intervention data, including 1 year of maintenance. Results indicate that implementation of the PBIS programme was significantly associated with increased student academic achievement (p = .001) and that the rate of change for students’ academic achievement in treatment schools was greater than for students in control schools. This study suggests that PBIS programmes, such as Safe & Civil Schools’ Foundations, may complement other efforts to improve academic outcomes.     

Analyzing measurement models of latent variables through multilevel confirmatory factor analysis and hierarchical linear modeling approaches

Confirmatory factor analysis (CFA) is often used in the social sciences to estimate a measurement model in which multiple measurement items are hypothesized to assess a particular latent construct. This article presents the utility of multilevel CFA (MCFA; Muthén, 1991, 1994) and hierarchical linear modeling (HLM; Raudenbush, Rowan, & Kang, 1991) methods in testing measurement models in which the underlying attribute may vary as a function of various levels of observation. An illustrative example using a real dataset is provided in which an unconditional model specification and parameter estimates from the MCFA and HLM are shown. The article demonstrates the comparability of the two methods in estimating measurement parameters of interest (i.e., true variance at levels the measures are used and measurement errors).     

An Investigation of the Alignment Method With Polytomous Indicators Under Conditions of Partial Measurement Invariance

The alignment method (Asparouhov & Muthén, 2014) is an alternative to multiple-group factor analysis for estimating measurement models and testing for measurement invariance across groups. Simulation studies evaluating the performance of the alignment for estimating measurement models across groups show promising results for continuous indicators. This simulation study builds on previous research by investigating the performance of the alignment method’s measurement models estimates with polytomous indicators under conditions of systematically increasing, partial measurement invariance. We also present an evaluation of the testing procedure, which has not been the focus of previous simulation studies. Results indicate that the alignment adequately recovers parameter estimates under small and moderate amounts of noninvariance, with issues only arising in extreme conditions. In addition, the statistical tests of invariance were fairly conservative, and had less power for items with more extreme skew. We include recommendations for using the alignment method based on these results.     

Relating Latent Change Score and Continuous Time Models

The primary goal of this article is to demonstrate the close relationship between 2 classes of dynamic models in psychological research: latent change score models and continuous time models. The secondary goal is to point out some differences. We begin with a brief review of both approaches, before demonstrating how the 2 methods are mathematically and conceptually related. It will be shown that most commonly used latent change score models are related to continuous time models by the difference equation approximation to the differential equation. One way in which the 2 approaches differ is the treatment of time. Whereas there are theoretical and practical restrictions regarding observation time points and intervals in latent change score models, no such limitations exist in continuous time models. We illustrate our arguments with three simulated data sets using a univariate and bivariate model with equal and unequal time intervals. As a by-product of this comparison, we discuss the use of phantom and definition variables to account for varying time intervals in latent change score models. We end with a reanalysis of the Bradway–McArdle longitudinal study on intellectual abilities (used before by McArdle & Hamagami, 2004) by means of the proportional change score model and the dual change score model in discrete and continuous time.     

Prediction of self-reported knowledge with over-claiming, fluid and crystallized intelligence and typical intellectual engagement

We investigated the usefulness of the Over-Claiming Questionnaire (OCQ) as a measure of cognitive abilities. In OCQs respondents are asked to rate their familiarity with items of academic or everyday knowledge (Paulhus, Harms, Bruce, & Lysy, 2003). Some items exist in reality (reals), and others do not (foils). We developed four OCQs, each consisting of 40 reals and 8 foils from the domains of Science, Humanities and Civics. The OCQs were administered in a longitudinal rotation design to 112 participants who attended the 9th school grade at the beginning of the study. In latent variable regression analyses 53% of variation in the reals could be explained by fluid and crystallized intelligence and over-claiming as indicated by responses to foils. Further variation in responses to reals and foils was explained by intellectual engagement. Our results show that self-reported knowledge, although positively related to measures of ability, to a large extent reflects over-claiming.     

Bias-Adjusted Three-Step Latent Markov Modeling With Covariates

Latent Markov models with covariates can be estimated via 1-step maximum likelihood. However, this 1-step approach has various disadvantages, such as that the inclusion of covariates in the model might alter the formation of the latent states and that parameter estimation could become infeasible with large numbers of time points, responses, and covariates. This is why researchers typically prefer performing the analysis in a stepwise manner; that is, they first construct the measurement model, then obtain the latent state classifications, and subsequently study the relationship between covariates and latent state memberships. However, such a stepwise approach yields downward-biased estimates of the covariate effects on initial state and transition probabilities. This article, shows how to overcome this problem using a generalization of the bias-corrected 3-step estimation method proposed for latent class analysis (Asparouhov & Muthén, 2014; Bolck, Croon, & Hagenaars, 2004; Vermunt, 2010). We give a formal derivation of the generalization to latent Markov models and discuss how it can be used with many time points by incorporating it into a Baum–Welch type of expectation-maximization algorithm. We evaluate the method through a simulation study and illustrate it using an application on household financial portfolio change. Our study shows that the proposed correction method yields unbiased parameter estimates and accurate standard errors, except for situations with very poorly separated classes and a small sample.     

Parent-child cohesion,self-esteem,and academic achievement: The longitudinal relations among elementary school students

This study addressed the longitudinal reciprocal relations among parent-child cohesion, self-esteem, and academic achievement in elementary school students. A total of 659 students from Grades 4–5 in China (M_age = 10.09 years, 46.3% female) was investigated on five occasions across 12 months. The results indicated that: (a) For the direct reciprocal relations, mother-child cohesion and academic achievement reciprocally enhanced each other; self-esteem and academic achievement transactionally facilitated each other; father-child cohesion and self-esteem reciprocally enhanced each other; self-esteem facilitated mother-child cohesion, but not vice versa. (b) For the indirect relations, father-child cohesion and students’ academic achievement facilitated each other indirectly via self-esteem; students’ academic achievement influenced mother-child cohesion indirectly via self-esteem, but not vice versa. These results delineated the dynamic system among these study variables, and also revealed that father-child cohesion and mother-child cohesion showed both similarities and differences. Implications for educational practices were discussed.     

Assessing the effectiveness of learning solid geometry by using an augmented reality-assisted learning system

This study integrates augmented reality (AR) technology into teaching activities to design a learning system that assists junior high-school students in learning solid geometry. The following issues are addressed: (1) the relationship between achievements in mathematics and performance in spatial perception; (2) whether system-assisted learning can improve the spatial perceptions of students; (3) whether students with high, average and low academic achievement learn effectively after taught with system assistance; (4) system usability; (5) system task load; and (6) the relationship among various factors. Study participants were 76 students from Tainan City, Taiwan. Qualitative and quantitative data are obtained using pre- and post-system-assisted learning paper–pencil tests, a system usability scale, National Aeronautics & Space Administration Task Load indeX, and observations and focus group interviews. Analysis results indicate that students' math scores and spatial perception during the test closely correspond to each other. Additionally, system-assisted can improve the spatial perceptions of student. Regarding group differences related to the significance of impact, students with average and low academic achievements exhibit small and medium levels of effectiveness, but are insignificant in high academic achievement students. As for system usability, students with low academic achievements demonstrate a positive attitude toward the intensive programs provided by the system. However, the correlation between students’ learning effectiveness and system usability/task load is low.     

Kindergarten Children's Executive Functions Predict Their Second-Grade Academic Achievement and Behavior

Whether and to what extent kindergarten children's executive functions (EF) constitute promising targets of early intervention is currently unclear. This study examined whether kindergarten children's EF predicted their second-grade academic achievement and behavior. This was done using (a) a longitudinal and nationally representative sample (N = 8,920, M_age = 97.6 months), (b) multiple measures of EF, academic achievement, and behavior, and (c) extensive statistical control including for domain-specific and domain-general lagged dependent variables. All three measures of EF—working memory, cognitive flexibility, and inhibitory control—positively and significantly predicted reading, mathematics, and science achievement. In addition, inhibitory control negatively predicted both externalizing and internalizing problem behaviors. Children's EF constitute promising targets of experimentally evaluated interventions for increasing academic and behavioral functioning.     

Phantom and big-fish-little-pond-effects on academic self-concept and academic achievement: Evidence from English early primary schools

The Big-Fish-Little-Pond Effect (BFLPE) suggests that school-average achievement has a negative effect on academic self-concept (ASC); some research has also verified a negative effect on students' academic achievement. Our study evaluates the compositional effects of school-average achievement on both outcomes, using a longitudinal sample of English early primary school students in Year 1 and Year 4. We provide evidence for BFLPEs in children as young as six to nine years of age. Further, we show that the BFLPE is a potential mechanism in the negative compositional effect of school average achievement in Year 1 on students' achievement in Year 4. Once adjustments for measurement error are made, the negative effect of school-average achievement on students' self-concept, and on their subsequent achievement, becomes even more negative. Our findings question previous research suggesting that attending a school with higher average achievement necessarily advances students’ outcomes.     

From Crisis Management to Academic Achievement: A University Cluster-mentoring Model for Black Undergraduates

In spite of the widening racial achievement gap among U.S. college students (U.S. Census Bureau, 2011), some universities are achieving success in supporting the graduation and postcollege goals of Black undergraduates (Apprey, Bassett, Preston-Grimes, Lewis, & Wood 2014/this issue; Baker, 2006; Hrabowski, 2003; Hrabowski & Maton, 2009). Although research has documented efforts to improve students’ college academic success in mathematics, science, and engineering (Maton & Hrabowski, 2004), little research has examined the role of undergraduate support programs across the academic disciplines to bridge success for students from high school graduation through graduate school matriculation. This is a key link in the pipeline to career and lifelong achievement for Black students. The following case study describes an inclusive cluster-mentoring model for Black undergraduates at a Research I university that includes four elements—(a) student peer-advising, (b) faculty–student academic mentoring and advising, (c) culturally sensitive initiatives, and (d) organized parental support—to create high impact with measurable results. This university-based model can serve as a guide to improve and expand services that support the academic and leadership success of Black undergraduate students in other higher education settings.     

Fitting Structural Equation Model Trees and Latent Growth Curve Mixture Models in Longitudinal Designs: The Influence of Model Misspecification

When conducting longitudinal research, the investigation of between-individual differences in patterns of within-individual change can provide important insights. In this article, we use simulation methods to investigate the performance of a model-based exploratory data mining technique—structural equation model trees (SEM trees; Brandmaier, Oertzen, McArdle, & Lindenberger, 2013)—as a tool for detecting population heterogeneity. We use a latent-change score model as a data generation model and manipulate the precision of the information provided by a covariate about the true latent profile as well as other factors, including sample size, under the possible influences of model misspecifications. Simulation results show that, compared with latent growth curve mixture models, SEM trees might be very sensitive to model misspecification in estimating the number of classes. This can be attributed to the lower statistical power in identifying classes, resulting from smaller differences of parameters prescribed by the template model between classes.     

Subjective evaluations of intelligence and academic self-concept predict academic achievement: Evidence from a selective student population

The study examined the relationship between implicit theories, goal orientations, subjective and test estimates of intelligence, academic self-concept, and achievement in a selective student population (N = 300). There was no direct impact of implicit theories of intelligence and goal orientations on achievement. However, subjective evaluations of intelligence and academic self-concept had incremental predictive value over conventional intelligence when predicting achievement accounting for more than 50% of its variance. The obtained pattern of results is presented via structural equation models and interpreted within a dynamic regulative systems framework suggesting the importance of further studying complex sets of achievement predictors that include ability, personality and mediating constructs.     

Developmental outcomes of children in classes for special educational needs: results from a longitudinal study

This longitudinal study investigates the differences in cognitive and socio‐emotional development and academic achievement between children educated in special education classes (N = 37) and regular classes (N = 37). The study is retrospective. The first measurement point was while children were attending play‐oriented kindergarten and no decision about their education had yet been made. The second measurement point followed after 2 years of schooling. Comparing carefully matched groups, no differences in executive functions (EFs) were found before beginning school. Children assigned to special education had poorer language, fine motor skills and a lower pre‐academic self‐concept, self‐regulatory skills and social integration. Notably, every fourth child in special education was an immigrant, 9% of whom later attended regular classes. After 2 years of schooling in either setting, the groups differed significantly in academic achievement, EFs, fine motor skills and cognitive self‐regulatory skills. However, it was not – as school officials had intended – that children in special education classes had caught up, except in regard to their academic self‐concept and social integration.     

Advanced Nonlinear Latent Variable Modeling: Distribution Analytic LMS and QML Estimators of Interaction and Quadratic Effects

Interaction and quadratic effects in latent variable models have to date only rarely been tested in practice. Traditional product indicator approaches need to create product indicators (e.g., x ₁ ², x ₁ x ₄) to serve as indicators of each nonlinear latent construct. These approaches require the use of complex nonlinear constraints and additional model specifications and do not directly address the nonnormal distribution of the product terms. In contrast, recently developed, easy-to-use distribution analytic approaches do not use product indicators, but rather directly model the nonlinear multivariate distribution of the measured indicators. This article outlines the theoretical properties of the distribution analytic Latent Moderated Structural Equations (LMS; Klein & Moosbrugger, 2000) and Quasi-Maximum Likelihood (QML; Klein & Muthén, 2007) estimators. It compares the properties of LMS and QML to those of the product indicator approaches. A small simulation study compares the two approaches and illustrates the advantages of the distribution analytic approaches as multicollinearity increases, particularly in complex models with multiple nonlinear terms. An empirical example from the field of work stress applies LMS and QML to a model with an interaction and 2 quadratic effects. Example syntax for the analyses with both approaches is provided.     

Middle school children's strategic behavior: Classification and relation to academic achievement and mathematical problem solving

Theories of problem solving (e.g., Verschaffelet al., 2000) hold strategic behavior centralto processing mathematical word problems. Thepresent study explores 80 sixth- andseventh-grade students' self-reported use of 14categories of strategies (Zimmerman &Martinez-Pons, 1986) and the relationship ofstrategy use to academic achievement,problem-solving behaviors, and problem-solvingsuccess. High and low achievement groupsdiffered in the number of different strategiesand categories of strategies reported but notin overall number of strategies, confidence inusing strategies, or frequency of strategy use.Students whose behaviors evidenced elaborationof the word problem's text reported moreself-evaluation; organizing and transforming;and goal setting and monitoring behavior.Implications for instructional practices thatsupport active stances toward problem solvingare discussed.     

Simultaneously Student and Teacher: Measuring Achievement Goals in Preservice Teachers

Abstract

Preservice teachers can be considered simultaneously students and teachers and therefore likely have both academic and professional goals. However, once in a professional program, predicting professional outcomes becomes somewhat more important than academic ones. This distinction may have implications for the selection of measurement tools used in research on preservice teachers’ motivation. We used a multitrait-multimethod (MTMM) design that included tests of alternative confirmatory factor analyses (CFAs), within and between method correlations, and relations with other variables to compare two measures of achievement goals: Elliot and Murayama’s (2008 Elliot, A. J., & Murayama, K. (2008). On the measurement of achievement goals: Critique, illustration, and application. Journal of Educational Psychology, 100(3), 613–628. doi: 10.1037/0022-0663.100.3.613[Crossref], [Web of Science ®] , [Google Scholar]) measure of students’ achievement goals and Butler’s (2007) measure of teachers’ achievement goals. Results of the CFAs suggested that the scales are measuring separate constructs. The MTMM correlations, however, revealed evidence that certain factors may function similarly. This was most evident for the homotrait-heteromethod factors of mastery approach, which correlated similarly with sense of self-efficacy, emotions, and classroom mastery goal structures.     

Academic Self-Perceptions and Academic Achievement in Chinese Children: A Multiwave Longitudinal Study

This study examined reciprocal contributions between academic self-perceptions and academic achievement. Data were collected each year in four consecutive years from a sample of children in China (initial N = 1,156; 581 boys; initial M_age = 9.33 years). Analyses using random intercept cross-lagged panel models revealed that the effects of academic achievement on self-perceptions tended to be more evident in lower grades and the effects of academic self-perceptions on achievement tended to be more evident in higher grades. Latent growth curve analyses showed that the initial level of academic self-perceptions and achievement moderated the growth of each other. The results indicate the contributions between academic self-perceptions and achievement that occur in a progressive cascading manner in Chinese children.     

Reformulating the depression model of learned hopelessness for academic outcomes

This review explores developments in the construct of learned hopelessness, which originated in the clinical literature dealing with depression. In that context, the model developed by Abramson, Metalsky, and Alloy [Abramson, L. Y., Metalsky, G. I., & Alloy, L. B. (1989). Hopelessness depression: A theory-based subtype of depression. Psychological Review, 96, 358–372] has been particularly influential. The purpose of this review is to reformulate this model in the context of academic outcomes and to consider its relevance to learning and achievement. As a means of specifying the variables and paths in the reformulated model of academic learned hopeless, correlates from relevant motivational theories and research, including value-expectancy, self-efficacy, and learning strategies, will be considered.