Multiple-Group Factor Analysis Alignment

This article presents a new method for multiple-group confirmatory factor analysis (CFA), referred to as the alignment method. The alignment method can be used to estimate group-specific factor means and variances without requiring exact measurement invariance. A strength of the method is the ability to conveniently estimate models for many groups. The method is a valuable alternative to the currently used multiple-group CFA methods for studying measurement invariance that require multiple manual model adjustments guided by modification indexes. Multiple-group CFA is not practical with many groups due to poor model fit of the scalar model and too many large modification indexes. In contrast, the alignment method is based on the configural model and essentially automates and greatly simplifies measurement invariance analysis. The method also provides a detailed account of parameter invariance for every model parameter in every group.     

Multigroup Confirmatory Factor Analysis: Locating the Invariant Referent Sets

Multigroup confirmatory factor analysis (MCFA) is a popular method for the examination of measurement invariance and specifically, factor invariance. Recent research has begun to focus on using MCFA to detect invariance for test items. MCFA requires certain parameters (e.g., factor loadings) to be constrained for model identification, which are assumed to be invariant across groups, and act as referent variables. When this invariance assumption is violated, location of the parameters that actually differ across groups becomes difficult. The factor ratio test and the stepwise partitioning procedure in combination have been suggested as methods to locate invariant referents, and appear to perform favorably with real data examples. However, the procedures have not been evaluated through simulations where the extent and magnitude of a lack of invariance is known. This simulation study examines these methods in terms of accuracy (i.e., true positive and false positive rates) of identifying invariant referent variables.     

HLM Behind the Curtain: Unveiling Decisions Behind the Use and Interpretation of HLM in Higher Education Research

Hierarchical linear modeling (HLM) has become increasingly popular in the higher education literature, but there is significant variability in the current approaches to the conducting and reporting of HLM. The field currently lacks a general consensus around important issues such as the number of levels of analysis that are important to include and how much variance should be accounted for at each level in order for the HLM analysis to have practical significance (Dedrick et al., Rev Educ Res 79:69–102, 2009). The purpose of this research is to explore the use of a 3-level HLM model, appropriate contextualizing of results of HLM, and the interpretation of HLM results that resonates with practice. We used an example of a 3-level model from the National Study of Living Learning Programs to highlight the practical issues that arise in the interpretation of HLM within a higher education context.     

Students' Behavioral and Emotional Participation in Academic Activities in the Mathematics Classroom: A Multilevel Confirmatory Factor Analysis

In this research, the authors raised the issue that prior studies had failed to address the nested structure of data in examining the construct validity of an instrument measuring students' behavioral and emotional participation in academic activities in the classroom. To address this question, the authors illustrated the utility of the multilevel confirmatory factor analysis (MCFA) approach to reexamine the construct validity of this instrument. The sample consisted of 2,041 students in 5th grade from 67 classes in Hong Kong. First, the results justified the requirement of MCFA and indicated that the 4-factor model tested with MCFA provided better fit to the data than that tested with a single-level confirmatory factor analysis (CFA). Second, the study also provided adequate support for a multilevel second-order two-factor model that distinguished engagement from disaffection. Third, the factor structure was invariant across the student level and the classroom level for both the 4-factor model and the second-order two-factor model. Fourth, the results highlighted the presence of ambiguity in differentiating between the dimensions at the classroom level and supported the unidimensionality of the classroom-level construct. Fifth, student engagement was significantly and positively correlated with mathematics test scores, teachers' classroom-management practices, teacher support, and student order in the classroom. Finally, the authors discuss the implications of the study and its limitations and offer suggestions for model selection and explorations for future research.     

Assessing the Organizational Social Context (OSC) of child welfare systems: Implications for research and practice

ObjectiveThe study: (1) provides the first assessment of the a priori measurement model and psychometric properties of the Organizational Social Context (OSC) measurement system in a US nationwide probability sample of child welfare systems; (2) illustrates the use of the OSC in constructing norm-based organizational culture and climate profiles for child welfare systems; and (3) estimates the association of child welfare system-level organizational culture and climate profiles with individual caseworker-level job satisfaction and organizational commitment.MethodsThe study applies confirmatory factor analysis (CFA) and hierarchical linear models (HLM) analysis to a US nationwide sample of 1,740 caseworkers from 81 child welfare systems participating in the second National Survey of Child and Adolescent Wellbeing (NSCAW II). The participating child welfare systems were selected using a national probability procedure reflecting the number of children served by child welfare systems nationwide.ResultsThe a priori OSC measurement model is confirmed in this nationwide sample of child welfare systems. In addition, caseworker responses to the OSC scales generate acceptable to high scale reliabilities, moderate to high within-system agreement, and significant between-system differences. Caseworkers in the child welfare systems with the best organizational culture and climate profiles report higher levels of job satisfaction and organizational commitment. Organizational climates characterized by high engagement and functionality, and organizational cultures characterized by low rigidity are associated with the most positive work attitudes.ConclusionsThe OSC is the first valid and reliable measure of organizational culture and climate with US national norms for child welfare systems. The OSC provides a useful measure of Organizational Social Context for child welfare service improvement and implementation research efforts which include a focus on child welfare system culture and climate.     

A Simulation Study of Polychoric Instrumental Variable Estimation in Structural Equation Models

Data collected from questionnaires are often in ordinal scale. Unweighted least squares (ULS), diagonally weighted least squares (DWLS) and normal-theory maximum likelihood (ML) are commonly used methods to fit structural equation models. Consistency of these estimators demands no structural misspecification. In this article, we conduct a simulation study to compare the equation-by-equation polychoric instrumental variable (PIV) estimation with ULS, DWLS, and ML. Accuracy of PIV for the correctly specified model and robustness of PIV for misspecified models are investigated through a confirmatory factor analysis (CFA) model and a structural equation model with ordinal indicators. The effects of sample size and nonnormality of the underlying continuous variables are also examined. The simulation results show that PIV produces robust factor loading estimates in the CFA model and in structural equation models. PIV also produces robust path coefficient estimates in the model where valid instruments are used. However, robustness highly depends on the validity of instruments.     

Book Review of Basics of Structural Equation Modeling,by Geoffrey M. Maruyama

Confirmatory factor analysis (CFA) is a statistical procedure frequently used to test the fit of data to measurement models. Published CFA studies typically report factor pattern coefficients. Few reports, however, also present factor structure coefficients, which can be essential for the accurate interpretation of CFA results. The interpretation errors that can arise when CFA results are interpreted without considering structure coefficients are described, and some examples from current literature illustrating these errors are also presented.     

Testing the Intervention Effect in Single-Case Experiments: A Monte Carlo Simulation Study

This article reports on a Monte Carlo simulation study, evaluating two approaches for testing the intervention effect in replicated randomized AB designs: two-level hierarchical linear modeling (HLM) and using the additive method to combine randomization test p values (RTcombiP). Four factors were manipulated: mean intervention effect, number of cases included in a study, number of measurement occasions for each case, and between-case variance. Under the simulated conditions, Type I error rate was under control at the nominal 5% level for both HLM and RTcombiP. Furthermore, for both procedures, a larger number of combined cases resulted in higher statistical power, with many realistic conditions reaching statistical power of 80% or higher. Smaller values for the between-case variance resulted in higher power for HLM. A larger number of data points resulted in higher power for RTcombiP.     

Neglect the Structure of Multitrait-Multimethod Data at Your Peril: Implications for Associations With External Variables

In 1959, Campbell and Fiske introduced the use of multitrait–multimethod (MTMM) matrices in psychology, and for the past 4 decades confirmatory factor analysis (CFA) has commonly been used to analyze MTMM data. However, researchers do not always fit CFA models when MTMM data are available; when CFA modeling is used, multiple models are available that have attendant strengths and weaknesses. In this article, we used a Monte Carlo simulation to investigate the drawbacks of either using CFA models that fail to match the data-generating model or completely ignore the MTMM structure of data when the research goal is to uncover associations between trait constructs and external variables. We then used data from the National Institute of Child Health and Human Development Study of Early Child Care and Youth Development to illustrate the substantive implications of fitting models that partially or completely ignore MTMM data structures. Results from analyses of both simulated and empirical data show noticeable biases when the MTMM data structure is partially or completely neglected.     

Examining factorial validity and measurement invariance of the Student-Teacher Relationship Scale

The purposes of this study were to (a) test the hypothesized factor structure of the Student-Teacher Relationship Scale (STRS; Pianta, 2001) for 308 African American (AA) and European American (EA) children using confirmatory factor analysis (CFA) and (b) examine the measurement invariance of the factor structure across AA and EA children. CFA of the hypothesized three-factor model with correlated latent factors did not yield an optimal model fit. Parameter estimates obtained from CFA identified items with low factor loadings and R² values, suggesting that content revision is required for those items on the STRS. Deletion of two items from the scale yielded a good model fit, suggesting that the remaining 26 items reliably and validly measure the constructs for the whole sample. Tests for configural invariance, however, revealed that the underlying constructs may differ for AA and EA groups. Subsequent exploratory factor analyses (EFAs) for AA and EA children were carried out to investigate the comparability of the measurement model of the STRS across the groups. The results of EFAs provided evidence suggesting differential factor models of the STRS across AA and EA groups. This study provides implications for construct validity research and substantive research using the STRS given that the STRS is extensively used in intervention and research in early childhood education.     

Self-regulated learning in Singaporean context: a congeneric approach of confirmatory factor analysis

The motivated strategies for learning questionnaire (MSLQ) is widely used as a self-report instrument to assess students’ motivation and self-regulation. This study utilized the MSLQ Junior High to examine the motivational beliefs and self-regulation of secondary school students (Grades 8 and 9) from Singapore. The instrument was slightly modified and administered to students (N?=?610) in mathematics and science classes. In the first sample, 314 students completed the MSLQ Junior High while a second sample of 296 students completed the revised MSLQ Junior High. Using a congeneric approach of confirmatory factor analysis (CFA), the five-factor measurement model was determined with the first sample. This measurement model was further supported using a second sample and its goodness-of-fit indices were compared with other CFA models. Statistical findings showed that the five-factor structure of the revised MSLQ Junior High had a good model fit. The present study contributes a novel methodological approach by investigating the parsimony confirmatory factor structure of the revised MSLQ Junior High in local academic contexts.     

Assessing Reading Comprehension: The Effects of Text-Based Interest,Gender, and Ability

Previous research has established that interest in a text is related to better comprehension and recall, but the impact of interest on comprehension in testing situations has not been widely investigated. This study used Hierarchical Linear Models (HLM) to examine the relationship between students' interest in reading passages and their performance on reading comprehension test items over those passages. The study involved 19,735 students in Grades 4 through 8. Stimuli consisted of 98 different reading passages. A small but significant relationship between interest and test performance was found, which was stronger for girls and for students of higher ability levels. A second HLM analysis was employed to explore whether certain passage characteristics were associated with higher or lower interest. The implications for test construction are discussed.     

Analyzing Different Types of Moderated Method Effects in Confirmatory Factor Models for Structurally Different Methods

An extension of two confirmatory factor models for multitrait-multimethod measurement designs with structurally different methods to the analysis of latent interaction effects is presented: the nonlinear latent difference (NL-LD) model and the nonlinear correlated trait–correlated method-minus-one (NL-CTC[M – 1]) model. Both models are compared with regard to (a) the psychometric definition of the latent variables, (b) the capabilities of explaining latent method effects, and (c) the analysis of latent interaction effects. Using the latent moderated structural equation approach, we show how moderated method effects can be examined in the NL-CTC(M – 1) model. This fine-grained analysis of method effects is not feasible using the classical NL-LD model. We propose an extended version of the NL-LD model, which recovers the results of the NL-CTC(M – 1) model. The different versions of the nonlinear multimethod models are illustrated using real data from a multirater study. Finally, the advantages and challenges of incorporating latent interaction effects in complex CFA–MTMM models are discussed.     

Measurement invariance across information and communication technology development index and gender: The case of the Pearson Test of English Academic reading

The computerization of reading assessments has presented a set of new challenges to test designers. From the vantage point of measurement invariance, test designers must investigate whether the traditionally recognized causes for violating invariance are still a concern in computer-mediated assessments. In addition, it is necessary to understand the technology-related causes of measurement invariance among test-taking populations. In this study, we used the available data (n = 800) from the previous administrations of the Pearson Test of English Academic (PTE Academic) reading, an international test of English comprising 10 test items, to investigate measurement invariance across gender and the Information and Communication Technology Development index (IDI). We conducted a multi-group confirmatory factor analysis (CFA) to assess invariance at four levels: configural, metric, scalar, and structural. Overall, we were able to confirm structural invariance for the PTE Academic, which is a necessary condition for conducting fair assessments. Implications for computer-based education and the assessment of reading are discussed.     

Exploratory Structural Equation Modeling,Integrating CFA and EFA: Application to Students' Evaluations of University Teaching

This study is a methodological-substantive synergy, demonstrating the power and flexibility of exploratory structural equation modeling (ESEM) methods that integrate confirmatory and exploratory factor analyses (CFA and EFA), as applied to substantively important questions based on multidimentional students' evaluations of university teaching (SETs). For these data, there is a well established ESEM structure but typical CFA models do not fit the data and substantially inflate correlations among the nine SET factors (median rs = .34 for ESEM, .72 for CFA) in a way that undermines discriminant validity and usefulness as diagnostic feedback. A 13-model taxonomy of ESEM measurement invariance is proposed, showing complete invariance (factor loadings, factor correlations, item uniquenesses, item intercepts, latent means) over multiple groups based on the SETs collected in the first and second halves of a 13-year period. Fully latent ESEM growth models that unconfounded measurement error from communality showed almost no linear or quadratic effects over this 13-year period. Latent multiple indicators multiple causes models showed that relations with background variables (workload/difficulty, class size, prior subject interest, expected grades) were small in size and varied systematically for different ESEM SET factors, supporting their discriminant validity and a construct validity interpretation of the relations. A new approach to higher order ESEM was demonstrated, but was not fully appropriate for these data. Based on ESEM methodology, substantively important questions were addressed that could not be appropriately addressed with a traditional CFA approach.     

Development of a survey instrument to measure TEFL academics’ perceptions about,individual and workplace characteristics for conducting research

A 30-item survey was devised to determine Chinese TEFL (Teaching English as a Foreign Language) academics’ potential for conducting research. A five-part Likert scale was used to gather data from 182 academics on four factors: (1) perceptions on teaching–research nexus, (2) personal perspectives for conducting research, (3) predispositions for conducting research and (4) workplace contexts for conducting research. Data were subjected to confirmatory factor analysis (CFA) for structural equation modelling using various fit indices. The independence model was rejected. Accordingly, the CFA model proposed that the four factors covaried and were associated with each indicated item. The hypothesized CFA model demonstrated an acceptable model fit. After statistical analysis, a revised CFA model presented more reliable fit measures, which required a reduction in survey item numbers. This instrument will require further testing but may be used to draw comparisons between past and present data and determine areas for enhancing research productivity.     

Within-year oral reading fluency with CBM: a comparison of models

This study examined the type of growth model that best fit within-year growth in oral reading fluency and between-student differences in growth. Participants were 2,465 students in grades 3–5. Hierarchical linear modeling (HLM) analyses modeled curriculum-based measurement (CBM) oral reading fluency benchmark measures in fall, winter, and spring with grade level and student characteristics (including special education and Limited English Proficiency status) as covariates. Results indicated that a discontinuous growth model fit the data better than a linear growth model, with greater growth in the fall than in the spring. Oral reading fluency growth rates also differed by grade and student characteristics. Implications for school practice and research are discussed.     

Assessing the Sensitivity of Weighted Least Squares Model Fit Indexes to Local Dependence in Item Response Theory Models

Given the relationships of item response theory (IRT) models to confirmatory factor analysis (CFA) models, IRT model misspecifications might be detectable through model fit indexes commonly used in categorical CFA. The purpose of this study is to investigate the sensitivity of weighted least squares with adjusted means and variance (WLSMV)-based root mean square error of approximation, comparative fit index, and Tucker–Lewis Index model fit indexes to IRT models that are misspecified due to local dependence (LD). It was found that WLSMV-based fit indexes have some functional relationships to parameter estimate bias in 2-parameter logistic models caused by violations of LD. Continued exploration into these functional relationships and development of LD-detection methods based on such relationships could hold much promise for providing IRT practitioners with global information on violations of local independence.     

Measuring authoritative teaching

High quality measurements are important to evaluate interventions. The study reports on the development of a measurement to investigate authoritative teaching understood as a two-dimensional construct of warmth and control. Through the application of confirmatory factor analysis (CFA) and structural equation modelling (SEM) the factor structure and measurement invariance is investigated. Generally, results suggest that the two-dimensional model of authoritative teaching has satisfactory psychometric properties for longitudinal measurement invariance, ensuring the measurement of the same concept over time. Different types of missing data in this study are discussed. Also, the relevance of such study for professional development is addressed.     

Investigating the relation of perceived teacher unfairness to science achievement by hierarchical linear modeling in 52 countries and economies

Abstract

This study used the data from the 2015 OECD Programme for International Student Assessment (PISA) to examine the relationship between perceived teacher unfairness and science achievement with a three-level hierarchical linear model (HLM) as the analytic approach. Data of 188,104 students from 4895 schools in 52 countries and economies were used for analysis. After accounting for student gender, student-, school-, and country-level economic, social, and cultural status, and students’ non-cognitive outcomes, the results of HLM analysis showed that perceived teacher unfairness negatively predicted science achievement with a modest effect size. The possible explanations of the results and the practical implication of the findings were discussed.