A Comparison of Propensity Score Weighting Methods for Evaluating the Effects of Programs With Multiple Versions

This Monte Carlo simulation study compares methods to estimate the effects of programs with multiple versions when assignment of individuals to program version is not random. These methods use generalized propensity scores, which are predicted probabilities of receiving a particular level of the treatment conditional on covariates, to remove selection bias. The results indicate that inverse probability of treatment weighting (IPTW) removes the most bias, followed by optimal full matching (OFM), and marginal mean weighting through stratification (MMWTS). The study also compared standard error estimation with Taylor series linearization, bootstrapping and the jackknife across propensity score methods. With IPTW, these standard error estimation methods performed adequately, but standard errors estimates were biased in most conditions with OFM and MMWTS.     

Bias Reduction in Quasi-Experiments With Little Selection Theory but Many Covariates

Abstract: In observational studies, selection bias will be completely removed only if the selection mechanism is ignorable, namely, all confounders of treatment selection and potential outcomes are reliably measured. Ideally, well-grounded substantive theories about the selection process and outcome-generating model are used to generate the sample of covariates. However, covariate selection is more heuristic in actual practice. Using two empirical data sets in a simulation study, we investigate four research questions about bias reduction when the selection mechanism is not known but many covariates are measured: (1) How important is the conceptual heterogeneity of the covariate domains in the data set? (2) How important is the number of covariates assessing each domain? (3) What are the joint effects of this conceptual heterogeneity and of the number of covariates per domain? (4) What happens to bias reduction when the set of covariates is deliberately impoverished by removing the covariates most responsible for selection bias, thus ensuring a slightly smaller but still heterogeneous set of covariates? The results indicate: (1) increasingly more bias is reduced as the number of covariate domains and the number of covariates per domain increase, though the rate of bias reduction is diminishing in each case; (2) sampling covariates from multiple heterogeneous covariate domains is more important than choosing many measures from fewer domains; (3) the most heterogeneous set of covariate domains removes almost all of the selection bias when at least five covariates are assessed in each domain; and (4) omitting the most crucial covariates generally replicates the pattern of results due to the number of domains and the number of covariates per domain, but the amount of bias reduction is less than when all variables are included and will surely not satisfy all consumers of causal research.     

A Comparison of Bias Reduction Methods: Clustering versus Propensity Score Subclassification and Weighting

Propensity score (PS) adjustments have become popular methods used to improve estimates of treatment effects in quasi-experiments. Although researchers continue to develop PS methods, other procedures can also be effective in reducing selection bias. One of these uses clustering to create balanced groups. However, the success of this new method depends on its efficacy compared to that of the existing methods. Therefore, this comparative study used experimental and nonexperimental data to examine bias reduction, case retention, and covariate balance in the clustering method, PS subclassification, and PS weighting. In general, results suggest that the cluster-based methods reduced at least as much bias as the PS methods. Under certain conditions, the PS methods reduced more bias than the cluster-based method, and under other conditions the cluster-based methods were more advantageous. Although all methods were equally effective in retaining cases and balancing covariates, other data-specific conditions may likely favor the use of a cluster-based approach.     

Detecting selection bias,using propensity score matching,and estimating treatment effects: an application to the private returns to a master’s degree

Most research in the area of higher education is plagued by the problem of endogeneity or self-selection bias. Unlike ordinary least squares (OLS) regression, propensity score matching addresses the issue of self-selection bias and allows for a decomposition of treatment effects on outcomes. Using panel data from a national survey of bachelor’s degree recipients, this approach is illustrated via an analysis of the effect of receiving a master’s degree, in various program areas, on wage earning outcomes. The results of this study reveal that substantial self-selection bias is undetected when using OLS regression techniques. This article also shows that, unlike OLS regression, propensity score matching allows for estimates of the average treatment effect, average treatment on the treated effect, and the average treatment on the untreated effect on student outcomes such as wage earnings.     

Time spent on the Internet by multicultural adolescents in Korea

This study analysed The 8th Online Survey of Adolescent Health and Behaviour in 2012, which is a government-approved statistical survey of Internet use and patterns in Korea. We conducted a propensity score matching (PSM) to control for economic status differences between monocultural and multicultural families and an ANOVA to estimate the effects of multicultural family status, adolescents’ gender, school grades, locality, and parents’ academic background on adolescents’ Internet use (for studying and entertainment). The study revealed the following: with regard to Internet use patterns, adolescents from both monocultural and multicultural families selected gaming as their first choice, followed by studying and data searching. Furthermore, main and interaction effects of independent variables on adolescents’ time spent on the Internet for studying were not significant. However, the interaction effects of multicultural family status and mothers’ academic background on adolescents’ time spent on the Internet for entertainment were statistically significant. Specifically, adolescents’ time spent on the Internet for entertainment was higher in multicultural families with mothers whose education level was lower than middle school. Finally, we suggested that the government should provide full support to less educated mothers in multicultural families for effectively and constructively controlling their children’s Internet use and teaching it to them.     

Is the cure worse than the disease? A longitudinal study on the effect of grade retention in secondary education on achievement and academic self-concept

Holding students back when they do not meet a specific attainment level is common practice in a lot of countries. However, this practice is not without controversy and recent studies point at the negative effects of grade retention, especially in the long-term. The majority of these studies focused on grade retention in primary education. In our study, we focused on the effect of grade retention in Grade 8 on language achievement and academic self-concept. We matched students who were and were not retained based on their propensity to be retained and compared both groups using a growth curve analysis. The basic treatment “grade retention vs. promotion” was extended with the certificate these students received at the end of Grade 8. With this growth curve analysis, we were able to draw conclusions on the effect in the short-term, as well as in the long-term. In the short-term (i.e. the year of retention), it seems that grade retention had no negative effect on language achievement in the short-term, and even a positive effect on academic self-concept in the year of retention. The effects became more negative when we considered effects in the long-term. Especially for language achievement, we found a strong decline in the achievement of grade retainees. We found no negative effect of grade retention on academic self-concept. We can conclude that grade retention has a negative effect on the achievement of retained students in the long run, but has no negative effect on academic self-concept. When we take into account the certificate they received, it seems that following the teacher’s advice to change track is a better decision than repeating the grade in the same track.     

On Rewarding and its Educational Significance

Propensity score matching (PSM) has become a popular approach for research studies when randomization is infeasible. However, there are significant differences in the effectiveness of selection bias reduction among the existing PSM methods and, therefore, it is challenging for researchers to select an appropriate matching method. This current study compares four commonly used PSM methods for reducing selection bias on observational data from which the treatment effects are intended to be assessed. The selection bias, standardized bias and percent bias reduction are evaluated for each of the PSM methods using empirical data drawn from the national Education Longitudinal Study of 2002. The results of the current study provide empirical evidence and helpful information for researchers to select effective PSM methods for their research studies.     

An Evaluation of Latent Growth Models for Propensity Score Matched Groups

Because random assignment is not possible in observational studies, estimates of treatment effects might be biased due to selection on observable and unobservable variables. To strengthen causal inference in longitudinal observational studies of multiple treatments, we present 4 latent growth models for propensity score matched groups, and evaluate their performance with a Monte Carlo simulation study. We found that the 4 models performed similarly with respect to model fit, bias of parameter estimates, Type I error, and power to test the treatment effect. To demonstrate a multigroup latent growth model with dummy treatment indicators, we estimated the effect of students changing schools during elementary school years on their reading and mathematics achievement, using data from the Early Childhood Longitudinal Study Kindergarten Cohort.     

The Model-Size Effect on Traditional and Modified Tests of Covariance Structures

The integration of modern methods for causal inference with latent class analysis (LCA) allows social, behavioral, and health researchers to address important questions about the determinants of latent class membership. In this article, 2 propensity score techniques, matching and inverse propensity weighting, are demonstrated for conducting causal inference in LCA. The different causal questions that can be addressed with these techniques are carefully delineated. An empirical analysis based on data from the National Longitudinal Survey of Youth 1979 is presented, where college enrollment is examined as the exposure (i.e., treatment) variable and its causal effect on adult substance use latent class membership is estimated. A step-by-step procedure for conducting causal inference in LCA, including multiple imputation of missing data on the confounders, exposure variable, and multivariate outcome, is included. Sample syntax for carrying out the analysis using SAS and R is given in an appendix.     

Recovering Predictor–Criterion Relations Using Covariate-Informed Factor Score Estimates

Although it is currently best practice to directly model latent factors whenever feasible, there remain many situations in which this approach is not tractable. Recent advances in covariate-informed factor score estimation can be used to provide manifest scores that are used in second-stage analysis, but these are currently understudied. Here we extend our prior work on factor score recovery to examine the use of factor score estimates as predictors both in the presence and absence of the same covariates that were used in score estimation. Results show that whereas the relation between the factor score estimates and the criterion are typically well recovered, substantial bias and increased variability is evident in the covariate effects themselves. Importantly, using covariate-informed factor score estimates substantially, and often wholly, mitigates these biases. We conclude with implications for future research and recommendations for the use of factor score estimates in practice.     

A Comparison of Three-Step Approaches for Auxiliary Variables in Latent Class and Latent Profile Analysis

In this article, 3-step methods to include predictors and distal outcomes in commonly used mixture models are evaluated. Two Monte Carlo simulation studies were conducted to compare the pseudo class (PC), Vermunt’s (2010), and the Lanza, Tan, and Bray (LTB) 3-step approaches with respect to bias of parameter estimates in latent class analysis (LCA) and latent profile analysis (LPA) models with auxiliary variables. For coefficients of predictors of class membership, results indicated that Vermunt’s method yielded more accurate estimates for LCA and LPA compared to the PC method. With distal outcomes of latent classes and latent profiles, the LTB method produced the lowest relative bias of coefficient estimates and Type I error rates close to nominal levels.     

Assessing the effectiveness of a hybrid-flipped model of learning on fluid mechanics instruction: overall course performance,homework, and far- and near-transfer of learning

To examine the impact of a hybrid-flipped model utilising active learning techniques, the researchers inverted one section of an undergraduate fluid mechanics course, reduced seat time, and engaged in active learning sessions in the classroom. We compared this model to the traditional section on four performance measures. We employed a propensity score method entailing a two-stage regression analysis that considered eight covariates to address the potential bias of treatment selection. First, we estimated the probability score based on the eight covariates, and second, we used the inverse of the probability score as a regression weight on the performance of learners who did not select into the hybrid course. Results suggest that enrolment in the hybrid-flipped section had a marginally significant negative impact on the total course score and a significant negative impact on homework performance, possibly because of poor video usage by the hybrid-flipped learners. Suggested considerations are also discussed.     

Preschool Teachers’ Professional Background,Process Quality,and Job Attitudes: A Person-Centered Approach

Research Findings: This exploratory study identified preschool teacher quality profiles in early childhood education settings using 9 indicators across teachers’ professional background, observed process quality, and job attitudes toward teaching (e.g., job-related stress, satisfaction, and intention to leave the job). The sample consisted of 96 teachers from 48 child care programs in a midwestern U.S. state. A latent profile analysis revealed 3 profiles: (a) less experienced, lower quality, and more positive attitudes; (b) less experienced, average quality, and less positive attitudes; and (c) more experienced, better quality, and mixed attitudes. Multiple program- and teacher-level covariates were incorporated into the latent profile model to examine the associations between program and teacher characteristics and profile membership. Practice or Policy: Results of this study provide important information for use in early child care interventions and suggest a new approach toward more integrated professional development programs that cover both teachers’ practices and job attitudes. This study also suggests the need for new coaching systems that can be individualized based on each teacher’s strengths and challenges.     

Adding up or Down: A Discussion

ABSTRACT

Increased access to algebra was a focal point of the National Mathematics Advisory Panel's 2008 report on improving mathematics learning in the United States. Past research found positive effects for early access to algebra, but the focus on average effects may mask important variation across student subgroups. The author addresses whether these positive effects hold up when the analysis is expanded to examine effect heterogeneity. Using a nationally representative sample of eighth-grade students in 1988, the author examined sensitivity of findings to methods for selection bias adjustment, heterogeneity across the propensity to take algebra in Grade 8, and across schools. The findings support past research regarding positive benefits to Grade 8 algebra and are consistent with policies that increase access to algebra in middle school.     

The effects of job-related training on the sense of control of male and female workers in Russia

ABSTRACT

Using data from the Russian Longitudinal Monitoring Study – Higher School of Economics (RLMS), we estimate the relationship between the sense of control, measured as the belief that one has control over one’s important future life circumstances and job-related training for women and men in a transitional context. We test the theory of alternative resources and the critical approaches in the analysis of the role of gender in individual outcomes from training. We show that while job-related training is associated with higher sense of control (measured using Pearlin Mastery Scale), its effect varies by gender and therefore, its absolute value is limited. We conclude that job-related training exacerbates the existing differences in the sense of control between women and men in Russia, which can potentially have prolonged, negative effects on the wider outcomes of women in the labour market.     

Latent Class Analysis in Higher Education: An Illustrative Example of Pluralistic Orientation

Although used frequently in related fields such as K-12 education research, educational psychology, sociology, and social survey research, latent class analysis (LCA) has been infrequently used in higher education. This article provides higher education researchers with a pedagogical application of LCA to classify entering freshmen based on their pluralistic orientation. This study utilized data on entering freshmen at a racially diverse institution on the West coast. LCA was used to estimate latent profile probabilities, classify freshmen into latent classes, and relate latent class probabilities to covariates. The findings indicated that a four-class model was the best fitting model: high pluralistic orientation; high-disposition, low-skill; low-disposition, high-skill; and low pluralistic orientation. Similar to previous research, the findings indicated that the probability of being classified into one group versus the other was dependent upon a student’s race/ethnicity and intended major. This approach can aid college administrators in their program planning and targeted interventions around issues of diversity.     

Estimating the “Impact” of Out‐of‐Home Placement on Child Well‐Being: Approaching the Problem of Selection Bias

This study used data on 2,453 children aged 4–17 from the National Survey of Child and Adolescent Well‐Being and 5 analytic methods that adjust for selection factors to estimate the impact of out‐of‐home placement on children’s cognitive skills and behavior problems. Methods included ordinary least squares (OLS) regressions and residualized change, simple change, difference‐in‐difference, and fixed effects models. Models were estimated using the full sample and a matched sample generated by propensity scoring. Although results from the unmatched OLS and residualized change models suggested that out‐of‐home placement is associated with increased child behavior problems, estimates from models that more rigorously adjust for selection bias indicated that placement has little effect on children’s cognitive skills or behavior problems.     

Professional learning communities among vocational school teachers: Profiles and relations with instructional quality

Although improving teachers' classroom strategies presents the primary goal of PLCs, empirical evidence is still scarce, mainly derived from case-study material and confined to (pre-) K12 settings. We use profiling techniques and a comparative design to examine (a) distinct configurations in which PLCs occur within vocational school departments, and (b) their relations to instructional quality. Multilevel latent profile analysis, based on teacher assessments of core PLC dimensions, reveals three configurations. Multilevel multiple group analysis of students’ instructional ratings shows that teachers from Advanced PLC departments create more authentic, application-oriented learning environments than teachers from other departments.     

Explaining Variability in Response Style Traits: A Covariate-Adjusted IRTree

Contamination of responses due to extreme and midpoint response style can confound the interpretation of scores, threatening the validity of inferences made from survey responses. This study incorporated person-level covariates in the multidimensional item response tree model to explain heterogeneity in response style. We include an empirical example and two simulation studies to support the use and interpretation of the model: parameter recovery using Markov chain Monte Carlo (MCMC) estimation and performance of the model under conditions with and without response styles present. Item intercepts mean bias and root mean square error were small at all sample sizes. Item discrimination mean bias and root mean square error were also small but tended to be smaller when covariates were unrelated to, or had a weak relationship with, the latent traits. Item and regression parameters are estimated with sufficient accuracy when sample sizes are greater than approximately 1,000 and MCMC estimation with the Gibbs sampler is used. The empirical example uses the National Longitudinal Study of Adolescent to Adult Health’s sexual knowledge scale. Meaningful predictors associated with high levels of extreme response latent trait included being non-White, being male, and having high levels of parental support and relationships. Meaningful predictors associated with high levels of the midpoint response latent trait included having low levels of parental support and relationships. Item-level covariates indicate the response style pseudo-items were less easy to endorse for self-oriented items, whereas the trait of interest pseudo-items were easier to endorse for self-oriented items.     

Bias hypotheses under scrutiny: investigating the validity of student assessment of university teaching by means of external observer ratings

To advance the discussion on the validity of student evaluations of university teaching, student ratings of two teaching dimensions – student involvement and rapport – were compared with corresponding observer ratings. Seven potential bias variables were tested with regard to their impact on the students’ teaching assessment: three teacher characteristics (first impression, enthusiasm, humour) and four student characteristics (prior interest, expected grades, study experience, class attendance). Bias was defined as an impediment of the students’ assessment of teaching on course level. By means of bivariate correlations with course averages and two-level latent moderated structural equations, data of 1,716 students in 80 courses were analysed. Results showed that all three teacher characteristics were genuinely connected to rapport, and even explained variance of the student-rated variable when controlling for observer-rated rapport. The assessment of student involvement was not modified by the teacher characteristics except for teacher enthusiasm, which affected the student evaluation when controlling for observed involvement and, moreover, moderated the relation between the observed and the student-rated variable. For the examined student characteristics, no biasing effects were found – neither on rapport nor on student involvement.