Mediation Effects In 2-1-1 Multilevel Model: Evaluation Of Alternative Estimation Methods

We compared six common methods in estimating the 2-1-1 (level-2 independent, level-1 mediator, level-1 dependent) multilevel mediation model with a random slope. They were the Bayesian with informative priors, the Bayesian with non-informative priors, the Monte-Carlo, the distribution of the product, the bias-corrected, and the bias-uncorrected parametric percentile residual bootstrap. The Bayesian method with informative priors was superior in relative mean square error (RMSE), power, interval width, and interval imbalance. The prior variance and prior mean were also varied and examined. Decreasing the prior variance increased the power, reduced RMSE and interval width when the prior mean was the true value, but decreasing the prior variance reduced the power when the prior mean was set incorrectly. The influence of misspecification of prior information of the b coefficient on multilevel mediation analysis was greater than that on coefficient a. An illustrate example with the Bayesian multilevel mediation was provided.     

Accuracy of Estimates and Statistical Power for Testing Meditation in Latent Growth Curve Modeling

The latent growth curve modeling (LGCM) approach has been increasingly utilized to investigate longitudinal mediation. However, little is known about the accuracy of the estimates and statistical power when mediation is evaluated in the LGCM framework. A simulation study was conducted to address these issues under various conditions including sample size, effect size of mediated effect, number of measurement occasions, and R ₂ of measured variables. In general, the results showed that relatively large samples were needed to accurately estimate the mediated effects and to have adequate statistical power, when testing mediation in the LGCM framework. Guidelines for designing studies to examine longitudinal mediation and ways to improve the accuracy of the estimates and statistical power were discussed.     

Moderated Mediation Analysis Using Bayesian Methods

Conventionally, moderated mediation analysis is conducted through adding relevant interaction terms into a mediation model of interest. In this study, we illustrate how to conduct moderated mediation analysis by directly modeling the relation between the indirect effect components including a and b and the moderators, to permit easier specification and interpretation of moderated mediation. With this idea, we introduce a general moderated mediation model that can be used to model many different moderated mediation scenarios including the scenarios described in Preacher, Rucker, and Hayes (2007). Then we discuss how to estimate and test the conditional indirect effects and to test whether a mediation effect is moderated using Bayesian approaches. How to implement the estimation in both BUGS and Mplus is also discussed. Performance of Bayesian methods is evaluated and compared to that of frequentist methods including maximum likelihood (ML) with 1st-order and 2nd-order delta method standard errors and mL with bootstrap (percentile or bias-corrected confidence intervals) via a simulation study. The results show that Bayesian methods with diffuse (vague) priors implemented in both BUGS and Mplus yielded unbiased estimates, higher power than the ML methods with delta method standard errors, and the ML method with bootstrap percentile confidence intervals, and comparable power to the ML method with bootstrap bias-corrected confidence intervals. We also illustrate the application of these methods with the real data example used in Preacher et al. (2007). Advantages and limitations of applying Bayesian methods to moderated mediation analysis are also discussed.     

Designing Intervention Studies: Selected Populations,Range Restrictions,and Statistical Power

ABSTRACT

An appropriate estimate of statistical power is critical for the design of intervention studies. Although the inclusion of a pretest covariate in the test of the primary outcome can increase statistical power, samples selected on the basis of pretest performance may demonstrate range restriction on the selection measure and other correlated measures. This can result in attenuated pretest–posttest correlations, reducing the variance explained by the pretest covariate. We investigated the implications of two potential range restriction scenarios: direct truncation on a selection measure and indirect range restriction on correlated measures. Empirical and simulated data indicated that direct range restriction on the pretest covariate greatly reduced statistical power and necessitated sample size increases of 82%–155% (dependent on selection criteria) to achieve equivalent statistical power to parameters with unrestricted samples. However, measures demonstrating indirect range restriction required much smaller sample size increases (32%–71%) under equivalent scenarios. Additional analyses manipulated the correlations between measures and pretest–posttest correlations to guide planning experiments. Results highlight the need to differentiate between selection measures and potential covariates and to investigate range restriction as a factor impacting statistical power.     

The power of statistical tests in science teachnig research

A calculation of the probability of rejecting H₀ when it should be rejected (power) was completed on each of the 66 applicable articles in Volumes 6 and 7 (1969, 1970) of the Journal of Research in Science Teaching. These power calculations utilized the effect size definitions and tables developed by Cohen (1969). The mean power of each article to detect small, medium, and large effect sizes was determined from its major statistical tests. These mean powers were then compiled and analyzed. The powers calculated for the different effect sizes were disturbingly low (small, 0.22; medium, 0.71; large, 0.87) but not generally as low as Cohen (1962) found in an analysis of another behavioral journal. Recommendations for improving confidence in research in science teaching is provided and centers on significant increases in sample sizes and an understanding of power and its relation to a, effect size and sample size.     

A Comparison of the English Vocabulary Used by Children of Non-American Ancestry in Hawaii before they Reach the Age of Seven Years with that of Kindergarten Children in Continental United States

The authors investigated 2 issues concerning the power of latent growth modeling (LGM) in detecting linear growth: the effect of the number of repeated measurements on LGM's power in detecting linear growth and the comparison between LGM and some other approaches in terms of power for detecting linear growth. A Monte Carlo simulation design was used, with 3 crossed factors (growth magnitude, number of repeated measurements, and sample size) and 1,000 replications within each cell condition. The major findings were as follows: For 3 repeated measurements, a substantial proportion of samples failed to converge in structural equation modeling; the number of repeated measurements did not show any effect on the statistical power of LGM in detecting linear growth; and the LGM approach outperformed both the dependent t test and repeated-measures analysis of variance (ANOVA) in terms of statistical power for detecting growth under the conditions of small growth magnitude and small to moderate sample size conditions. The multivariate repeated-measures ANOVA approach consistently underperformed the other tests.     

Optimal Sample Allocation in Group-Randomized Mediation Studies with a Group-Level Mediator

We derive sample-allocation formulas that maximize the power of several mediation tests in two-level–group-randomized studies under a linear cost structure and fixed budget. The results suggest that the optimal individual sample size is typically smaller than that associated with the detection of a main effect and is frequently less than 10 under parameter values commonly seen in the literature. However, the optimal sample allocation can be heavily influenced by the group-to-individual cost ratio, the ratio of the treatment-mediator to mediator-outcome path coefficients, and the outcome variance structure. We illustrate these findings with a hypothetical group-randomized trial examining a school-discipline reform policy. To encourage utilization of the sample allocation formulas we implement them in the R package PowerUpR and powerupr Shiny application.     

Strategies for Efficient Experimental Design in Studies Probing 2-1-1 Mediation

Abstract

When well-implemented, mediation analyses play a critical role in probing theories of action because their results help lay the ground work for the critical development of a treatment and the iterative advancement of theories that are foundational to a discipline. Despite strong interest in designs that incorporate mediation, few studies have developed effective and efficient strategies to plan experiments examining multilevel mediation. We probe several design strategies for cluster-randomized designs and derive sampling plans that maximize power under cost constraints. The results suggest that among the more durable design strategies for mediation is covariance adjustment on variables predictive of the outcome and optimal sample allocation. The statistical power and optimal sample allocation results are implemented in the R package PowerUpR.     

Testing the Intervention Effect in Single-Case Experiments: A Monte Carlo Simulation Study

This article reports on a Monte Carlo simulation study, evaluating two approaches for testing the intervention effect in replicated randomized AB designs: two-level hierarchical linear modeling (HLM) and using the additive method to combine randomization test p values (RTcombiP). Four factors were manipulated: mean intervention effect, number of cases included in a study, number of measurement occasions for each case, and between-case variance. Under the simulated conditions, Type I error rate was under control at the nominal 5% level for both HLM and RTcombiP. Furthermore, for both procedures, a larger number of combined cases resulted in higher statistical power, with many realistic conditions reaching statistical power of 80% or higher. Smaller values for the between-case variance resulted in higher power for HLM. A larger number of data points resulted in higher power for RTcombiP.     

A meta‐analysis of the relationship between student attitudes towards reading and achievement in reading

A meta‐analysis of the relationship between attitudes in reading and achievement in reading was conducted to provide a statistical summary to the observed variability in the magnitude of previously reported effect sizes. A total of 32 studies, with a total sample size of 224,615 were used, and included a total of 118 effect sizes. A multi‐level approach was used in meta‐analysis to determine if variance in the magnitude of effect sizes could be partitioned to study (level 1) and moderator (level 2) levels by using a mixed model approach. Results from the meta‐analysis indicated that the mean strength of the relationship between reading attitudes and achievement is moderate (Z_r=.32), while stronger for students in elementary school (Z_r=.44) when compared with middle school students (Z_r=.24). Findings related to selected moderator variables are discussed, with suggestions for future research.     

Using Excel for White's Test—An Important Technique for Evaluating the Equality of Variance Assumption and Model Specification in a Regression Analysis

There is consensus in the statistical literature that severe departures from its assumptions invalidate the use of regression modeling for purposes of inference. The assumptions of regression modeling are usually evaluated subjectively through visual, graphic displays in a residual analysis but such an approach, taken alone, may be insufficient for assessing the appropriateness of the fitted model. Here, an easy‐to‐use test of the assumption of equal variance (i.e., homoscedasticity) as well as model specification is provided. Given the importance of the equal‐variance assumption (i.e., if uncorrected, severe violations preclude the use of statistical inference and moderate violations result in a loss of statistical power) and given the fact that, if uncorrected, a misspecified or underspecified model could invalidate an entire study, the test developed by Halbert White in 1980 is recommended for supplementing a graphic residual analysis when teaching regression modeling to business students at both the undergraduate and graduate levels. Using this confirmatory approach to supplement a traditional residual analysis has value because students often find that graphic displays are too subjective for determining what constitutes severe from moderate departures from the equal variance assumption or for assessing patterns in plots that might indicate model misspecification or underspecification.     

A study of the power associated with testing factor mean differences under violations of factorial invariance

We examine the power associated with the test of factor mean differences when the assumption of factorial invariance is violated. Utilizing the Wald test for obtaining power, issues of model size, sample size, and total versus partial noninvariance are considered along with variation of actual factor mean differences. Results of a population study show that power is profoundly affected by true factor mean differences but is relatively unaffected by the degree of factor loading noninvariance. Inequality of sample size has a profound effect on power probabilities with power decreasing as sample sizes become increasingly disparate. Sample size variations operate such that power is uniformly lower when the group with the smaller generalized variance is associated with the smaller sample size. An increase in the number of variables yields uniformly larger power probabilities. No substantial differences are found between total and partial noninvariance. Results are related to work in the area of robustness of Hotelling's T ² statistic and discussed in terms of asymptotic covariability of factor means and factor loadings. Implications for practice are considered.     

Parent-Child Relationships from the Child’S Point of View

The authors examined the distributional properties of 3 improvement-over-chance, I, effect sizes each derived from linear and quadratic predictive discriminant analysis and from logistic regression analysis for the 2-group univariate classification. These 3 classification methods (3 levels) were studied under varying levels of data conditions, including population separation (3 levels), variance pattern (3 levels), total sample size (3 levels), and prior probabilities (5 levels). The results indicated that the decision of which effect size to choose is primarily determined by the variance pattern and prior probabilities. Some of the I indices performed well for some small sample cases and quadratic predictive discriminant analysis I tended to work well with extreme variance heterogeneity and differing prior probabilities.     

Two steps forward,one step back: Effect size reporting in gifted education research from 1995–2000

The APA Task Force on Statistical Inference recently recommended reporting effect sizes alongside results of statistical significance tests. The purpose of this article is to investigate effect size usage in gifted education research and to follow up on a similar investigation published by Plucker (1997). A content analysis of effect size reporting was conducted of articles published in the Journal for the Education of the Gifted, Roeper Review, and Gifted Child Quarterly from 1995–2000. Results of the present study were similar to the findings of Plucker (1997): No statistical difference in reporting was found across journals or across years, and a moderate difference was found between effect size reporting in univariate versus multivariate statistics. The benefits to gifted education research of understanding the relationship among sample size, effect size, and statistical power are discussed.     

The effectiveness of intelligent tutoring systems on K-12 students' reading comprehension: A meta-analysis

This meta-analysis examined the effectiveness of improving reading comprehension for students in K-12 classrooms using intelligent tutoring systems (ITSs), a computer-based learning environment that provides customizable and immediate feedback to the learner. Nineteen studies from 13 publications incorporating approximately 10 000 students were included in the final analysis; using robust variance estimation to account for statistical dependencies, the 19 studies yielded 88 effect size estimates. The meta-analysis indicated that the overall random effect size of ITSs on reading comprehension was 0.60 (using a mix of standardized and researcher-designed measures) with a 95% confidence interval 0.36 to 0.85 (p < 0.001). This review confirms previous studies comparing ITSs to human tutoring: ITSs produced a small effect size when compared to human tutoring (0.20, 0.02–0.38, p = 0.036, n = 21). All comparisons to human tutoring used standardized measures. This review also found that ITSs produced a larger effect size on reading comprehension when compared to traditional instruction (0.86) for mixed measures and (0.26) for standardized measures. These findings may be of interest to practitioners and policy makers seeking to improve reading comprehension using consistent and accessible ITSs. Recommendations for researchers include conducting studies to understand the difference between traditional and updated versions of ITSs and employing valid and reliable standardized tests and researcher-designed measures.     

Allocating Sample Sizes to Reduce Budget for Fixed-Effect 2×2 Heterogeneous Analysis of Variance

This article discusses the sample size requirements for the interaction, row, and column effects, respectively, by forming a linear contrast for a 2×2 factorial design for fixed-effects heterogeneous analysis of variance. The proposed method uses the Welch t test and its corresponding degrees of freedom to calculate the final sample size in a 2-step procedure. The simulation results show that the proposed sample size allocation ratio can minimize the sampling cost, while at the same time the designated power is achieved. The article concludes with a discussion to reiterate the importance of sample size planning, especially for testing the iteration effect.     

The Relationship of Hobbies to Personality Characteristics of School Children

The relation between test reliability and statistical power has been a controversial issue, perhaps due in part to a 1975 publication in the Psychological Bulletin by Overall and Woodward, “Unreliability of Difference Scores: A Paradox for the Measurement of Change”, in which they demonstrated that a Student t test based on pretest-posttest differences can attain its greatest power when the difference score reliability is zero. In the present article, the authors attempt to explain this paradox by demonstrating in several ways that power is not a mathematical function of reliability unless either true score variance or error score variance is constant.     

Multimediation Method With Balanced Repeated Replications For Analysis Of Complex Surveys

We propose a method suitable for analysis of cross-sectional studies with complex sampling and continuous variables. The method consists of R + 4 steps, where R denotes the number of replications. In the first R + 1 step, the main and R replicate weights are used (one at a time) to estimate the product of coefficients for all mediation effects using a structural equation model. In step R + 2, the standard errors of these estimates are computed via balanced repeated replications. In step R + 3, the raw p values corresponding to mediation effects are computed based on the generalized Sobel’s tests. In the final step, R + 4, the p values are adjusted for multiplicity and statistical inferences regarding mediation effects are drawn. To illustrate the approach we examined significance of attitudes toward smoking bans as mediators in the association between smoking restrictions at work and nicotine dependence among male daily smokers.     

Statistical Power to Detect the Correct Number of Classes in Latent Profile Analysis

Little research has examined factors influencing statistical power to detect the correct number of latent classes using latent profile analysis (LPA). This simulation study examined power related to interclass distance between latent classes given true number of classes, sample size, and number of indicators. Seven model selection methods were evaluated. None had adequate power to select the correct number of classes with a small (Cohen's d = .2) or medium (d = .5) degree of separation. With a very large degree of separation (d = 1.5), the Lo–Mendell–Rubin test (LMR), adjusted LMR, bootstrap likelihood ratio test, Bayesian Information Criterion (BIC), and sample-size-adjusted BIC were good at selecting the correct number of classes. However, with a large degree of separation (d = .8), power depended on number of indicators and sample size. Akaike's Information Criterion and entropy poorly selected the correct number of classes, regardless of degree of separation, number of indicators, or sample size.