Detecting Intervention Effects Across Context: An Examination of the Precision of Cluster Randomized Trials

The Institute of Education Sciences has funded more than 100 experiments to evaluate educational interventions in an effort to generate scientific evidence of program effectiveness on which to base education policy and practice. In general, these studies are designed with the goal of having adequate statistical power to detect the average treatment effect. However, the average treatment effect may be less informative if the treatment effects vary substantially from site to site or if the intervention effects differ across context or subpopulations. This article considers the precision of studies to detect different types of treatment effect heterogeneity. Calculations are demonstrated using a set of Institute of Education Sciences funded cluster randomized trials. Strategies for planning future studies with adequate precision for estimating treatment effect heterogeneity are discussed.     

Progress in the past decade: an examination of the precision of cluster randomized trials funded by the U.S. Institute of Education Sciences

ABSTRACT

This article examines the statistical precision of cluster randomized trials (CRTs) funded by the Institute of Education Sciences (IES). Specifically, it compares the total number of clusters randomized and the minimum detectable effect size (MDES) of two sets of studies, those funded in the early years of IES (2002–2004) and those funded in the recent years (2011–2013). Overall, the average precision in terms of MDES of studies in the recent cohort was more than double that of the early cohort (i.e. 0.48 compared to 0.23). The findings suggest a consistent and substantial increase in the precision of CRTs funded by IES in the past decade which is a critical step towards designing studies that have the potential to yield high-quality evidence about the effectiveness of educational interventions.     

The Power of the Test for Treatment Effects in Three-Level Cluster Randomized Designs

Abstract

Experiments that involve nested structures often assign entire groups (such as schools) to treatment conditions. Key aspects of the design of such experiments include knowledge of the intraclass correlation structure and the sample sizes necessary to achieve adequate power to detect the treatment effect. This study provides methods for computing power in three-level cluster randomized balanced designs (with two levels of nesting), where, for example, students are nested within classrooms and classrooms are nested within schools and schools are assigned to treatments. The power computations take into account nesting effects at the second (classroom) and at the third (school) level, sample size effects (e.g., number of schools, classrooms, and individuals), and covariate effects (e.g., pretreatment measures). The methods are applicable to quasi-experimental studies that examine group differences in an outcome.     

A More Powerful Test in Three-Level Cluster Randomized Designs

Abstract

Field experiments that involve nested structures frequently assign treatment conditions to entire groups (such as schools). A key aspect of the design of such experiments includes knowledge of the clustering effects that are often expressed via intraclass correlation. This study provides methods for constructing a more powerful test for the treatment effect in three-level cluster randomized designs with two levels of nesting (at the second and third levels). When the intraclass correlation structure at the second and third level is assumed to be known, the proposed test provides higher estimates of power than those obtained from the typical test based on level-3 unit means, because it preserves the degrees of freedom associated with the number of level-2 and level-1 units. The advantage in power estimates is more pronounced when the number of level-3 units (e.g., schools) is small and the samples are homogeneous (e.g., low-achieving schools).     

The Power of the Test for Treatment Effects in Three-Level Block Randomized Designs

Abstract

Experiments that involve nested structures may assign treatment conditions either to subgroups (such as classrooms) or individuals within subgroups (such as students). The design of such experiments requires knowledge of the intraclass correlation structure to compute the sample sizes necessary to achieve adequate power to detect the treatment effect. This study provides methods for computing power in three-level block randomized balanced designs (with two levels of nesting) where, for example, students are nested within classrooms and classrooms are nested within schools. The power computations take into account nesting effects at the second (classroom) and at the third (school) level, sample size effects (e.g., number of level-1, level-2, and level-3 units), and covariate effects (e.g., pretreatment measures). The methods are generalizable to quasi-experimental studies that examine group differences on an outcome.     

New Empirical Evidence for the Design of Group Randomized Trials in Education

Abstract

This article provides practical guidance for researchers who are designing studies that randomize groups to measure the impacts of educational interventions. The article (a) provides new empirical information about the values of parameters that influence the precision of impact estimates (intraclass correlations and R ² values) and includes outcomes other than standardized test scores and data with a three-level structure rather than a two-level structure, and (b) discusses the error (both generalizability and estimation error) that exists in estimates of key design parameters and the implications this error has for design decisions. Data for the paper come primarily from two studies: the Chicago Literacy Initiative: Making Better Early Readers Study (CLIMBERS) and the School Breakfast Pilot Project (SBPP). The analysis sample from CLIMBERS comprised 430 four-year-old children from 47 preschool classrooms in 23 Chicago public schools. The analysis sample from the SBPP study comprised 1,151 third graders from 233 classrooms in 111 schools from 6 school districts. Student achievement data from the Reading First Impact Study is also used to supplement the discussion.     

Standardized Effect Size Measures for Mediation Analysis in Cluster-Randomized Trials

This article presents 3 standardized effect size measures to use when sharing results of an analysis of mediation of treatment effects for cluster-randomized trials. The authors discuss 3 examples of mediation analysis (upper-level mediation, cross-level mediation, and cross-level mediation with a contextual effect) with demonstration of the calculation and interpretation of the effect size measures using a simulated dataset and an empirical dataset from a cluster-randomized trial of peer tutoring. SAS syntax is provided for parametric percentile bootstrapped confidence intervals of the effect sizes. The use of any of the 3 standardized effect size measures depends on the nature of the inference the researcher wishes to make within a single site, across the broad population, or at the site level.     

Power Computations for Polynomial Change Models in Block-Randomized Designs

Education experiments frequently assign students to treatment or control conditions within schools. Longitudinal components added in these studies (e.g., students followed over time) allow researchers to assess treatment effects in average rates of change (e.g., linear or quadratic). We provide methods for a priori power analysis in three-level polynomial change models for block-randomized designs. We discuss unconditional models and models with covariates at the second and third level. We illustrate how power is influenced by the number of measurement occasions, the sample sizes at the second and third levels, and the covariates at the second and third levels.     

具有遗传印记效应传递/不平衡检验(TDT)的功效(英文)

根据Knapp在 1999年提出的求TDT渐近功效的方法 ,应用标准的统计大样本理论 ,从两个方面对Knapp的方法进行了推广 :(1)在疾病的遗传中考查了遗传印记的影响 ;(2 )在求概率时考查了标记位点与候选疾病位点之间的位置关系 .     

Accounting for the Relationship Between Initial Status and Growth in Regression Models

Educational analysts studying achievement and other educational outcomes frequently encounter an association between initial status and growth, which has important implications for the analysis of covariate effects, including group differences in growth. As explicated by Allison (1990 Allison, P. D. (1990). Change scores as dependent variables in regression analyses. Sociological Methodology, 20, 93–114.[Crossref] , [Google Scholar]), where only two time points of data are available, identifying a preferred model can be difficult or impossible. In this paper we extend Allison's inquiry by considering multiple sources of the association between initial status and growth simultaneously, including measurement error but also intrinsic associations between initial status and growth. We illustrate the potential trade-offs between the change-score model specifications (models without a control for initial status) and regressor-variable specifications (with a control for initial status) using simulated data.     

A Comparison of Method Effects in Two Confirmatory Factor Models for Structurally Different Methods

It is well known that measurement error in observable variables induces bias in estimates in standard regression analysis and that structural equation models are a typical solution to this problem. Often, multiple indicator equations are subsumed as part of the structural equation model, allowing for consistent estimation of the relevant regression parameters. In many instances, however, embedding the measurement model into structural equation models is not possible because the model would not be identified. To correct for measurement error one has no other recourse than to provide the exact values of the variances of the measurement error terms of the model, although in practice such variances cannot be ascertained exactly, but only estimated from an independent study. The usual approach so far has been to treat the estimated values of error variances as if they were known exact population values in the subsequent structural equation modeling (SEM) analysis. In this article we show that fixing measurement error variance estimates as if they were true values can make the reported standard errors of the structural parameters of the model smaller than they should be. Inferences about the parameters of interest will be incorrect if the estimated nature of the variances is not taken into account. For general SEM, we derive an explicit expression that provides the terms to be added to the standard errors provided by the standard SEM software that treats the estimated variances as exact population values. Interestingly, we find there is a differential impact of the corrections to be added to the standard errors depending on which parameter of the model is estimated. The theoretical results are illustrated with simulations and also with empirical data on a typical SEM model.