The Power of the Test for Treatment Effects in Three-Level Block Randomized Designs

Abstract

Experiments that involve nested structures may assign treatment conditions either to subgroups (such as classrooms) or individuals within subgroups (such as students). The design of such experiments requires knowledge of the intraclass correlation structure to compute the sample sizes necessary to achieve adequate power to detect the treatment effect. This study provides methods for computing power in three-level block randomized balanced designs (with two levels of nesting) where, for example, students are nested within classrooms and classrooms are nested within schools. The power computations take into account nesting effects at the second (classroom) and at the third (school) level, sample size effects (e.g., number of level-1, level-2, and level-3 units), and covariate effects (e.g., pretreatment measures). The methods are generalizable to quasi-experimental studies that examine group differences on an outcome.     

The Power of the Test for Treatment Effects in Three-Level Cluster Randomized Designs

Abstract

Experiments that involve nested structures often assign entire groups (such as schools) to treatment conditions. Key aspects of the design of such experiments include knowledge of the intraclass correlation structure and the sample sizes necessary to achieve adequate power to detect the treatment effect. This study provides methods for computing power in three-level cluster randomized balanced designs (with two levels of nesting), where, for example, students are nested within classrooms and classrooms are nested within schools and schools are assigned to treatments. The power computations take into account nesting effects at the second (classroom) and at the third (school) level, sample size effects (e.g., number of schools, classrooms, and individuals), and covariate effects (e.g., pretreatment measures). The methods are applicable to quasi-experimental studies that examine group differences in an outcome.     

An Experiment with the Controlled Reader

Abstract

This study was an investigation of items on the Peabody Picture Vocabulary Test (PPVT) to ascertain if verbal responses to items missed indicated that the concept was familiar at the same level of abstraction as the word in the PPVT. One hundred 8-year-old children-25 black boys, 25 black girls, 25 white boys, and 25 white girls-were administered Form A of the PPVT. Eighty-eight children responded verbally to the pictures of the stimulus words missed. Data were analyzed by means of a two-way analysis of variance. A chi square test of significance was used to determine significance level of difference between items for each group. Judges analyzed verbal responses to determine if responses elicited were 1) at the same level of abstraction as the stimulus word, 2) considered to be synonymous to the stimulus word, and 3) indicated the student's understanding of the concept signified by the word. A total of 23 words were identified as being missed disproportionately by one group more than the other. Verbal responses indicated that the concept was familiar for 16 items and unfamiliar for three items. Of the remaining four items, there was indication of differences among the groups.     

Covariate Balance in Bayesian Propensity Score Approaches for Observational Studies

Abstract

Bayesian alternatives to frequentist propensity score approaches have recently been proposed. However, few studies have investigated their covariate balancing properties. This article compares a recently developed two-step Bayesian propensity score approach to the frequentist approach with respect to covariate balance. The effects of different priors on covariate balance are evaluated and the differences between frequentist and Bayesian covariate balance are discussed. Results of the case study reveal that both the Bayesian and frequentist propensity score approaches achieve good covariate balance. The frequentist propensity score approach performs slightly better on covariate balance for stratification and weighting methods, whereas the two-step Bayesian approach offers slightly better covariate balance in the optimal full matching method. Results of a comprehensive simulation study reveal that accuracy and precision of prior information on propensity score model parameters do not greatly influence balance performance. Results of the simulation study also show that overall, the optimal full matching method provides the best covariate balance and treatment effect estimates compared to the stratification and weighting methods. A unique feature of covariate balance within Bayesian propensity score analysis is that we can obtain a distribution of balance indices in addition to the point estimates so that the variation in balance indices can be naturally captured to assist in covariate balance checking.     

Effects of Feedback on Absenteeism in the Junior High School

Abstract

This study assessed the effects of a special feedback program in reducing absenteeism among adolescent junior high school students" Parents of students in the experimental group received followup calls in cases of student absences on a more immediate basis than parents of other absentees. These parents also received positive phone calls and letters when their children improved in attendance. Two comparison groups received no positive feedback, but one was involved in a special social skills training program. Both an analysis of covariance, using absences the prior year as the covariate, and chi square analyses indicated significant differences between the groups. Both the experimental group and social skills group showed significant improvement in relation to the no-treatment control group.     

Under What Circumstances Does External Knowledge About the Correlation Structure Improve Power in Cluster Randomized Designs?

Abstract

Recent publications have drawn attention to the idea of utilizing prior information about the correlation structure to improve statistical power in cluster randomized experiments. Because power in cluster randomized designs is a function of many different parameters, it has been difficult for applied researchers to discern a simple rule explaining when prior correlation information will substantially improve power. This article provides bounds on the maximum possible improvement in power as a function of a single parameter, the number of clusters at the highest level of a multilevel experiment. The maximum improvement in power is less than 0.05 unless the number of clusters at the highest level is less than 20. Thus, the utility of using prior correlation information is limited to experiments with very small cluster-level sample sizes. Situations where small cluster-level sample sizes could still result in experiments with good statistical power are discussed, as is the relative utility of prior information about intracluster correlations as compared with covariate information that can explain cluster level variability in the outcome.     

A study of the power associated with testing factor mean differences under violations of factorial invariance

We examine the power associated with the test of factor mean differences when the assumption of factorial invariance is violated. Utilizing the Wald test for obtaining power, issues of model size, sample size, and total versus partial noninvariance are considered along with variation of actual factor mean differences. Results of a population study show that power is profoundly affected by true factor mean differences but is relatively unaffected by the degree of factor loading noninvariance. Inequality of sample size has a profound effect on power probabilities with power decreasing as sample sizes become increasingly disparate. Sample size variations operate such that power is uniformly lower when the group with the smaller generalized variance is associated with the smaller sample size. An increase in the number of variables yields uniformly larger power probabilities. No substantial differences are found between total and partial noninvariance. Results are related to work in the area of robustness of Hotelling's T ² statistic and discussed in terms of asymptotic covariability of factor means and factor loadings. Implications for practice are considered.     

Designing Intervention Studies: Selected Populations,Range Restrictions,and Statistical Power

ABSTRACT

An appropriate estimate of statistical power is critical for the design of intervention studies. Although the inclusion of a pretest covariate in the test of the primary outcome can increase statistical power, samples selected on the basis of pretest performance may demonstrate range restriction on the selection measure and other correlated measures. This can result in attenuated pretest–posttest correlations, reducing the variance explained by the pretest covariate. We investigated the implications of two potential range restriction scenarios: direct truncation on a selection measure and indirect range restriction on correlated measures. Empirical and simulated data indicated that direct range restriction on the pretest covariate greatly reduced statistical power and necessitated sample size increases of 82%–155% (dependent on selection criteria) to achieve equivalent statistical power to parameters with unrestricted samples. However, measures demonstrating indirect range restriction required much smaller sample size increases (32%–71%) under equivalent scenarios. Additional analyses manipulated the correlations between measures and pretest–posttest correlations to guide planning experiments. Results highlight the need to differentiate between selection measures and potential covariates and to investigate range restriction as a factor impacting statistical power.     

Differences in Mental Ability and Academic Achievement of Two Groups of High School Graduates

Abstract

This study investigated the differences between the means and the standard deviations in mental ability and academic achievement of two groups of high school students who were differentiated on their indicated aspirations for higher education. The subjects were 636 randomly selected 1961 graduates from nine public, comprehensive high schools in Portland, Oregon.

The hypothesis and sub-hypotheses were stated in the null form. Tests of significance at the .01 level of confidence were determined by utilizing the t tests.

The findings indicated that there were high significant differences between the means of the two groups for all characteristics except the GPA's, and no significant differences between the variability of the scores except in the Stanford Reading Scores.     

Medical leadership competences among medical students at Hashemite University in Jordan

ABSTRACT

The purpose of this study is to identify Medical Leadership Competences among medical students at Hashemite University, and analyze the significant differences in the university participants based on their academic level, gender and GPA. Data collection randomly selected from medical students, the response rate for were (260) students. A 40 items survey covered the five areas of Medical Leadership Competences were used. The responses indicated that the most leadership competences for medical students ascending by means were: ‘Demonstrating personal qualities’, ‘Working with others’, ‘Managing services’, ’Setting direction’, an ‘Improving services’, all of these competencies were in high level. There were no significant differences at the 0.05 level among medical leadership competences of medical students’ at Hashemite University related to their academic year, gender, and GPA.     

Hispanic-Serving Community Colleges

The article examines student tuition, ad valorem property taxes, and state appropriations utilizing a revenue-per-contact-hour model to identify disparities in the Texas' community college funding mechanism. Methodology is presented to identify differences between and among Caucasian-serving, African-American-serving, Hispanic-serving, and other public community colleges. Statistical Packages for the Social Sciences (SPSS) was utilized to conduct multiple-factor analysis of variance (ANOVA) on the data set. The statistical testing utilized a significance level of 0.05. Posthoc tests were performed where necessary.     

Evaluating Intercept-Slope Interactions in Latent Growth Modeling

The effects of misspecifying intercept-covariate interactions in a 4 time-point latent growth model were the focus of this investigation. The investigation was motivated by school growth studies in which students' entry-level skills may affect their rate of growth. We studied the latent interaction of intercept and a covariate in predicting growth with respect to 3 factors: sample size (100, 200, and 500), 4 levels of magnitude of interaction effect, and 3 correlation values between intercept and covariate (.3, .5, and .7). Correctly specified models were examined to determine power and Type I error rates, and misspecified models were examined to evaluate the effects on power, parameter estimation, bias, and fit indexes. Results showed that, under correctly specified models, power increased as expected with increasing sample size, larger magnitude of interaction, and larger intercept-covariate correlation. Under misspecification, omitting a non-null interaction results in significant change in the estimation of the direct effects of both covariate and intercept in both magnitude and direction, with results dependent on sign of parameter values for main effects and interaction. Including a spurious interaction does not affect estimation of direct effects of intercept and covariate but does increase standard errors. The primary problem in ignoring a non-null interaction lies in misinterpretation of the model, as interactions yield important insights into the nature of the processes being studied.     

Two steps forward,one step back: Effect size reporting in gifted education research from 1995–2000

The APA Task Force on Statistical Inference recently recommended reporting effect sizes alongside results of statistical significance tests. The purpose of this article is to investigate effect size usage in gifted education research and to follow up on a similar investigation published by Plucker (1997). A content analysis of effect size reporting was conducted of articles published in the Journal for the Education of the Gifted, Roeper Review, and Gifted Child Quarterly from 1995–2000. Results of the present study were similar to the findings of Plucker (1997): No statistical difference in reporting was found across journals or across years, and a moderate difference was found between effect size reporting in univariate versus multivariate statistics. The benefits to gifted education research of understanding the relationship among sample size, effect size, and statistical power are discussed.     

Planning Educational Research

Abstract

In planning a research study, investigators are frequently uncertain regarding the minimal number of subjects needed to adequately test a hypothesis of interest. The present paper discusses the sample size problem and four factors which affect its solution: significance level, statistical power, analysis procedure, and effect size. The interrelationship between these factors is discussed and demonstrated by calculating minimal sample size requirements for a variety of research conditions.     

Age as a Predictor of Student Success in Community College Technical Certificate Programs

ABSTRACT

This study was initiated to help determine if there was an associated probability between a student’s age relative to persistence and retention leading to the completion of a Technical Certificate at the community college level. The research model used in this study is based on a quantitative binary logistic regression model, whereby the dependent variable was completion versus non-completion of a Technical Certificate program. The study’s findings showed that there were statistically significant relationships between the various student’s age groups and completion of a Technical Certificate program. The baseline age group for this study was traditional age students as defined by those students who were between 16 and 21 years old. The age groups of 22 to 27, 28 to 34, 35 to 39, and students over the age of 39 were compared to the baseline group to determine if there was statistical significance between the graduation rates accordingly.     

The Effects of Special Curricular Study of Mexican Culture on Anglo and Mexican-American Students’ Perceptions of Mexican-Americans

Anglo and Mexican-American students in six sixth-grade classrooms were randomly assigned to control or experimental conditions (four weeks of special curricular study of Mexican culture). The effect of the treatment on perceptions of MexicanAmericans was studied by having students respond anonymously to an attitude inventory. The posttest means were significantly lower than the pretest means for all groups, apparently as a consequence of greater honesty in responding. There were significant differences in the ethnic group means, but no sex difference. The length and intensity of the treatment were not sufficient to cause changes in the ethnic attitudes and stereotypes.     

Needed Research in the Field of Learning

Abstract

One hundred and thirty-four children in grades 1-6 were administered a reading altitude inventory to ascertain if altitudes changed as boys and girls progressed through the grades. Results indicated no altitude changes from grades 1-3 nor between grades 5 and 6. A difference was found between grades 4 and 5 that appeared attributable to a more positive attitude toward nonclassroom-type reading in grade 5. Sex differences approaching statistical significance favored girls; however, no interaction between sex and grade level were reported.     

Statistical Techniques for the Study of Institutional Differences

David P. Ausubel, in his theory of meaningful, verbal reception learning, espouses the presentation of appropriate subsumers (organizers) to students prior to the presentation of the learning task. The question to be answered in the present study was: Are there significant differences in learning when students are presented organizers before the learning passages (pre-organizers), after the learning passages (post-organizers), or when they are presented only the learning passages (no-organizers). An analysis of covariance, using reading vocabulary achievement as the covariate, was used to compare adjusted treatment group means. No significant differences were found among adjusted means (p<. 15) in the available pool of third- and sixth-grade classes. Organizers do not facilitate learning when presented to similar classes using similar social studies materials.     

Changes in the Attitudes of Young Women Toward Child Guidance Over a Two-Year Period

Abstract

A non traditional model for teaching composition has extensive teaching and research value. The purpose of this research was to test the effect of oral discourse in teaching written discourse. The control group went from stimulus to problem to written discussion; the experimental group went from stimulus to problem to controlled oral discussion to written discussion on the model. Teachers evaluated unidentified papers of both pretest and posttest of control and experimental groups. With the use of the chi-square statistic, it was shown that the use of controlled oral discourse produced more grade increases and fewer decreases in written discourse in the experimental group than in the control group at the .01 level of significance.     

Ageism at university: A comparative analysis of young and older adult students

ABSTRACT

This study undertakes a comparative analysis of the negative stereotypes of older persons described by young and older adult university students at the University of Castilla-La Mancha. This study is innovative in that it compares three analytical perspectives: the perception of young students (18–27 years); the perception of older adult students (50 years and above) regarding older persons (hetero-stereotypes); and, finally, the perception of older adult students with regard to themselves as part of the group of older adult students (self-stereotypes).

The sample is made up of 460 young students and 148 older adult students who completed the Questionnaire on Negative Ageing Stereotypes (QNAS), comprising three dimensions: health; character and personality; and social relationships and motivation. A statistical analysis was subsequently carried out using the independent samples t-test in the IBM Statistics 19.0 program.

Of note among the principal results is the low level of negative stereotypes reported among the groups studied, as well as the existence of significant differences in their means, which indicate that the negative stereotypes of the groups differed. The most stereotyped of the dimensions analyzed was character and personality. In contrast, the least-stereotyped dimension was that of social relationships and motivation. Finally, the dimension of health occupied an intermediate position. With regard to the three perspectives studied, the older adult group (hetero-stereotypes) was the most stereotyped, followed by the young students and, finally, the older adult students with reference to self-stereotypes.