The effect of clustering on statistical tests: an illustration using classroom environment data

This paper discusses the effect of clustering on statistical tests and illustrates this effect using classroom environment data. Most classroom environment studies involve the collection of data from students nested within classrooms and the hierarchical nature to these data cannot be ignored. In particular, this paper studies the influence of intraclass correlations on tests of statistical significance conducted with the individual as the unit of analysis. Theory that adjusts t‐test scores for nested data in two‐group comparisons is presented and applied to classroom environment data. This paper demonstrates that Type I error rates inflate greatly as the intraclass correlation increases. Data analysis techniques that recognise the clustering of students in classrooms in classroom environment studies are essential, and it is recommended that either multilevel analysis or adjustments to statistical parameters be undertaken in studies involving nested data.