Conducting statistical tests with data from clustered school samples

This article discusses issues associated with statistical testing conducted with data from clustered school samples. Empirical researchers often conduct tests of statistical inference on sample data to ascertain the extent to which differences exist within groups in the population. Typically, much school‐related data are collected from students. These data are hierarchical because students are nested within classes within schools. This article studies the influence of this nesting on tests of statistical significance conducted with the student as the unit of analysis. Theory that adjusts F‐test scores for nested data in multi‐group comparisons is presented and applied to a teacher interaction dataset. The article demonstrates the potential impact of data hierarchy on the results of statistical testing if clustering is ignored. Data analysis techniques that recognize the clustering of students in classes are essential, and it is recommended that either multilevel analysis or adjustments to statistical parameters be undertaken in studies involving nested data.     

Statistical tests conducted with school environment data: The effect of teachers being clustered in schools

This article discusses the effect of clustering on statistical tests conducted with school environment data. Because most school environment studies involve the collection of data from teachers nested within schools, the hierarchical nature to these data cannot be ignored. In particular, this article considers the influence of intraschool correlations on tests of statistical significance conducted with the individual teacher as the unit of analysis. Theory that adjusts t test scores for nested data in two-group comparisons is presented and applied to school environment data. This article demonstrates that Type I error rates inflate greatly as the intraschool correlation increases. Because data analysis techniques that recognise the clustering of teachers in schools are essential, it is recommended that either multilevel analysis or adjustments to statistical parameters be undertaken in school environment studies involving nested data.     

The Power of the Test for Treatment Effects in Three-Level Block Randomized Designs

Abstract

Experiments that involve nested structures may assign treatment conditions either to subgroups (such as classrooms) or individuals within subgroups (such as students). The design of such experiments requires knowledge of the intraclass correlation structure to compute the sample sizes necessary to achieve adequate power to detect the treatment effect. This study provides methods for computing power in three-level block randomized balanced designs (with two levels of nesting) where, for example, students are nested within classrooms and classrooms are nested within schools. The power computations take into account nesting effects at the second (classroom) and at the third (school) level, sample size effects (e.g., number of level-1, level-2, and level-3 units), and covariate effects (e.g., pretreatment measures). The methods are generalizable to quasi-experimental studies that examine group differences on an outcome.     

The Effect of Ignoring Classroom‐Level Variance in Estimating the Generalizability of School Mean Scores

Contemporary educational accountability systems, including state‐level systems prescribed under No Child Left Behind as well as those envisioned under the “Race to the Top” comprehensive assessment competition, rely on school‐level summaries of student test scores. The precision of these score summaries is almost always evaluated using models that ignore the classroom‐level clustering of students within schools. This paper reports balanced and unbalanced generalizability analyses investigating the consequences of ignoring variation at the level of classrooms within schools when analyzing the reliability of such school‐level accountability measures. Results show that the reliability of school means cannot be determined accurately when classroom‐level effects are ignored. Failure to take between‐classroom variance into account biases generalizability (G) coefficient estimates downward and standard errors (SEs) upward if classroom‐level effects are regarded as fixed, and biases G‐coefficient estimates upward and SEs downward if they are regarded as random. These biases become more severe as the difference between the school‐level intraclass correlation (ICC) and the class‐level ICC increases. School‐accountability systems should be designed so that classroom (or teacher) level variation can be taken into consideration when quantifying the precision of school rankings, and statistical models for school mean score reliability should incorporate this information.     

The Power of the Test for Treatment Effects in Three-Level Cluster Randomized Designs

Abstract

Experiments that involve nested structures often assign entire groups (such as schools) to treatment conditions. Key aspects of the design of such experiments include knowledge of the intraclass correlation structure and the sample sizes necessary to achieve adequate power to detect the treatment effect. This study provides methods for computing power in three-level cluster randomized balanced designs (with two levels of nesting), where, for example, students are nested within classrooms and classrooms are nested within schools and schools are assigned to treatments. The power computations take into account nesting effects at the second (classroom) and at the third (school) level, sample size effects (e.g., number of schools, classrooms, and individuals), and covariate effects (e.g., pretreatment measures). The methods are applicable to quasi-experimental studies that examine group differences in an outcome.     

Learning environment,motivation, and achievement in high school science

In a study of the relationship between high school students' perceptions of their science learning environments and their motivation, learning strategies, and achievement, 377 students in 22 introductory science classrooms completed surveys in the fall and spring of their ninth‐grade year. Hierarchical linear regression was used to model the effects of variables at both the classroom and individual level simultaneously. High intraclass agreement (indicated by high parameter reliability) on all classroom environment measures indicated that students shared perceptions of the classroom learning environment. Controlling for other factors, shared perceptions that only the most able could succeed in science classrooms and that instruction was fast‐paced and focused on correct answers negatively predicted science achievement, as measured on a districtwide curriculum‐linked test. Shared perceptions that classrooms focused on understanding and independent thinking positively predicted students' self‐reported satisfaction with learning. Implications of these results for both teaching and research into classroom environments are discussed. © 2003 Wiley Periodicals, Inc. J Res Sci Teach 40: 347–368, 2003     

Effects of flipping the classroom on learning outcomes and satisfaction: A meta-analysis

In a flipped classroom, students study instructional material before class and apply this material during class. To provide a statistical synthesis of current research on effects of flipped classrooms, we conducted meta-analyses that included 114 studies which compared flipped and non-flipped classrooms in secondary and postsecondary education. We found a small positive effect on learning outcomes, but no effect was found on student satisfaction regarding the learning environment. In addition, we found considerable heterogeneity between studies. Moderator analyses showed that students in flipped classrooms achieve higher learning outcomes when the face-to-face class time was not reduced compared to non-flipped classrooms, or when quizzes were added in the flipped classrooms. We conclude that a flipping the classroom (FTC) approach is a promising pedagogical approach when appropriately designed. Our results provide insights into effective instructional FTC design characteristics that support an evidence-informed application of FTC.     

Changing the Learning Environment to Enhance Students' Metacognition in Hong Kong Primary School Classrooms

This study reports on the impact of a 2-month classroom intervention that sought to alter the learning environment of two Hong Kong Primary Year 3 general studies classrooms. Mixed methodology, employing quantitative and qualitative data-gathering strategies, was used to investigate changes to the learning environments, including changes to the teachers' language and ultimately the students' metacognition. The quantitative facet of the research involved the development of a 15-item learning environments instrument, the General Studies Metacognitive Orientation Scale (GSMOS), that evaluated elements of the metacognitive orientation of the classrooms' learning environments. While the data from the administration of the GSMOS suggested no statistical differences between the pre- and post-intervention environments of the classrooms, student interviews and classroom observations provided supportive data for some changes, which resulted in students developing metacognitive knowledge of teacher-selected thinking and learning strategies, as well as some awareness and limited control of their use of such strategies in their classrooms.     

Noise Levels in Hong Kong Primary Schools: Implications for classroom listening

Many researchers have stressed that the acoustic environment is crucial to the speech perception, academic performance, attention, and participation of students in classrooms. Classrooms in highly urbanised locations are especially vulnerable to noise, a major influence on the acoustic environment. The purpose of this investigation was to determine noise levels in one such urban environment, Hong Kong. The ambient noise level, and its relationship to the speech intensity levels of both teachers and students was surveyed in 47 primary school classrooms. Moreover, the presence of acoustical treatments for noise reduction and the use of classroom amplification systems were documented for each classroom. The survey found that the mean occupied noise level was 60.74 dB (A); the mean unamplified and amplified speech‐to‐noise ratios of teachers were 13.53 dB and 18.45 dB, respectively; while the mean unamplified speech‐to‐noise ratio for students was 4.13 dB. Most of the classrooms exhibited insufficient acoustical treatments to provide significant noise reduction. The listening environment in many Hong Kong primary schools was not favourable for optimal classroom learning. Recommendations for improving the acoustical environment in classrooms in highly urbanised locations such as Hong Kong are discussed.     

Technology,learning and instruction: distributed cognition in the secondary English classroom

In this paper, we analyse interactions between secondary students and pre‐service teachers in an online environment in order to understand how their meaning‐making processes embody distributed cognition. We begin by providing a theoretical review of the ways in which literacy learning is distributed across learners, objects, tools, symbols, technologies and the environment in modern English language arts classrooms. This is followed by a case study where we identify how programme values, textual resources and cultural schema function as distributed tools. In traditional schools, with an emphasis on taking standardised tests, the learning environment is designed on the view that learning is a transaction that happens solely ‘inside the head’. Unfortunately, this pushes many students to the margins of classroom engagement and participation. By analysing students' and pre‐service teachers' online discourse, we argue that virtual spaces can facilitate critical dialogue and can act as catalysts for a distributed theory of mind.     

The diffusion and appropriation of ideas in the science classroom: Developing a taxonomy of events occurring between groups of learners

Abstract: While research on group learning has focused almost exclusively on interactions among individuals within groups, there has been little research on phenomena occurring between groups of learners in classrooms. This exploratory study identifies, describes, and categorizes events occurring between members of different learning groups in three ninth‐grade physical science classrooms. Analysis of interaction data from a collaborative activity involving the construction of complex electrical circuits was used to create a working taxonomy of inter‐group events. This taxonomy was then tested for generalizability with four other collaborative student projects and was found to account for all inter‐group events during these activities. Evidence gathered from videotape, interview, and observational data further indicated that many types of inter‐group interactions are qualitatively different from intra‐group interactions, and that inter‐group interactions contribute significantly to learning within a design‐based classroom context. Students in these classrooms effectively used the special expertise of others outside of their assigned groups and exploited features of the material environment in specific ways with others outside of their assigned groups to create complex products. Examples are included of how concepts, ideas, tools, tool‐related practices, and materials diffused throughout the classroom environment and were appropriated by learners in various ways to contribute to the construction of the design artifacts. © 2000 John Wiley & Sons, Inc. J Res Sci Teach 38: 17–42, 2001     

Agreement among classroom observers of children's stylistic learning behaviors

Standardized and reliable rating scales have an important role in educational assessment and behavioral classroom intervention. The Learning Behaviors Scale (LBS) is a standardized behavior rating scale designed to report how individual students respond to classroom learning situations. This study investigated the interobserver agreement of the LBS with the use of linear and intraclass correlation methods. The methods jointly assessed the three salient aspects of observer judgments—severity level, rank order, and directionality. Participants were 72 students enrolled in special education programs as observed by 16 educators in eight self-contained classrooms. Both linear and intraclass coefficients were substantial (averages = .83 and .84, respectively). No significant observer effect was found. Moreover, the LBS produced comparable levels of differential learning styles for assessments of individual children. © 1998 John Wiley & Sons, Inc.     

Women's talk in the ivory tower

Inequities in higher education persist, as revealed both by statistical indicators and in continuing reports by women students and faculty that the academic environment itself is alienating. Our review of the research on academic settings, including classroom interaction and the patterns of interaction among and between females and males, leads us to identify the structure of classroom talk itself as one reason many women find the classroom inhospitable. The norms of classroom interaction are more closely aligned with typical male patterns of interaction than with typical female patterns. We suggest that both can exist in classrooms side‐by‐side, as important resources for females and males. We conclude with suggestions for teaching and research.     

The Student Actions Coding Sheet (SACS): An instrument for illuminating the shifts toward student‐centered science classrooms

This study describes the development of an instrument to investigate the extent to which student‐centered actions are occurring in science classrooms. The instrument was developed through the following five stages: (1) student action identification, (2) use of both national and international content experts to establish content validity, (3) refinement of the item pool based on reviewer comments, (4) pilot testing of the instrument, and (5) statistical reliability and item analysis leading to additional refinement and finalization of the instrument. In the field test, the instrument consisted of 26 items separated into four categories originally derived from student‐centered instruction literature and used by the authors to sort student actions in previous research. The SACS was administered across 22 Grade 6–8 classrooms by 22 groups of observers, with a total of 67 SACS ratings completed. The finalized instrument was found to be internally consistent, with acceptable estimates from inter‐rater intraclass correlation reliability coefficients at the p < 0.01 level. After the final stage of development, the SACS instrument consisted of 24 items separated into three categories, which aligned with the factor analysis clustering of the items. Additionally, concurrent validity of the SACS was established with the Reformed Teaching Observation Protocol. Based on the analyses completed, the SACS appears to be a useful instrument for inclusion in comprehensive assessment packages for illuminating the extent to which student‐centered actions are occurring in science classrooms.     

Parent and student perceptions of classroom learning environment and its association with student outcomes

This research is distinctive in that parents’ perceptions were utilised in conjunction with students’ perceptions in investigating science classroom learning environments among Grade 4 and 5 students in South Florida. The What Is Happening In this Class? (WIHIC) questionnaire was modified for young students and their parents and administered to 520 students and 120 parents. Data analyses supported the WIHIC’s factorial validity, internal consistency reliability and ability to differentiate between the perceptions of students in different classrooms. Both students and parents preferred a more positive classroom environment than the one perceived to be actually present, but effect sizes for actual-preferred differences were larger for parents than for students. Associations were found between some learning environment dimensions (especially task orientation) and student outcomes (especially attitudes). Qualitative methods suggested that students and parents were generally satisfied with the classroom environment, but that students would prefer more investigation while parents would prefer more teacher support. The study provides a pioneering look at how parents and students perceive the science learning environment and opens the way for further learning environment studies involving both parents and students.     

Teacher interpersonal competence for Dutch secondary multicultural classrooms

In the Netherlands, as in many European countries, classrooms display a growing cultural diversity. There are only very limited empirically supported data on the interpersonal competence teachers need in Dutch classrooms and studies from other countries cannot be generalized because of differences in the composition of the student population. This paper reports on an exploratory study in two schools on teachers' experiences in multicultural classes followed by an in‐depth case study of one expert teacher. We use this study to answer the question to what degree this teacher displays interpersonal competence (teaching behaviours, knowledge and attitudes) specific for teaching in multicultural classrooms by comparing this teacher's competence with generic interpersonal teaching competence. The teacher studied seems to be aware of special needs of students from different cultures and to use this knowledge to apply specific teaching strategies and interpersonal cues to create a positive classroom atmosphere and to cater for needs of diverse students. Further, the results indicate that many elements of this teacher's interpersonal competence in teaching a multicultural class can be considered aspects of generic teaching competence. It seems that the multicultural classroom puts heavier demands on this competence than a less diverse classroom.     

Associations between school-level environment and science classroom environment in secondary schools

This article describes a study of links between school environment and science classroom environment. Instruments to assess seven dimensions of school environment (viz., Empowerment, Student Support, Affiliation, Professional Interest, Mission Consensus, Resource Adequacy and Work Pressure) and seven dimensions of classroom environment (viz., Student Affiliation, Interactions, Cooperation, Task Orientation, Order & Organisation, Individualisati n and Teacher Control) in secondary school science classrooms were developed and validated. The study involved a sample of 1,318 students in 64 year 9 and year 12 science classes and 128 teachers of science in Australian secondary schools. Using the class mean as the unit of analysis for student data, associations between school and classroom environment were investigated using simple, multiple and canonical correlational analyses. In general, results indicated weak relationships between school and classroom environments and they reinforced the view that characteristics of the school environment are not transmitted automatically into science classrooms.     

Exploring the nature of divergent thinking: A multilevel analysis

In this study, a large sample with a clustered data structure from an educational context was utilized to analyze the relationship between cognitive abilities, school type, gender, and divergent thinking. The sample comprised 1098 students in 55 classrooms. A sequence of nested multilevel regression analyses revealed that processing capacity, as a core component of intelligence, as well as processing speed shared a significant amount of variance with divergent thinking. Additionally, school type affected divergent thinking even when controlling for intellectual abilities. Furthermore, a significant gender difference in divergent thinking favouring females was found. Gender differences as well as the effect of processing capacity and processing speed varied across classrooms, reflecting differences in the classroom environment. The nature of the relationships of cognitive and social factors to divergent thinking was considered.     

智慧教室环境感知与互联网自我效能感的关系研究

为探究智慧教室环境下大学生环境感知与互联网自我效能感之间的关系,文章以华中地区某高校学生为研究对象,调查了学生对智慧教室的环境感知;同时,通过准实验研究,对智慧教室环境与传统多媒体环境下的课堂教学对学生互联网自我效能感影响的差异进行了探究。文章发现:学生对智慧教室的物理环境和社会心理环境普遍持积极认可的态度,采用智慧教室的实验组学生的互联网自我效能感显著高于采用传统多媒体教室的对照组学生。此外,文章的相关性分析结果显示:学生关于智慧教室的环境感知与其互联网自我效能感显著正相关。此结论表明,改善学生关于智慧教室的环境感知对于提高其互联网自我效能感具有重要意义。