Conducting statistical tests with data from clustered school samples

This article discusses issues associated with statistical testing conducted with data from clustered school samples. Empirical researchers often conduct tests of statistical inference on sample data to ascertain the extent to which differences exist within groups in the population. Typically, much school‐related data are collected from students. These data are hierarchical because students are nested within classes within schools. This article studies the influence of this nesting on tests of statistical significance conducted with the student as the unit of analysis. Theory that adjusts F‐test scores for nested data in multi‐group comparisons is presented and applied to a teacher interaction dataset. The article demonstrates the potential impact of data hierarchy on the results of statistical testing if clustering is ignored. Data analysis techniques that recognize the clustering of students in classes are essential, and it is recommended that either multilevel analysis or adjustments to statistical parameters be undertaken in studies involving nested data.     

The effect of clustering on statistical tests: an illustration using classroom environment data

This paper discusses the effect of clustering on statistical tests and illustrates this effect using classroom environment data. Most classroom environment studies involve the collection of data from students nested within classrooms and the hierarchical nature to these data cannot be ignored. In particular, this paper studies the influence of intraclass correlations on tests of statistical significance conducted with the individual as the unit of analysis. Theory that adjusts t‐test scores for nested data in two‐group comparisons is presented and applied to classroom environment data. This paper demonstrates that Type I error rates inflate greatly as the intraclass correlation increases. Data analysis techniques that recognise the clustering of students in classrooms in classroom environment studies are essential, and it is recommended that either multilevel analysis or adjustments to statistical parameters be undertaken in studies involving nested data.     

Ecological rationality in teachers' conceptions of assessment across samples from Cyprus and New Zealand

Teacher conceptions of assessment are influential mediators of how assessment policy initiatives are implemented in schools. Four hierarchical, intercorrelated factors (i.e., assessment for improvement, school accountability, and student accountability, and assessment as irrelevant) of how teachers' conceive of assessment have been reported. However, most studies have been conducted only in English in jurisdictions with policies of low-stakes testing. This paper extends the research by surveying 249 Greek-Cypriot teachers with a Greek translation of the Teachers' Conceptions of Assessment inventory. Cyprus has a relatively low-stakes assessment policy during the compulsory school years, suggesting, under the assumption of ecological rationality, that conceptions would be similar to previous English-language studies. Exploratory factor analysis of the Cyprus data led to a five-factor solution with 24 items within two inversely correlated second-order factors (i.e., assessment is positive and negative; r = −0.49). A multigroup nested invariance confirmatory factor analysis found statistical invariance between the Cyprus and the New Zealand data. Mean score differences were small for two improvement-oriented conceptions, moderate for assessment that is bad, and large for school accountability and ignoring assessment factors. Similarities and differences in conceptions appear to reflect commonalities and discrepancies in educational system policies and practices.     

Teachers’ intentions to use national literacy and numeracy assessment data: a pilot study

In recent years the educational policy environment has emphasised data-driven change. This has increased the expectation for school personnel to use statistical information to inform their programs and to improve teaching practices. Such data include system reports of student achievement tests and socio-economic profiles provided to schools by various state education departments’ data services. This paper reports on a pilot study that explored factors affecting Mathematics and English teachers’ intentions to engage with the statistical data their schools receive and to consider these data when making decisions about their teaching practices. It was found that most teachers perceived that such data identify weak students and some teachers (mostly mathematics teachers) thought that they can help to identify curriculum topics that need attention. Most teachers felt that the reports were not easy to understand. Confidence in dealing with statistical data was a problem for many teachers, but especially for English teachers.     

Teacher collaboration in the context of the Responsive Classroom approach

This mixed‐method study examined characteristics and predictors of teacher collaboration. Limited research exists that describes the characteristics of teacher collaboration, and surprisingly little work explains the ways in which teaching experience and teachers’ perceptions of the school environment influence teacher collaboration. Questionnaire data were collected from a sample of 118 elementary school teachers in six schools in a northeast urban school district, and interviews were conducted with administrators in each school. Three schools were in their second year implementing the Responsive Classroom ^® (RC) approach, and three schools were comparison schools. Teachers reported collaborating approximately once or twice per month, generally with fellow grade‐level teachers about student‐centered topics. Teachers in RC schools reported more frequent formal collaboration than comparison school teachers. In regards to predicting teacher collaboration, teachers who used more RC practices and/or resources reported collaborating more, valuing collaboration to a higher degree, and perceiving greater involvement in school decision‐making, controlling for whether they taught at a RC school. Also, teachers’ perceptions of the school environment related positively to teacher collaboration. The current study adds to the understanding of teacher collaboration and its antecedents, contributing uniquely to the literature on how a school‐wide educational initiative is associated with teachers’ perceptions of their school environment as well as their collaborative behaviors and beliefs.     

Teachers' Images of their Schools and Perceptions of their Work Environments

The Images of Schools through Metaphor (ISM) was developed as a simple, economical questionnaire to allow school leaders and researchers to survey teachers' images of their school. The article describes briefly aspects of the development and validation of this questionnaire and its application in an investigation of relationships between teachers' images of their schools and their perceptions of its work environment as assessed by the School Level Environment Questionnaire (SLEQ). The sample used consisted of 162 teachers in 48 schools in Tasmania, Australia, each of whom completed both the ISM and the SLEQ. The various results revealed were generally of considerable statistical significance indicating a strong relationship between the images teachers have of their school and the perceptions they have of its work environment.     

Exploring UK secondary teachers' professional values: an overview of the literature since 2000

Background:?Professional commitment from teachers requires more than contractual compliance as personal and professional values are integral to teaching practice. Secondary school teachers are expected to act as role models and demonstrate positive attributes and attitudes in an evolving educational context. Little is known about how they understand or interpret their own values, or realise the shared values that lie at the heart of a school community.

Purpose:?The purpose of this review is to summarise the literature relating to values that teachers hold. The review is concerned with theoretical and empirical studies in which such values are considered, and also any tensions identified between the personal values of teachers and organisational values of the schools reflected in professional practice. This scoping exercise is part of two-year funded empirical research carried out by the authors in five secondary schools in England, using data from individuals, groups and schools on the personal values of teachers and head teachers. This research explored whether these values are congruent with the organisational values schools seek to establish.

Design and methods:?An extensive literature review was undertaken relating to the values held by secondary school professionals. The search was restricted to more recent works (i.e. the year 2000 onwards), and was mainly UK and secondary education focused. The search was conducted using the educational database, Education Indexes on Dialog, and the Electronic Journal Service which is a database covering wider disciplines. To reach harder-to-find research reports, additional strategies were adopted including hand searching of key journals and reference to existing publication lists. The review generated over 1000 published references that were then screened to determine whether publications were based on empirical research conducted in schools, and/or helped a deeper theoretical understanding about the meaning of values.

Conclusion:?The review highlights the integral part values continue to play in education. This article presents a framework of understanding derived from the literary contexts in which values are discussed. There is little empirical data to support current understanding about the values that secondary teachers hold, and how these fit with the organisational values of the schools in which they work. Studies in the context of values and professionalism stress the need for dialogue and reflection so that ‘implicit’ values that teachers hold become explicit, and refined through practice.     

Safety and security in schools in KwaZulu-Natal

This article is based on research conducted on the topic: Safety and security in schools: The case of KwaZulu-Natal. For the research project a purposive sample consisting of secondary school learners, teachers, school governing body chairpersons and principals were selected from the rural and township schools used in this study to explore the views of respondents about the state of safety and security in their school environments. The research was grounded in the social interpretive tradition, using an ethnographic research design, with one-on-one and focus group interviews. Data were analysed using critical discourse analysis to construct the main theme: namely, safety within schools. The findings indicated that learners and teachers are not safe in their schools. Lack of safety and security is characterised by the vulnerability of schools with regard to issues such as fighting, violent murders, sexual harassment, stabbings, the use of derogatory language and threats. It is argued that there is an urgent need to address the concerns of the participants.     

Which Pedagogical Parameters Predict the General Quality of ICT Integration from the Perspective of Elementary School Leaders?

Technological changes in the digital age require schools to integrate innovative technologies in learning and the curriculum. This study analyzes data collected from elementary schools toward the end of the second and the third years of the national program for the gradual integration of ICT in Israeli schools. The study examines how school principals and ICT coordinators assess the systemic changes that have occurred in their schools. The parameters explored were collaboration (intraschool vs. interschool collaboration), digital content (using vs. designing digital content), and e-communication (within the teaching staff vs. between staff and families). An online questionnaire was distributed to the entire district of the Ministry of Education. The analysis was carried out on a total of 358 schools. Regression analysis showed that intraschool collaboration, digital content use and design, pedagogical update of class websites, and e-communication within the teaching staff explained 47.7% of variance in the general quality of ICT integration. It seems that school ICT leaders assess the general quality of ICT integration according to internal factors—in terms of collaboration within their schools and online interactions with colleagues—rather than external factors—collaborative activities between schools or e-communication with students and parents.     

School effectiveness in 14 sub-Saharan African countries: Links with 6th Graders' reading achievement

In this article school characteristics linked with educational effectiveness in 14 sub-Saharan African countries are identified. Effectiveness has been defined in terms of students' literacy achievement at the end of Grade 6, after taking their social and academic backgrounds into account. The data used are from the second major educational policy research study conducted by the 14 countries that constitute the Southern African Consortium for Monitoring Educational Quality study (SACMEQ). This study, known as the SACMEQ II Project, collected questionnaire and test survey data from around 42,000 students in 2,300 schools in the 14 countries. Information about schools was obtained from questionnaires administered to teachers and principals. The multilevel methods (HLM) we used are appropriate for such analyses and the SACMEQ II nested data design. Schools are differentiated in terms of their social compositions, contexts, and resources (physical and human). Though student achievement is strongly associated with students' social and academic background in all SACMEQ countries, school effects vary across countries. Evident is a pattern of higher achievement in urban schools, with more resources and higher-quality teachers; achievement is typically lower in large schools and those that offer education in “shifts.” Policy implications are emphasized.     

Using value-added data for school self-evaluation: a case study of practice in inner-city schools

This study demonstrates how value-added research undertaken at the local education authority (LEA) is used directly by heads and teachers in their efforts to raise standards in schools. It draws on a decade of experience of supporting schools in the effective use of performance data for school self-improvement. The article highlights the various approaches of feeding back research findings to schools, including illustrations of contextual and value-added information, and examples of working with schools to support school self-evaluation. This is followed by detailed discussions of key issues raised during the training sessions on the use of performance data and analysis of the views of headteachers, teachers and governors about the performance data feedback to schools. The main findings from this research suggest that schools use performance data and research findings effectively for school improvement purposes. Reasons for this success story in the LEA's schools are critically discussed. The article concludes that research which addresses headteachers, classroom teachers, governors and policy makers' concerns, as is the case with this study, is likely to attract their attention and be used for raising standards.     

A multilevel examination of an instrument measuring school renewal via teachers

The purpose of the present study was to assess an instrument that uses responses from teachers to measure principal behaviors on the critical educational issue of school renewal. To account for the data hierarchy of teachers nested within schools, we employed a multilevel confirmatory factor analysis and estimated multilevel reliabilities at both teacher and school levels to examine the instrument. With 2765 teachers from 148 schools, the instrument showed acceptable structural validity. Specifically, we found that when teachers provide responses as indicators of school renewal, the instrument can generate a valid and highly reliable measure or estimate of school renewal at the school level. However, data analysis at the teacher level attempting to use teacher perceptions of school renewal as either dependent variables or independent variables should be avoided because of the low reliability at the teacher level.     

Designing your sample efficiently: clustering effects in education surveys

Background:?Education, and information about education, is highly structured: individuals are grouped into classes, which are grouped into schools, which are grouped into local authorities, which are grouped into countries. The degree of similarity among members of a group, such as a school or classroom, is a very important factor in the design and analysis of studies in education.

Purpose:?The aim of this article is to provide information on this degree of similarity within schools to enable those involved in carrying out surveys of schools to do so most efficiently in terms of resources and minimum disturbance of schools.

Sources of data:?This paper uses data from 13 studies at primary and secondary level conducted by the National Foundation for Educational Research in England and Wales.

Main argument:?The degree of similarity among members of a group is measured by two statistics, the intra-cluster correlation and the design effect. The study described here classifies outcomes into a number of categories and estimates intra-cluster correlation and design effect. The relevance of the results to survey design and analysis is discussed, and examples of how to use these are given.

Conclusions:?The main findings of this study, rather than conclusions as such, are the intra-cluster correlations for each topic category. However, the paper reaches some tentative conclusions about the degree of clustering by topic. Using Hox's convention (Multilevel analysis: Techniques and applications, Lawrence Erlbaum, London, 2002) for the size of intra-cluster correlations, it was found that the degree of clustering of achievement was high, while ethnic and language variables were highly clustered in secondary but not primary. By contrast, attitudes towards school, educationally relevant home characteristics, and perception of school policies have quite low values of ρ (mean < 0.05), defined as small.     

The impact of multilevel factors on technology integration: the case of Taiwanese grade 1–9 teachers and schools

Technology integration is influenced by many factors related to the teacher and the school environment. While many studies have examined factors that influence teachers’ use of Information and Communication Technology (ICT) at the teacher level, a growing number of studies have suggested that it is beneficial to examine those factors using multilevel modeling. Multilevel analysis can separate the impact of teachers from the school environment and provide insight into the influence of those factors at each level. This study uses multilevel modeling to analyze data from 3,652 grade 1–9 teachers in 289 schools in Taiwan. The results of this multilevel analysis found both teacher-level and school-level factors contributing to the integration of ICT in the teaching process. Among teacher level factors, teachers’ beliefs and hours of training in the previous year predicted ICT integration proficiency well. Among school level factors, training hours and teachers’ perceived school support are the most important factors that impact ICT integration. School support is not only a significant factor at the school level but also a strong predictor at the teacher level. In addition, important school level variables included access to Internet connectivity, availability of projectors, and stability of computers. These results highlight the importance of individual teachers and the role schools play in ICT integration.     

Designing science laboratories: learning environments,school architecture and teaching and learning models

This article on secondary schools science laboratories in Portugal focuses on how school space functions as a pedagogical and political instrument by contributing to shape the conditions for teaching and learning dynamics. The article places the impact of changes to school layouts within the larger context of a public school renovation programme, discussing how school space functions as a pedagogical and political instrument. The focus is on science laboratories as a particular learning environment for science education. The study, conducted between 2010 and 2011 in 13 renovated schools within the framework of the Portuguese Secondary School Modernisation Programme, drew on document analysis, interviews, pupil and teacher surveys and site-specific focus groups. One of the main findings is that teachers found that science laboratories were the most controversial and debated of all the renovated learning spaces. Considering that the science laboratory layout was intended to be universal across all schools, there was little intervention by the architects responsible for the renovation of the schools. Focusing on the analysis of the decision to change the science laboratory design within the aims of the education policy, this article discusses how teachers’ criticisms were a response to some of the educational policy goals underlying the renovation of school buildings and the potential impact on science education, namely, the relationship between flexibility of space organisation and pedagogical approaches.     

Towards state‐mandated testing in Germany: how do teachers assess the pedagogical relevance of performance feedback information?

The paper analyses how teachers view state‐mandated tests in Baden‐Württemberg, a large state in the south‐west of Germany. For the first time in 2006, public secondary schools were bound by law to administer Vergleichsarbeiten [state‐wide tests] in up to three relevant subjects after Grade 6. Scholars in the field of school improvement emphasise that there is a gap between performance feedback provided by external tests and how this information is internally processed. To bridge this gap, a crucial precondition is that teachers must accept state‐wide testing as a useful instrument for improving the performance of schools. To examine how teachers view the pedagogical relevance of performance feedback information, 256 secondary schools were approached, and an anonymous questionnaire was completed by a representative sample of 307 teachers with Grade 6 classes. Analysis of the quantitative data reveals that teachers in general secondary schools (the lower level of the academically streamed secondary system) are more open‐minded towards the tests than their colleagues in intermediate schools or in grammar schools (the highest level). Even so, only a small percentage of teachers in general secondary schools acknowledge the pedagogical relevance of performance feedback information for improving teaching strategies. This result suggests that state‐mandated testing in Baden‐Württemberg is highly unlikely to initiate or support school improvement activities for the time being.     

Implementing a new science National Curriculum for England: how trainee teachers see the How Science Works strand in schools

This article reports empirical work conducted by three university teacher education providers into how a major revision of the science curriculum in England, the How Science Works strand, has been translated into practice for 14–16 year olds in schools. Data were collected from pre-service teachers about their understanding, experiences, perceived challenges and resourcing of How Science Works during their final school experience. The results indicate that implementation of How Science Works in schools lies on a spectrum between two extremes referred to as creative implementation and restricted implementation; a recognition that success with this new approach requires a fuller engagement in higher order thinking skills; and that the importance of differentiation is paramount if all pupils are to connect with the new curriculum. The article concludes that support is needed for major transitions in pedagogy with both serving and trainee teachers. This needs to start with teacher education and encompass teacher development, with greater involvement both in school and university programmes.     

Teachers’ perceptions of school climate as an indicator of their beliefs of effective teaching

The ability to express oneself clearly in both a mother tongue and a foreign language is a foundation principle of the new national curriculum in Estonia. Therefore, research was conducted to determine whether there was a possible relationship between English as a foreign language (EFL) teachers’ perceptions of effective teaching and the contextual evaluation of the school. The aim of this article is to describe the relationship between the perceptions of school climate and effective teaching among EFL teachers. The study revealed that EFL teachers’ perceptions of the school climate correlate to some extent with their perceptions of effective teaching practices, with some differences between teachers of Estonian-language schools compared with those of Russian-language schools. The research also highlighted that senior management should place more emphasis on creating a cooperative school climate for developing a learner-centred teaching approach as one of the foundation principles of the new national curriculum.     

Irish teachers,starting on a journey of data use for school self-evaluation

With much of the literature on school self-evaluation (SSE) stressing the importance of data use, this article explores how teachers in Irish post-primary schools are coming to terms with this new challenge. Since 2012, all schools in Ireland are required to engage in SSE for the purpose of improving student outcomes. For the first time, teachers and school leaders are being asked to systematically gather and analyse various types of data, devise improvement plans and implement improvements. Despite such demands, the compulsory education system in Ireland operates within a low-stakes accountability environment, with an absence of published school league tables and no consequences for poor school performance. It is also interesting to explore the introduction of a school improvement process that requires data as evidence for self-evaluation, but where very little data currently exists compared to other jurisdictions and where the discourse of data use in schools is relatively new.This article outlines the experience of 13 post-primary schools that were supported by the DCU Centre for Evaluation, Quality and Inspection to complete an SSE process, during which, each school gathered and analysed a range of data. This study is part of a larger action research project which explores various aspects of the implementation of SSE in schools, including models of support and continuing professional development for schools. This article looks specifically at the use of data by the schools involved. The key research questions ask: what data was gathered by the schools and what was the attitude to and experience of data-use among teachers? In doing so, this article explores some of the current research questions in relation to data use in schools. Overall, the findings indicate that schools gathered a range of data, which was mainly quantitative due to a focus on quantitative target setting. Despite a generally positive attitude to the usefulness of data and the skills learned, participants did not appear convinced that they would be involved in data use on an ongoing basis.     

Exploring the Relationship Between Attentional Capture and Prior Knowledge in a Science-Based Multi-user Virtual Environment: an Individual Growth Model Analysis

In this study, we investigate trends in the relationship between what students know and the types of data that capture their attention over time in a science-based multi-user virtual environment. Longitudinal analyses of the patterns of data collected by 143 middle school students (nested within 5 teachers) showed that student prior knowledge was marginally (p?<?.10) related to variation in the attentional value (visibility and location) of data they collected over time, explaining about 2% of said variation. By contrast, accounting for the clustering of students by teacher was statistically significantly (p?<?.05) related to variation in trends in attentional values and explained 36% of said variation.