Exploring the Technical Quality of Using Assignments and Student Work as Indicators of Classroom Practice

This study presents findings from research developing indicators of classroom practice to monitor the influence of school reform initiatives on students' learning environments and to support the improvement of instructional practice. The work reported here entailed collecting assignments and student work and observing classrooms in schools participating in a large-scale urban school reform effort. Overall, the reliability and consistency of the assignment ratings were good. Results also indicate that the quality of the assignments was statistically significantly associated with the quality of observed instruction and student work. It appears that our method shows promise for use in large-scale evaluation settings and identifies important dimensions of practice that could support teacher self-evaluation and reflection.     

Observing instructional quality in the context of school evaluation

Observational measures of instructional quality are used in both research and evaluation contexts. In European school evaluation systems, one main intended use of classroom observation results is to inform schools' instructional improvement efforts. This study examined the quality of such diagnostic information in the context of a school evaluation system in Germany. The factor analytic results indicate that the empirical structure of the observation instrument lacks correspondence with its original normative model but does mirror a five-factor model based on recent literature, including classroom management, two different aspects of student orientation, cognitive activation, and classroom assessment. The generalizability analyses show that both indicators and observers represent important sources of measurement error. Additionally, their magnitude is larger for shorter observations (lesson segments). At least two, sometimes three, observers and between five and ten indicators per dimension are needed to achieve sufficiently reliable results, but results importantly depend on which dimension of instructional quality is being assessed.     

Quantifying Error in Survey Measures of School and Classroom Environments

Developing indicators that reflect important aspects of school and classroom environments has become central in a nationwide effort to develop comprehensive programs that measure teacher quality and effectiveness. Formulating teacher evaluation policy necessitates accurate and reliable methods for measuring these environmental variables. This article investigates different approaches to quantify measurement error in school- and classroom-level indicators constructed from survey data collected at a lower level of aggregation (i.e., teachers or students). Within a generalizability theory framework the article first compares four widely used approaches for accounting for measurement error in school- and classroom-level aggregate indicators. Then, it uses two empirical examples to demonstrate how each of these approaches can lead to different conclusions about the precision of aggregate indicators, and can influence inferences about the relationships between environmental variables and policy-relevant outcomes. Finally, the article discusses the degree to which these commonly used models accurately represent the structure of the data found in common survey administration scenarios.     

Evaluation of Handheld Computers for Direct Systematic Classroom Observation

Through this study the authors evaluate outcomes associated with the use of handheld computers by interventionists in improving the efficiency of direct systematic classroom observation. Information from observations is used by interventionists for treatment planning and evaluation. In this study, interventionists were trained to use personal digital assistants with classroom observational software for use with students who displayed low levels of academic engagement. Results indicated that the personal digital assistants and observational software were perceived as user-friendly, increased computer self-efficacy, and facilitated treatment planning and evaluation. Discussion focuses on implications for use of handheld computers and mobile devices by interventionists.     

The differential impact of observational learning and practice-based learning on the development of oral presentation skills in higher education

The present study focuses on the design and evaluation of an innovative instructional approach to developing oral presentation skills. The intervention builds on the observational learning theoretical perspective. This perspective is contrasted with the traditional training and practice approach. Two sequencing approaches – learners starting with observational learning versus learners starting with practice opportunities only – were compared. It was hypothesised that learners starting with observational learning would outperform learners in the practice only condition. The results suggest a significant differential impact on development of oral presentation skills. This impact of the observational learning training approach is only found in a limited number of evaluation criteria. Results additionally suggest that students are highly motivated to learn this type of skill. Interaction effects between student characteristics and instructional interventions were not significant.     

Predictors of developmentally appropriate classroom practices in kindergarten through third grade

This study was designed to (a) test the psychometric properties of a new observation measure of developmentally appropriate classroom practices in kindergarten through third-grade classrooms, and (b) determine how well classroom and teacher characteristics predict developmentally appropriate classroom practices. Teacher-reported and observational data from 69 classrooms provided support for construct validity, internal consistency, and interrater agreement of the Assessment of Practices in Early Elementary Classrooms (APEEC) measure. Hierarchical multiple regression analyses indicated that classroom characteristics (grade, class size, number of children with disabilities), teacher characteristics (education level, years of experience) and teacher beliefs (developmentally appropriate beliefs and developmentally inappropriate beliefs) accounted for 42% of the variance in observed classroom practices. With all variables in the model, teacher education, grade, and beliefs in developmentally appropriate and inappropriate practice accounted for most of the variance in observed classroom practices.     

Characterizing Mathematics Classroom Practice: Impact of Observation and Coding Choices

Large‐scale observational measures of classroom practice increasingly focus on opportunities for student participation as an indicator of instructional quality. Each observational measure necessitates making design and coding choices on how to best measure student participation. This study investigated variations of coding approaches that may be feasible in large‐scale studies, and the ramifications of these variations for drawing inferences about instructional quality. Using data from classroom videos, we found that decisions about whether to keep track of individual students in the coding, observe multiple contexts in the classroom (e.g., whole‐class and small‐group discussions), and capture nuances of student participation changed the resulting characterizations of classroom practice. Most importantly, simplifying the coding approach did not fully capture and even misrepresented the level and nature of student participation in many classrooms.     

Teaching and Learning Science Through Song: Exploring the experiences of students and teachers

This qualitative, multi-case study explored the use of science-content music for teaching and learning in six middle school science classrooms. The researcher sought to understand how teachers made use of content-rich songs for teaching science, how they impacted student engagement and learning, and what the experiences of these teachers and students suggested about using songs for middle school classroom science instruction. Data gathered included three teacher interviews, one classroom observation and a student focus-group discussion from each of six cases. The data from each unit of analysis were examined independently and then synthesized in a multi-case analysis, resulting in a number of merged findings, or assertions, about the experience. The results of this study indicated that teachers used content-rich music to enhance student understanding of concepts in science by developing content-based vocabulary, providing students with alternative examples and explanations of concepts, and as a sense-making experience to help build conceptual understanding. The use of science-content songs engaged students by providing both situational and personal interest, and provided a mnemonic device for remembering key concepts in science. The use of songs has relevance from a constructivist approach as they were used to help students build meaning; from a socio-cultural perspective in terms of student engagement; and from a cognitive viewpoint in that in these cases they helped students make connections in learning. The results of this research have implications for science teachers and the science education community in developing new instructional strategies for the middle school science classroom.     

Von der Lehrerausbildung in den Beruf – Fachbezogenes Wissen als Voraussetzung für Wahrnehmung,Interpretation und Handeln im Unterricht

The transition from teacher training into the teaching profession is seen to be key in the biography of a teacher. Yet there is a lack of enquiry into the development of the content knowledge (CK) and pedagogical content knowledge (PCK) held by teachers at the end of their education, and into the significance of this knowledge for situational and behavioural cognition. On the basis of data from the international comparative study TEDS-M 2008 and the Germany-based 2012 follow-up, this contribution investigates the knowledge and skills development of teachers during the transition from teacher training to the teaching profession in a four-year longitudinal study with n = 171 mathematics teachers. Three test formats (paper-and-pencil, video-vignettes and speed tests) are used to examine various indicators for teacher expertise. In addition to the analysis of changes in mathematical CK and PCK levels, path analysis is used to determine the extent to which this knowledge functions as a predictor for the perception and interpretation of classroom situations, the decision-making in these situations, and the quick identification of student mistakes. The results show only small changes in the group mean (for MCK there is a small loss, whereas MPCK remains constant), but high fluctuation (i. e. weak correlation) in the MPCK ranking between the two measurement occasions, pointing to differential learning effects gained from teaching practice. There are also significant differences in predictability: MCK is important for the speed indicators of teacher expertise; MCK and MPCK for the perception and interpretation of situations, and for the generation of action strategies. These findings can on the one hand help to facilitate the evaluation of beginning teachers’ need for support and, on the other hand, they indicate the necessity for the early integration of practice-based elements into teacher training programmes with the aim of developing PCK and related skills.     

构建普通高校体育与健康课堂教学发展性评价的理论研究

体育与健康课堂教学发展性评价是包括评价形势、内容、方法、组织实施在内的,将教师与学生融为一体参与评价的一种课堂教学评价体系.把发展性评价作为体育与健康课堂教学的有机组成部分,贯穿在教学计划的始终,以及时调整课堂教学策略,改善实践行动,高质量的完成课堂教学.     

Exploring tensions in developing assessment for learning

This paper is based on a study of classroom practice of primary school teachers who were engaged in a programme of professional development to implement formative assessment in their classrooms. The programme sought to develop the skills and expertise of teachers to enable formative assessment to be used to support and improve the learning of students. This study examined changes in practice in these teachers’ classrooms, their students’ learning experiences, pedagogical decision‐making, and the challenges experienced by teachers and students in developing assessment for learning. Activity theory was used as an analytical tool and enabled the identification of important contradictions in the changing system that produced tensions and difficulties but also provided driving forces for change. The development of formative assessment practices was of necessity accompanied by a culture change in the complex classroom systems. For teachers change was characterised as a process of expansive learning that was motivated by a contradiction between the teachers’ beliefs about learning and the existing culture in the classroom. The change in classroom practice was enabled by the formative assessment philosophy and a range of mediating artefacts.     

Live video classroom observation: an effective approach to reducing reactivity in collecting observational information for teacher professional development

This paper examines the significance of live video classroom observations of teaching practice to reduce reactivity (the observer effect) so as to obtain more credible observational information for teacher professional development in a secondary school in the largest city in southern China. Although much has been discussed regarding the use of remote live video classroom observation for professional development, the advantage of remote live video classroom observation in reducing reactivity has often been overlooked. Using the case study method, the research reported here focuses on two components: conducting live video classroom observation and the use of the live video classroom observation in reducing reactivity. This paper suggests that live video classroom observation is effective in reducing reactivity and helps avoid subjective judgement and increase feedback sources, thus providing a solution to compensate for the limitations of traditional classroom observations.     

Evaluation of Preservice Teachers’ Internships: A Model to Encourage Career Continuity and Program Reform

Quality teacher evaluation is a complex subject, requiring complex methods that draw from multiple data sources (Peterson, 2000). Most importantly, preservice teacher evaluation should match teacher education objectives (Popham, 1993) and, ultimately, be used to inform teacher practices and reform educational programming (Darling-Hammond, 1990). The purpose of this study is to present an evaluation model that uses multiple data sources for a preservice teacher’s internship experience. This model is employed within a teacher education program at a large land-grant university; the multiple data sources match program objectives and draw parallels between preservice and inservice teacher evaluation tools at use in this particular state. The evaluation model incorporates two guiding objectives within this college of education’s mission statement—objectives that focus on performance and reflection. First, preservice teachers are educated to assume roles of leadership and service in classroom practice, and second, preservice teachers are taught to become reflective practitioners, The first objective is measured by using a research-based classroom observation rating form during the internship that closely resembles the tool used by the state-licensing department of education. The second program objective is measured through the use of portfolios. In addition to using the results from these instruments to advise preservice teachers regarding their professional growth, the data can also guide program development within the college and suggest programmatic reform, an often overlooked yet integral factor of personnel evaluation. Discussion of specific rating results per instrument and specific avenues for program development are presented.     

Measuring instructional practice in science using classroom artifacts: lessons learned from two validation studies

With growing interest in the role of teachers as the key mediators between educational policies and outcomes, the importance of developing good measures of classroom processes has become increasingly apparent. Yet, collecting reliable and valid information about a construct as complex as instruction poses important conceptual and technical challenges. This article summarizes the results of two studies that investigated the properties of measures of instruction based on a teacher‐generated instrument (the Scoop Notebook) that combines features of portfolios and self‐report. Classroom artifacts and teacher reflections were collected from samples of middle school science classrooms and rated along 10 dimensions of science instruction derived from the National Science Education Standards; ratings based on direct classroom observations were used as comparison. The results suggest that instruments that combine artifacts and self‐reports hold promise for measuring science instruction with reliability similar to, and sizeable correlations with, measures based on classroom observation. We discuss the implications and lessons learned from this work for the conceptualization, design, and use of artifact‐based instruments for measuring instructional practice in different contexts and for different purposes. Artifact‐based instruments may illuminate features of instruction not apparent even through direct classroom observation; moreover, the process of structured collection and reflection on artifacts may have value for professional development. However, their potential value and applicability on a larger scale depends on careful consideration of the match between the instrument and the model of instruction, the intended uses of the measures, and the aspects of classroom practice most amenable to reliable scoring through artifacts. We outline a research agenda for addressing unresolved questions and advancing theoretical and practical knowledge around the measurement of instructional practice. © 2011 Wiley Periodicals, Inc. J Res Sci Teach 49: 38–67, 2012     

Teaching for Scientific Literacy? An Examination of Instructional Practices in Secondary Schools in Barbados

This study examined the extent to which the instructional practices of science teachers in Barbados are congruent with best practices for teaching for scientific literacy. Additionally, through observation of practice, it sought to determine the teachers’ demonstrated role in the classroom, their demonstration of learning through discourse, learning goals and the nature of classroom activities. Five hundred nineteen students from 12 of the 23 secondary schools on the island and 15 teachers across 8 schools participated in the study. Data were collected by means of a questionnaire, an observational schedule and field notes. It was found that while problem-solving and questioning were mainly used in the classroom, the use of experiments was among the least popular teaching strategies. Additionally, results showed that teachers’ display of the knowledge of the characteristics of scientific literacy was unsatisfactory. Generally, the findings indicate a gap between teaching for scientific literacy as expressed in the literature and current instructional practices in secondary science classrooms in Barbados.     

中小学教师数据素养评价指标体系构建

教育数据在教育领域不断渗透,教师的数据素养水平对于有效利用教育数据、合理判断学生的行为具有重要影响。本研究旨在用德尔菲法构建中小学教师数据素养评价指标体系,为评估中小学教师数据素养提供参考依据。通过文献分析梳理中小学教师数据素养评价指标,选择相关领域专家12人,采用德尔菲法进行2轮专家意见征询,根据专家打分计算指标权重,形成中小学教师数据素养评价指标体系。问卷征询专家的积极系数为100%,专家权威程度均值为0.72,确定的一、二级指标重要性均值均大于4.00,变异系数均小于0.25,符合德尔菲法要求。最终形成了包含数据意识、数据知识、数据操控技能、数据应用能力、评价交流数据、数据思维6个一级指标和19个二级指标与40条具体内容的中小学教师数据素养评价指标体系。对46位教师进行小范围试用,验证了其科学性与可靠性,该评价指标体系可作为评价中小学教师数据素养发展水平的参考测量工具。     

Beurteilereffekte bei der Messung von Unterrichtsqualität

In the external evaluation of schools the technique of classroom observation belongs to the methodological standard repertoire. Nevertheless the measurement of quality of classroom teaching based upon selected lesson sequences, which are as a rule inspected only briefly, is fraught with a lot of methodological problems. Therefore it is relevant for a substantiated quality assurance to reveal problems in the measurement of quality of classroom teaching due to an implementation of adequate empirical methods. This is made possible by using the generalizability theory and the many-facet Rasch model. Analyses based upon data of the Hamburg school inspection point out that by using an appropriate data collection procedure rater effects in classroom observations turn out comparatively low at about nine percent of total variance. Furthermore analyses prove that it is insufficient to simply quantify the agreement among raters by using global reliability measures, but that it is necessary to check up on intra rater consistency for getting valid and in this way reliable results from classroom observations for the practice.     

Data-informed curriculum reform: Which data,what purposes,and promoting and hindering factors

Schools face a lot of data on the functioning of their school which they can use to make improvements in teaching, learning and the organization. For data use to lead to improvement, it is important to further research the concept data-driven decision making. The results of this explorative study in the Netherlands show that teachers mainly use classroom level data for making instructional decisions at classroom level, and school leaders mainly use school level data for policy development decisions. This article ends with suggestions with regard to enhancing the effectiveness of data-driven decision making, for example by stressing the importance of developing teachers' competence in the use of data.