Building the field’s knowledge of teaching and learning: Centering the socio-cultural contexts of observation systems to ensure valid score interpretation

The articles in a special issue of this journal, Merits and Limitations of Researching Teaching Quality More Synergistically, grapple with the assertion that the field does not share a common language or structural decomposition of teaching and that the current range of frameworks and observation systems used by researchers jeopardizes the accumulation of knowledge in the field. We analyze these arguments from a socio-cultural perspective, theorizing that teaching and the improvement of teaching occur in socially situated contexts that give meaning to all research frameworks and measurement tools. Rather than asking whether a common framework of teaching might be useful, we ask when such a framework might be useful, when can such efforts be limiting, and why? Building on contemporary validity theory, we bring the role of context back into the current conversation. We suggest that while there are important affordances of a unified framework of teaching quality, such a framework will be unable to fully address some of the issues identified in this special issue. For practical, theoretical, empirical, and socio-cultural reasons, researchers will require multiple frameworks and associated observation systems to support the study and improvement of teaching across contexts.     

Quality lies in the eyes of the beholder: A mismatch between student evaluation and peer observation of teaching

Abstract

The study described in this article was prompted by the poor performance of students in an ‘at risk subject’ in a science faculty at a university in South Africa. Teacher performance could contribute to poor performance among students, therefore the performance of one of the science teachers whose students were performing poorly was evaluated by his students and through peer observation of his teaching. The article draws on a merger between Bernstein's ideas on framing as well as deep and surface approaches to learning to form a theoretical framework that underpinned the study. Peer observation showed that the teacher employed predominantly teacher-centred, passive approaches to teaching, and the facilitation of active learning was minimal, that is, framing was strong. Students, however, evaluated their teacher positively, indicating that he was an ‘effective’ teacher. Therefore, the perception of what constitutes ‘quality teaching’ is viewed differently by peers and students. This is most likely due to the incompatibility between peers' conception of teaching and students' conception of learning. Therefore, students' feedback on teaching is not necessarily accurate or useful.     

Evaluating instructional quality

ABSTRACT

Building on the papers in this special issue, this article uses modern conceptions of validity theory to provide a framework for considering the evaluation of teaching quality. The 3 facets of teaching quality focused on are domain conceptualization, evidence and inferences, and their evaluation. Domain definitions vary in their specificity with tradeoffs in their range of applicability and specificity of inference. Evidence collection can range from highly standardized assessments to observations that must attend to evidence from a myriad of classroom interactions. For all assessments, however, even the most standardized, different interpretations of assessment tasks can threaten the validity of score interpretations. The papers consider a range of processes that are designed to generate, support, and interrogate the validity of inferences based on assessment scores. A fundamental question underlying this type of measurement is whether differences in the quality of teaching that students experience can be causally attributed to the teacher.     

A Student Guide for Interpreting Case Material

Abstract

The use of peers in the evaluation of teaching is part of a larger trend in postsecondary education toward a more systematic assessment of classroom performance. Many scholars believe that certain aspects of teaching can be assessed only by classroom observation. This study examines the use that peer reviewers make of teaching products, especially peer observation reports, during the promotion and tenure review process. Results indicate that peer observation reports are seen as an important component in evaluating teaching effectiveness, though perhaps not the best indicator of effective teaching. Despite flaws in peer observation instruments, the results from classroom observation are seen as valid and are used in deliberations about faculty teaching performance.     

‘Left with a title but nothing else’: the challenges of embedding professional recognition schemes for teachers within higher education institutions

ABSTRACT

With increasing moves globally towards the professionalisation of teaching in Higher Education, there is growing interest in the role of accredited professional recognition schemes that provide professional development for established university teaching staff. In the UK, There are now over 120 professional recognition schemes, resulting in institutionally focused evaluation studies examining their impact. This article contributes to this emerging body of work; it draws on cross-institutional data and Foucauldian theorising to address two important questions. In what ways does engagement with an institutional professional recognition scheme impact on participants’ teaching development, and how does institutional culture influence that engagement? The data illustrate that whilst institutional culture drives engagement, it did little to promote teaching development. Across the case-study institutions, neo-liberalism agendas were apparent. Some staff felt pushed to achieve professional recognition in response to the increasing use of metrics to measure the student experience and to inform institutional standing in league tables. Whilst evidence shows the process of seeking accreditation can lead to an enhancement in teaching practices, caution must be taken to ensure that the professional development opportunities offered by accreditation schemes are fully realised.     

Trying to Get Ahead of the Curve: Raising and Understanding Current Themes in New Literacies Practices

Abstract

This article addresses the following questions: What impact does using the theoretical framework of new literacies have on understanding language, literacy, and learning practices today as technologies are constantly being developed and used? What is the state of research in this area? What are some new directions the field might take in order to develop in new ways? The conclusion suggests some possibilities for new research questions and positions in relation to new technologies.     

Evaluating the effects of faculty development: theoretical framework and empirical implementation

ABSTRACT

This article implements a systemic framework to evaluate the long-term effects of a program characterized by a SoTL approach and a hybrid learning environment. The importance of the difference between the perception of long-term effects on individual professional development on the one hand and the development of collective practices on the other, is interpreted with reference to certain shortcomings of the faculty development program’s learning design and its non-alignment with institutional practices. Based on this in-depth research, concrete ways to take account of the complexity of faculty development programs are proposed, both for research into and development of such programs and their impact at individual and institutional levels.     

Classroom observation as an instrument for school development: School principals’ perspectives on its relevance and problems

Classroom observation can have diagnostic, evaluative and counseling purposes and can be regarded as a traditional instrument for the processes that are planned for school development. However, the evaluative character of classroom observation can also be interpreted as a controlling measure and therefore, lead to stress and fear in the observed individuals. This might bring about resistance against the feedback which results from classroom observation. In order to be able to focus on the useful aspects and the benefits of classroom observation, it is important to understand which factors make the instruments of classroom observation credible and reliable for the teachers who are being observed. It is central to our current topic to understand the specific criticism concerning the area of teaching and learning within a school inspection and to work out the factors which contribute to the credibility of classroom observations on one hand, and the factors which reduce it on the other hand. For this purpose, we analyzed the statements of fifty school principals on classroom observation taken from overall interviews concerning the expectations of those principals towards school inspections in North Rhine-Westphalia (NRW). The results clearly indicate that classroom observation is a critical process for teachers. Not being able to observe the classroom credibly can have several negative effects. Therefore, this research concentrates on ways that can improve the credibility of classroom observation.     

Re-mediating knowledge infrastructures: a site for innovation in teacher education

ABSTRACT

The article outlines a framework for studying and organizing infrastructure, social and material to advance consequential knowledge. To demonstrate the utility of the framework, three examples of innovation in teacher education are presented that involve re-mediating infrastructure to imagine equity-oriented teacher learning. The first case focuses on re-mediating social resources to create new pathways for knowledge and expertise to flow between a school and a university teacher education programme. The second case considers how re-mediating social and material resources, specifically who is involved in designing the goals of a teacher education programme, expanded conceptions of the valued outcomes of teacher preparation. The third case addresses the impact of an innovation that re-mediated technical and social resources to support teachers’ professional judgment in classroom interactions. As teacher educators, we need to be open to shifting our purposes and our designs in response to changing institutional, political, and practical circumstances. This framework for organizing and re-mediating infrastructure provides a means of doing this work in principled and responsible ways.     

The classroom observation protocol for socioscientific issue-based instruction: development and implementation of a new research tool

Background: For the past decade, science educators have been exploring the use of Socio-scientific Issues (SSI) as contexts for science teaching and learning, and research indicates that doing so can support significant learning gains. However, research related to how teachers take up the practice of SSI-based instruction is far more limited, due in part to a lack of tools for use in this kind of research.

Purpose: The focus of this research is development and testing of a new classroom observation protocol specifically designed for SSI-based instructional contexts.

Design and methods: Development of this SSI-Observation Protocol (SSI-OP) took place in four distinct phases: review of existing protocols and SSI-based instruction frameworks, writing and revision of protocol items, initial testing of the draft protocol, and soliciting feedback from SSI experts.

Sample: Following the four stages of SSI-OP development, we progressed to a series of field tests. The field tests were conducted with three different samples. The first sample was an experienced (10 + years) high school biology teacher and one of her honors biology classes. The second sample consisted of seven Turkish Pre-service Science Teachers (PST) participating in a science methods course. The third sample included two Thai PST from a field experience course embedded within a teacher education program.

Results: The final version of the protocol addressed five dimensions of SSI-based instructional activities: focus of instruction, teaching moves, role of teacher, role of students, and classroom environment.

Conclusions: The SSI-OP could be used in a variety of ways for research including documentation of current practices, impacts of professional development and/or curricula on teaching practices, and changes in teaching over time. We offer the SSI-OP as a new tool with the potential to contribute to science teacher education and research that may advance the teaching and learning of science through SSI.     

Queering Constructs: Proposing a Dynamic Gender and Sexuality Model

Abstract

Higher education educators commonly understand social identities, including gender, to be fluid and dynamic. Lev's (2004) model of four components of sexual identity is commonly used to demonstrate the fluidity of sex, gender, and sexuality for individuals, but it does little to address the fixedness of those constructs. Through a multipronged intersectional framework and by centering trans* -students, this article proposes a more dynamic model for gender and sexuality.     

The Management of Dilemmas in School Restructuring: A case analysis

While there is growing interest in studying principals' perceptions of their work lives in terms of dilemmas, relatively few studies have gone beyond this to investigate how leaders manage and cope with such 'intractable' situations and the consequential effects and outcomes. Accordingly, this article provides an in-depth qualitative case study of the dilemmas faced by a principal who is involved in the restructuring of his school. It then analyses the ways in which he manages and copes with these intractable situations, and the effects and outcomes that result. The article begins by outlining a framework used in the analysis. It addresses some considerations of method, before describing relevant school and system contexts. Finally, the in-depth case study is presented using the structure associated with the framework described earlier. Among the key findings are that dilemmas present opportunities as well as challenges for visionary, proactive and creative school leaders.     

Establishing a crosswalk between the Common European Framework for Languages (CEFR) and writing domains scored by automated essay scoring

ABSTRACT

This article employs the Common European Framework Reference for Language Acquisition (CEFR) as a basis for evaluating writing in the context of machine scoring. The CEFR was designed as a framework for evaluating proficiency levels of speaking for the 49 languages comprising the European Union. The intent was to impact language instruction so that “mastery” of one language has the same meaning as it does in another. A second objective is to provide a crosswalk for what one automated writing evaluation (AWE) system does in attending to the dimensions of the framework. The CEFR Framework is divided into five traits and different proficiency levels. The question then becomes: Does the AWE system attempt to measure these dimensions of writing? And, if so, how is this operationalized? Is it measuring aspects of communication that are not specified? The goal here is to create a common vocabulary between the writing community and those interested in AWE systems as to what is actually being measured by their software, and mapping that to a developmental scale of writing performance.     

内容依托教学评估问题刍议

把评估贯穿于内容依托教学的整个过程,在内容依托教学中举足轻重。阐述评估过程中的一系列问题——评估内容、评估目的与类型、互动的评估方式、评估原则以及评估过程参与者,以期在内容依托课程的框架下为评估学生表现提供可行方法与建设性的指导原则,同时提供了内容依托教学理论操作性强的范例。     

Pastoral Care for Senior Students: An Analysis of a Voluntary Staff Counselling Scheme in a Sixth-form College

This article is written for new and experienced tutors to review the ways in which they are supporting their students’ learning. It considers the purposes of tutoring, drawing on the voices of students themselves. The article then considers the ways in which the tutor can support the learning of young people, including academic tutoring. It concludes with some guidance for the tutor to develop their own learning about tutoring.     

Knowledge outside the box – sustainable development education in Swedish schools

Background: Sustainable development, as an area of knowledge, appears in several different places in the curriculum and does not fit neatly within the scope of traditional subject areas. In many countries, including Sweden, it has long been upheld as an important tool for increasing understanding of, and dealing with, environmental problems. It is not clear, however, what role education can actually have in the making of a more sustainable future. Even though there are several potential ways for sustainable development to be involved in education, the concept raises many questions when transferred to the school context.

Purpose: This paper investigates how teachers deal with the difficulty of defining and approaching sustainable development as an area of knowledge in Swedish schools.

Sample: This article is based on semi-structured interviews with 40 teachers, 13 of whom were lower secondary school teachers (pupil age 12–15) and 27 were upper secondary school teachers (pupil age 15–18). The study involves teachers in all subjects where sustainable development is a goal in the syllabus. The study is also based on participant observation in one upper secondary class. A total of 17 different schools were involved, from a wide range of locations in Sweden.

Design and methods: The paper builds on qualitative data and the analysis of transcribed interviews and group interviews with teachers in Swedish lower and upper secondary schools. Group interviews, involving three or more people, were conducted on eight occasions. The pupils at an upper secondary school were also observed while they were working on a course called ‘policy and sustainable development’. Data were transcribed and analysed thematically.

Findings: The analysis suggests that, according to the teachers’ experiences, the demands of equivalence and measurability in school have increased and that this affects how sustainable development is approached in teaching and learning. Three main categories of knowledge were identified. The study also presents two representations that model how teachers may approach knowledge about sustainable development – metaphorically termed ‘the Accountant’ and ‘the Adventurer’ – and their different effects on knowledge.

Conclusions: There is a tendency for complex knowledge areas such as sustainable development, which do not fit seamlessly into traditional curriculum subjects, to become oversimplified when translated into teaching situations. According to the representations that we described metaphorically, the teacher, as an accountant, is characterised by ‘knowledge instrumentalism’, which means that teachers administer knowledge and the pupils consume it. In this transactional model, the accountant is also very dependent on external governance and control. Alternatively, the teacher, as an adventurer, is characterised by authority, knowledge and self-control. In this model, knowledge sometimes grows in an unpredictable way in the meeting between people who share common experiences. For adventurers, sustainable development is a matter of commitment and awareness, and it involves an explicit stance. The metaphors can be placed on a continuum which describes how teachers manage the demands of the school system in relation to the knowledge area of sustainable development.     

Measuring faculty learning in curriculum and teaching competence online courses

Online education is used for a variety of purposes in higher education. Two such purposes are improving one's performance over time and understanding one's professional development in the context of online teaching and learning. Relying on data from online staff development courses delivered in five Spanish universities, this article explores online faculty learning through the lens of staff development theory. This theoretical perspective emphasizes the universities' quality assurance contexts and offers an empirical examination of the ways in which faculty members learn curriculum and teaching competencies (CTCs) in online staff development programmes. At the core of this analysis is the contention that faculty members understand and respond to quality teaching lessons and activities. Finally, this study highlights the points deemed important when designing, implementing, and evaluating Internet CTC training courses.     

A competence-based constructivist tool for evaluation

Abstract

Recently the educational system in general, and the university in particular, have suffered a transformation in the way learning, teaching, and subsequently, evaluation are considered due to changes in the European context, including the Bologna Declaration and the constructivist view of learning. There are several adjustments required to implement successfully traditional university teaching practices, such as the use of new technologies in teaching, new ways of tutoring, different groupings, the teaching of competencies and social skills, and, as a result, new forms of evaluation, which constitute the object of this paper. Thus, we show an objective alternative —or complement—to traditional evaluation: a very exhaustive tool for evaluating oral and written presentations which includes four sections: general aspects, contents, questions concerning design and layout, and psychological aspects concerning general competencies. Throughout the paper, the constructivist notions of learning, teaching and evaluation are substantiated and validated with the use of this tool for group or individual presentations or end-of-degree dissertations.     

Complexity Conceptual Perspectives for Research About Educational Complex Systems

Abstract

This article is premised on the observation that in educational research there is an increasing use of terminology and concepts related to the field of complexity and the study of complex physical, biological, and social systems. As we are at an early stage in this cross-fertilization of complex systems ideas and methods in educational research, this article provides an overview of key complex systems conceptual perspectives and research methods in conjunction with a discussion of why educational researchers should consider their use. It is hoped that this overview of complex systems concepts and nomenclature used to describe research that views education as a complex system will help promote a broader awareness and acceptance of this work in the educational research community.     

Toward a Direct Realist Account of Observation

The concept of observation is central to science, but there are several ways that this can be understood. My aim is to criticize the account of observation presented by Alan Chalmers in What is This Thing Called Science? and provide an alternative based on direct realist approaches to perception. This issue, the nature and objectivity of perceptual knowledge, is one of the oldest in philosophy. Chalmers adopts a simple form of indirect realism, according to which the real objects of perception are our experiences or sensations. I believe this is a misleading way to understand observation in science. There are many important philosophical issues that relate to observation, such as: How are observations justified? and What do we observe?, however space does not permit extensive discussion of these topics.