What is successful adaptation of lesson study in the US?

This paper features a case study of one US K-8 school district pioneering the use of “lesson study,” a teacher professional development approach adapted from Japan. The case explores events that occurred in the district over more than 4 years (Spring 2000–Fall 2004) as lesson study spread nationally and within the district. We document four categories of changes that occurred in the district’s lesson study approach, and describe some of the early consequences of these changes as well as conditions that enabled the changes to occur. We argue that this case illustrates much of what we would hope to see in a maturing lesson study effort, and conclude that other US sites may need to go through similar changes, organize similar supports, and persist in their learning about lesson study to successfully adapt this model to their local contexts.     

Observer ratings of instructional quality: Do they fulfill what they promise?

Despite considerable interest in the topic of instructional quality in research as well as practice, little is known about the quality of its assessment. Using generalizability analysis as well as content analysis, the present study investigates how reliably and validly instructional quality is measured by observer ratings. Twelve trained raters judged 57 videotaped lesson sequences with regard to aspects of domain-independent instructional quality. Additionally, 3 of these sequences were judged by 390 untrained raters (i.e., student teachers and teachers). Depending on scale level and dimension, 16–44% of the variance in ratings could be attributed to instructional quality, whereas rater bias accounted for 12–40% of the variance. Although the trained raters referred more often to aspects considered essential for instructional quality, this was not reflected in the reliability of their ratings. The results indicate that observer ratings should be treated in a more differentiated manner in the future.     

What is instructional strategy? Seeking hidden dimensions

Educational technology research and development - Instructional strategy is defined in abstract terms independent of instructional theories, philosophies, or standard formulas. Strategies share...     

How valid are student perceptions of teaching quality across education systems?

In an international context, teaching quality is often assessed via student surveys. Yet, there is little empirical evidence for the validity of students' perceptions of teaching quality across education systems. Following an argumentative validation approach, we evaluate the reliability, stability, and international comparability of students' perceptions of teaching quality as well as their predictivity on student outcomes. Based on longitudinal data from six education systems our main findings include that student perceptions of teaching quality are 1) reliable on the class level, 2) stable between two measurements, and 3) metric invariant across education systems, yet 4) their effects on achievement, interest, and self-efficacy vary between the aspects of teaching quality and education systems. Our study provides first substantial empirical findings for the validity of student perceptions across education systems and points towards culture-specific differences in the effectiveness of teaching.     

Evaluating instructional programs: How much is enough?

Instructional developers need to determine the degree of economically justifiable rigor for evaluating instructional products. Rigorous evaluation, involving outside evaluators, random assignment of masses of individuals from well-defined populations, and massive amounts of data, is often beyond their means. At the opposite extreme, representing no evaluation at all desite the size of consumer market, are most standard educational materials. A successive approximations scheme to empirical determination of effectiveness is needed, with consumers aware of the risks involved. The fear to “teaching to the test” (recently termed “wholly indefensible”) conflics with the basic assumptions of the teachonology of instruction and with the achievement of accountability of the instructional system to its consumers.     

One size fits all? The limitations of standardised assessment in critical thinking

In today’s market-driven educational culture, universities are coming under increasing pressure to justify funding through the disclosure of measurable outcomes in education and research. One educational objective that receives particular attention is critical thinking, regarded as an essential skill in both academic and work environments. The assessment of critical thinking has become a significant enterprise, with a number of standardised tests available for both individuals and organisations. While these tests are based on well-known taxonomies of critical thinking, this paper argues that institutions should be wary of using them as a means to measure educational outcomes. First, they fail to take into account fundamentally contested issues within conceptions of critical thinking. They also have significant weaknesses in terms of validity and reliability. Finally, and most importantly, they provide only a limited assessment of critical thinking, failing to evaluate the skills exercised in real-life academic tasks. A more effective approach to critical thinking testing would be one implemented at a faculty level, with assessments carried out on coursework integral to the curricula of specific academic disciplines.     

What is an expert instructional designer? Evidence of expert performance during formative evaluation

Two instructional designers were provided with formative evaluation data from the sources recommended in the literature and were asked to revise a piece of instructional text while thinking aloud. Their verbal protocols were recorded, transcribed, segmented and coded, using a coding scheme derived from Newell and Simon's (1972) problem-solving model. Relevant codes were analyzed for evidence of the characteristics of expert performers identified by Glaser and Chi (1988) and Shanteau (1992). Results were unclear when the number of coded segments in each category was counted, but examination of the content of the segments showed clearly that one of the research participants demonstrated more attributes of expertise than the other, based on the identified characteristics.This paper was originally presented at the Annual Meeting of the American Educational Research Association, Chicago, IL, March 1997.The author acknowledges, with gratitude, the helpful comments made by Cynthia Weston, Wayne Nelson, and three reviewers on earlier drafts of this paper.This research was supported in part by grants from the Social Sciences and Humanities Research Council of Canada (SSHRC) and Fonds pour la formation des chercheurs et l'aide a la recherche (FCAR).     

At a first glance – How reliable and valid is the thin slices technique to assess instructional quality?

Various approaches of assessing instructional quality have emerged in educational research. In this article, we present two studies that apply the thin slices procedure, investigating the reliability and validity of the ratings of three dimensions of instructional quality based solely on the first impressions of untrained social observers. Thirty undergraduate students rated 30-s clips from English lessons (Study 1) and Math lessons (Study 2) regarding three quality dimensions. The findings suggest high reliability in these ratings. Multilevel confirmatory analyses suggested construct validity in terms of differentiation between the three dimensions of instructional quality. Finally, we found some overlap between the thin slices ratings of classroom management and constructive support with ratings of trained raters based on observations of full lessons, as well as students’ ratings of these dimensions. We discuss these results with respect to the potential of first impressions of untrained observers to measure instructional quality.     

Professional knowledge or motivation? Investigating the role of teachers’ expertise on the quality of technology-enhanced lesson plans

In an expertise study with 94 mathematics teachers varying in their relative teacher expertise (i.e., student teachers, trainee teachers, in-service teachers), we examined effects of teachers' professional knowledge and motivational beliefs on their ability to integrate technology within a lesson plan scenario. Therefore, we assessed teachers' professional knowledge (i.e., content knowledge, pedagogical content knowledge, technological knowledge), and their motivational beliefs (i.e., self-efficacy, utility-value). Furthermore, teachers were asked to develop a lesson plan for introducing the Pythagorean theorem to secondary students. Lesson plans by advanced teachers (i.e., trainee teachers, in-service teachers) comprised higher levels of instructional quality and technology exploitation than the ones of novice teachers (i.e., pre-service teachers). The effect of expertise was mediated by teachers' perceived utility-value of educational technology, but not by their professional knowledge. These findings suggest that teachers’ motivational beliefs play a crucial role for effectively applying technology in mathematics instruction.     

Uncovering everyday dynamics in students’ perceptions of instructional quality with experience sampling

Within-student dynamics in perceptions of instructional quality have been neglected, although student states constitute a major share of these perceptions. The present study examined the structure and correlates of student state perceptions of the three basic dimensions, teacher support, cognitive activation, and classroom management. We conducted a three-week experience sampling study using state measures in four subjects (observations: n_mathematics = 2,681, n_physics = 1,555, n_German = 2,026, n_English = 1,835) and analyzed data from 372 German secondary school students (M_age = 15.3 years), conducting two-level confirmatory factor analyses. Against more parsimonious solutions, the postulated three-factor structure was confirmed within- and between-students across subjects, entailing 51% within-student variance on average. Similar to trait-like perceptions, state perceptions were positively related to grades and academic interest. Our results support the factorial and convergent validity of state student perceptions of instructional quality, expanding upon between-person-based literature and uncovering opportunities to enhance teaching effectiveness.     

How effective is instructional support for learning with computer simulations?

The study examined the effects of two different instructional interventions as support for scientific discovery learning using computer simulations. In two well-known categories of difficulty, data interpretation and self-regulation, instructional interventions for learning with computer simulations on the topic “ecosystem water” were developed and tested using a sample of 124 eighth graders in science classes. The results demonstrate the effectiveness of instructional support for domain-specific factual, conceptual, and procedural knowledge acquisition. Students who received either only instructional support for data interpretation or only for self-regulation achieved the highest learning outcomes. However, a combination of instructional support for data interpretation and self-regulation seemed detrimental for knowledge acquisition. Students who received instructional interventions for both data interpretation and self-regulation also showed the highest values of perceived cognitive load. High cognitive load could be a reason for why a combination of particular instructional interventions does not lead to the expected positive learning outcomes.     

One size fits all? A different perspective on university rankings

In roughly a decade, university rankings gained the foreground in the policy arena for higher education and their influence is not going to decrease. However, several methodological shortcomings and warnings about the unintended consequences for national higher education systems have been raised. Against this background, this paper stresses that the individual recipients of information contained in university rankings are currently overlooked. Indeed, university rankings are addressed to a generic recipient, but actually, there are multiple audiences for rankings, and each of these audiences has different needs and each one attributes a different value to information attached to rankings. Referring to a theoretical tool borrowed from bioethics, this paper highlights that the ranking game involves a variety of recipients and that the current setting of the ranking panorama leaves room for gaps to emerge.     

Obtaining secondary students’ perceptions of instructional quality: Two-level structure and measurement invariance

Students' perceptions of instructional quality have become an important information source for teachers' professional development. This requires knowledge of the structure of these perceptions, their validity, and generalizability. To this end, we conducted a study with 15,005 German 512 grade students from 690 classrooms in three different school types and three different grade levels. Assuming three basic dimensions of instructional quality with 7 facets, we investigated the factorial structure of students' perceptions with two-level confirmatory factor analyses as well as their generalizability with two-level measurement invariance analyses. Our results confirmed the postulated factorial structure and strict invariance across subject groups, school types, and grade levels. We confirmed the same structure in teachers' assessments of their instructional quality that were positively correlated with the students' assessments. As such, these findings shed light on the structure, validity, and generalizability of students’ perceptions of instructional quality.     

Does what you get depend on who you are with? Effects of student composition on teaching quality

This study focuses on the association between classroom composition characteristics and teaching quality, with the aim of understanding the role of teaching quality as a mechanism underlying compositional effects. We examine how teaching quality is associated with the student body that is taught. Multilevel analyses of a sample of 1,070 3rd graders have indicated that cognitive and motivational composition of the classroom had an association with the classroom management dimension of teaching quality, whereas sociocultural composition was not associated to any of the teaching quality dimensions rated by the students. Correlations at the classroom level have also pointed at a relationship between the teachers’ and observers’ ratings of classroom management and the composition of general cognitive abilities and interest. We suggest that these findings also have implications for the theoretical conceptualization and the empirical assessment of teaching quality.     

Implementation of anti-bullying lessons in primary classrooms: how important is head teacher support?

Background: Evidence suggests that the KiVa anti-bullying programme may contribute to a reduction in bullying and victimisation, especially in primary school level. What is more, the level of implementation moderates the programme effects: the more the programme was implemented, the more bullying was reduced.Purpose: Consequently, it is of interest to investigate the factors affecting the implementation of the KiVa anti-bullying programme. We analysed whether teachers’ perceptions of support from the head teacher predicted the implementation adherence of the programme.Sample: We used a randomised sample of 93 Grade 1–3 (students’ age 7–9 years) teachers in 27 Finnish primary schools.Design and methods: Online surveys on perceptions of two types of head teacher support: (1) general support as a superior and (2) programme support for the KiVa anti-bullying programme were administered to the teachers; written observational reports on implementation adherence (frequency, contents and duration) were also utilised. Data was collected during the academic year 2008–2009. Effects of head teacher support on implementation adherence were modelled both at the teacher level and at the school level.Results: On average, implementation adherence of the student lessons was good. Teacher’s experience of programme support from the head teacher was positively related to implementation adherence, whereas, surprisingly, teacher’s experience of general support did not enhance implementation and was even negatively related to it.Conclusions: The school-based prevention and intervention programmes are delivered through social systems and individuals, and they inevitably trigger multilevel dynamics. Implementation is not only the responsibility of individual teachers but it happens in the context of the school community, the head teacher and the local-level resources and policies.     

Why is quality in higher education not achieved? The view of academics

Quality assurance is currently an established activity in Europe, driven either by national quality assurance agencies or by institutions themselves. However, whether quality assurance is perceived as actually being capable of promoting quality is still a question open to discussion. Based on three different views on quality derived from the literature – quality as culture, as compliance and as consistency – the current article sheds light on this issue by identifying the main obstacles to quality as perceived by Portuguese academics. The analysis reveals that academics tend to see those obstacles as essentially related to the view of quality as culture and especially with its structural component, including the design and functioning of institutional governance and management. Although this may indeed correspond to the Portuguese higher education institutions’ reality, one must not forget that academics tend to have a poor knowledge about the structural frames that are embedded in governance and management. By highlighting obstacles related to different views on quality, this study may serve as a base to the development of more effective quality assurance mechanisms.