The relationship between individual instructional characteristics and the overall assessment of teaching effectiveness across different instructional contexts

There is an ongoing debate in the student evaluation of teaching literature about whether an overall rating or factorial dimensions of teaching effectiveness should be used in personnel decisions. Marsh and his colleagues have advocated the use of a weighted average approach to computing overall evaluations. A policy-capturing experiment was carried out where students in three different instructional contexts made overall evaluations of hypothetical instructors based on a manipulation of the teaching factors in Marsh's SEEQ. The results indicated (1) amount learned was consistently the most important factor affecting overall evaluations; (2) course difficulty was consistently the least important factor affecting overall evaluations; and (3) there was a strong similarity among the three groups in the relative importance of the various teaching factors in arriving at an overall evaluation. The implications of this research are discussed, as well as directions for future research.     

An Empirical Investigation Demonstrating the Multidimensional DIF Paradigm: A Cognitive Explanation for DIF

Differential Item Functioning (DIF) is traditionally used to identify different item performance patterns between intact groups, most commonly involving race or sex comparisons. This study advocates expanding the utility of DIF as a step in construct validation. Rather than grouping examinees based on cultural differences, the reference and focal groups are chosen from two extremes along a distinct cognitive dimension that is hypothesized to supplement the dominant latent trait being measured. Specifically, this study investigates DIF between proficient and non-proficient fourth- and seventh-grade writers on open-ended mathematics test items that require students to communicate about mathematics. It is suggested that the occurrence of DIF in this situation actually enhances, rather than detracts from, the construct validity of the test because, according to the National Council of Teachers of Mathematics (NCTM), mathematical communication is an important component of mathematical ability, the dominant construct being assessed. However, the presence of DIF influences the validity of inferences that can be made from test scores and suggests that two scores should be reported, one for general mathematical ability and one for mathematical communication. The fact that currently only one test score is reported, a simple composite of scores on multiple-choice and open-ended items, may lead to incorrect decisions being made about examinees.     

Increasing the impact of a Master’s programme on teacher leadership and school development by means of boundary crossing

In this case study, we investigate how the development and impact of teacher leadership through a Master’s programme can be supported by a design that encourages boundary crossing activities between schools and universities. The case study focuses on 42 experienced teachers from three vocational colleges who were promoted to senior teacher positions and participated in a Master’s programme. Through individual and focus group interviews, data were collected on how boundary crossing activities can connect professional development and school development, and on favourable conditions for effective boundary crossing. The study shows that strategic alignment and shared ownership between university and school, a collective approach with multiple participants from one school, and the use of boundary objects, created the conditions through which the Master’s programme could serve as a catalyst, stimulating innovation of work practices and development of new leadership practices. The leadership of the senior teachers initiated a development process that shifted from formally mandated forms of leadership to more culturally embedded forms of leadership. This case study leads to a new understanding of design criteria for Master’s programmes and boundary objects that can support the process of developing teacher leadership within schools by facilitating boundary crossing between university and school.     

How do instructional designers evaluate? A qualitative study of evaluation in practice

This study employed a qualitative research design to investigate how instructional designers use evaluation in everyday design practice. While past research has examined how designers spend their time, how they generally make decisions, and expert-novice differences, little attention has been paid to use of context, input, process, or product evaluation, from the perspective of practicing designers. Based on interviews of practitioners, our findings included ten themes regarding how designers use evaluation to improve their products. While these results substantiate to some degree the claim that practitioners believe clients will not pay for formal evaluations, they also suggest that practitioners use evaluation in important but less formal ways. Other conclusions regarding the role of evaluation in design are provided and future directions for training and research are discussed.