Similarity of student ratings across instructors,courses, and time

This study raised three questions about the similarity or generalizability of student ratings of courses and instructors. First, how stable are student ratings of the same instructor giving the same course during twodifferent semesters? Second, how similar are student ratings of the same instructor in twodifferent courses? Third, how similar are student ratings of a given course being taught bydifferent instructors? Instances were identified in which student ratings on seven different factors were available for pairs of courses for each of these questions. For the case of the same instructor — same course—different semesters, student ratings were reasonably similar (median r for seven factors about 0.70). For the case of the same instructor—different courses, the median r was surprisingly low — about 0.40. For the case of the same course—different instructors, substantial correlations were obtained for some factors and insignificant correlations for other factors. Implications of these findings for practical use of student ratings and suggestions for further research in the area are discussed.     

Effects of grading practices and time of rating on student ratings of faculty performance and student learning

The effects of grading practices (strict, lenient) and time of rating (after a lecture, after taking an examination, after receiving feedback on performance on the examination) on student ratings of faculty performance and student learning (i.e., performance on an examination) were assessed in two experimental studies. Results indicated that (1) student ratings were directly affected by grading practices and this effect was a function of the time at which ratings were cornpleted, and (2) student learning and students' ratings of faculty were positively correlated, and grading practices had no effect on student learning. Implications of these findings for the administrative use of student ratings are considered.     

Predicting student ratings: the relationship between actual student ratings and instructors’ predictions

This study examined the relationship between student ratings and instructors’ predictions of these ratings, taking into account other instructor, student, and course characteristics. Participants in the study were 198 instructors in the School of Education at a major teacher training college in Israel. Data corresponding to one randomly selected course per instructor were collected using student and instructor questionnaires and college records. Results indicate a systematic positive relationship between instructors’ predictions and actual student ratings with respect to overall ratings and the ratings of three dimensions of teaching. Results also demonstrate a systematic trend whereby low‐rated instructors tend to overestimate their student ratings, high‐rated instructors underestimated ratings, and moderately rated instructors gave accurate predictions. Results have implications for using predictions to motivate teaching improvement.     

University instructors' ratings profiles: Stability over time,and disciplinary differences

This study investigated university instructors' teaching profiles (rank order of teaching behaviors) as they emerge from student ratings. The data consisted of student ratings of all faculty members in the physics and chemistry departments at Tel Aviv University, with no exception, who taught the same undergraduate course over two consecutive years. The ratings of these instructors were taken twice a semester (at mid- and end of semester), during the two academic years of the study, using a short rating instrument. A (4 × 5) × 2 ANOVA was conducted, with time of measurement and item serving as repeated measures, and department as a grouping factor. The same procedure was used for identifying differences between pairs of measurements for each item. These pairs compared the corresponding measures of the first year versus the second year, and separately those of mid-semester versus the end of semester. Findings indicate a very high degree of stability of instructors' profiles across time of measurement and activities to improve instructor performance on weak points; no clear-cut improvement from mid- to end-of-semester ratings except for teachers who undertake special improvement activities; and differences in teaching profiles for the two academic departments.     

The role of scales in student ratings

Questions used in course evaluations should mainly measure the quality of teaching, and students' answers to those questions should not be influenced by other factors. This paper investigates how seemingly neutral rating scales and multiple-choice questions might have an impact on the results of such evaluations. In several studies, it has been shown that the way in which a scale is constructed may strongly influence the answers it elicits in surveys and tests. Whether and to what extent common course-related ratings of students are also affected by the kinds of scales used is the main topic of this paper. Four studies examined the influence of scale polarity (unipolar vs. bipolar scales), the role of different ranges in time scales, and the impact of ordering choices in a certain way. Participants' ratings and answers were strongly influenced by all these manipulations. It is recommended to pay special attention to the role of scales when constructing questionnaires for course evaluations and when interpreting course evaluation reports.     

Validity of student self-reported ratings of learning

This study examined the relationship between individual student self-reported ratings of progress on relevant learning objectives and performance on exams administered during a college course. Across three sections of the same course taught by a single instructor, 188 students rated themselves at the end of the course on two objectives identified by the instructor as either essential or important. They also rated themselves on 10 other objectives the instructor identified as having minor or no importance. Self-ratings on course-relevant objectives correlated significantly and positively with four out of five exams and the course total, whereas ratings on irrelevant objectives did not. Students who rated their progress as either exceptional or substantial generally performed better on course examinations than those who rated their progress as moderate or less. These findings support the validity of student self-reported ratings of learning.     

Reconceptualizing student ratings of teaching to support quality discourse on student learning: a systems perspective

Higher Education - This paper summarizes the discourse on student ratings of teaching in higher education. It reconceptualizes student ratings within a larger process of promoting quality in...     

A two-dimensional analysis of student ratings of instruction

Student ratings of instruction were analyzed in terms of two global factors. One factor, which includes items on advanced planning, presentation clarity, and increased student knowledge, was named pedagogical skill. The other factor taps information about class discussion, grading, and the availability of help and was named rapport. Ratings on the skill factor did not covary with class size or the leniency of the instructor's grading but did correlate with a reasonable external criterion of student learning. Ratings of rapport correlated inversely with class size and directly with average class grade and showed only a weak relationship to the external criterion of student learning. The skill factor showed more interclass stability than the rapport factor. Previous research studies which have examined the reliability and validity of instructional ratings and their relationship to student grades and class size have reported inconsistent findings. These inconsistencies appear to result from an inappropriate unidimensional analysis of ratings which should be examined in terms of two of more separate attitude dimensions.     

The quality of student ratings of teacher behaviour

In a research project into the effectiveness of mathematics teaching in the first year of secondary education, external observers and students rated teachers' behaviour. The reliability and validity of both methods were established. The results show that teacher behaviour is assessed well when student ratings are aggregated at the classroom level. The quality of aggregated student ratings is as good as the quality of data from external observers. The predictive validity of aggregated student ratings is higher than the predictive validity of external observations when subject motivation is taken as a dependent variable.     

Teacher ratings of student engagement with educational software: An exploratory study

The quality of students' learning engagement may significantly influence their learning. Can teachers accurately judge student learning engagement with educational software? In this exploratory study, 3 fifth-grade teachers used a seven-level taxonomy to rate the frequency of different forms of engagement among 42 students interacting with different types of educational software. Teachers spontaneously treated the seven levels of engagement as a continuum, rating students highest on one level or a set of contiguous levels. Teachers generally agreed when ranking students by their typical levels of engagement, but disagreed regarding the actual frequencies of different engagement types. Ratings of software engagement conceived of as interpretive activity were correlated significantly with student reading test scores. Given the authentic classroom conditions in which this study took place, the results are promising for the classroom utility of the seven-level conception of student engagement with software.     

Effect of survey size on student ratings of teaching

Questionnaires filled in by students have become increasingly important due to the part they play in decisions about the tenure, promotion and merit pay of faculty. Since it is rare to get a complete return, it is important to determine what reliance to place on various completion rates. Using data from the faculty of engineering in one university the authors conclude that if class size is 30 or more, a 50% response rate gives an acceptable indication of rating. In a small class, less than 30, something like an 80% return is needed if the same degree of confidence is to be achieved.     

The relationship between psychological identification with instructors and student ratings of college courses

Drawing upon Freud's seminal discussion of psychological identification vis-a-vis group psychodynamics, this study investigated the relationship between degree of student identification with an instructor and students' ratings of instructional effectiveness. Specifically, it was hypothesized that students who highly identify with their instructor will rate the course more favorably, perceive greater progress on their part toward course related objectives, and report higher motivation to learn than students who do not identify with their instructor. Using a 50 item personality trait Q-sample, 96 undergraduates from eight different courses taught by three different instructors were asked to describe the instructor, their actual self, and their ideal self; Q-sorts for the instructor and ideal self were correlated separately for each student as the measure of identification. Instructional effectiveness was assessed using the Kansas State IDEA form. Correlational and path analyses of the data provided support for the hypothesis. Results are discussed in light of the unique contribution made by the traditional notion of identification to instructional outcomes and evaluation.     

Investigating substantive and consequential validity of student ratings of instruction

This study aimed at exploring faculty and student perspectives on student evaluations, as well as identifying their perceptions of the usefulness and appropriateness of the ratings for evaluating teaching effectiveness. More specifically, the study aimed at identifying the consequences, both intended and unintended, of using the evaluations, in addition to better understanding the process students used in responding to evaluations and what use faculty members made of them. Two surveys were developed and placed on the website of the Office of Institutional Research and Assessment. Emails were sent to all students and faculty participating in evaluations soliciting their cooperation and requesting their input. Faculty and student perceptions were compared qualitatively and statistically. Results revealed that students and faculty believe in the effectiveness and usefulness of the system with the need to overcome some negative consequences and biases inherent in its application at the University. Recommendations for system improvement are provided.     

Teaching effectiveness as measured by student ratings and instructor self-evaluation

The present study is designed to evaluate teaching effectiveness based on effective teacher behaviors. Operationally, it is measured by assessing levels of agreement between perceptions of instructors and students on reported employment rates and rated ability for specific instructional behavior attributes. Individually-based and organizationally-based factors are incorporated in the study for a possible contribution to teaching effectiveness variance. Findings indicate a lack of agreement between perceptions on abstract instructional attributes but that agreement exists for attributes of a concrete nature. Analysis suggests that individually-based and organizationally-based factors do not contribute to agreement variance.     

The impact of individual consultation on student ratings of teaching

Abstract

The present study addressed the impact of individual consultation on teaching improvement as measured by changes in student ratings. Subjects included 91 professors who presented naturally for individual consultation services over a seven‐year period at the teaching centre of a Canadian university. Interventions by the consultant fell into three categories: 1) Feedback‐Consultation, 2) Feedback‐Consultation‐Class Observation, and 3) Feedback‐Consultation‐Class Observation and Student Consultation. End of term student ratings for the course that was the subject of the consultation were compared with student ratings for the same course taught between one and three years prior to the consultation service, and for the same course taught between one to three years following consultation. The results showed that, overall, consultation was effective in improving the quality of the consultees’ teaching, as evidenced by an increase in mean student ratings of instruction. This effect persisted post consultation. Not all intervention groups, however, showed the same pattern of results. Change was evident immediately after the intervention except in the case of brief consultation, although follow‐up data showed improved teaching for the latter group. Control data provided evidence that the change in student ratings post consultation could reasonably be attributed to consultation effects.     

Kinds of Keys of student ratings of faculty teaching effectiveness

Three substudies of effects of different formats on student ratings of faculty teaching effectiveness were conducted. One substudy, Kinds of Keys, investigated three main kinds of keys: Agreement, Evaluation, and Needs Improvement. The second, NO TUP, investigated numbers of positive rating categories. The third, Item Wording Direction, investigated the same items worded positively, negatively, and neutrally, respectively. Practically important differences in level of ratings obtained in Kinds of Keys and practically and statistically significant differences obtained in NO TUP and Item Wording Direction. It was concluded that additional research is necessary to determine if apparent differences in teaching effectiveness are actually differences in teaching effectiveness or differences due to the methods of measurement.     

How should we use student ratings to evaluate teaching?

The proper use of student ratings to evaluate teaching is the subject matter of this paper. In particular, two divergent views are contrasted: the multidimensional view of evaluation and the unidimensional. Marsh (Int. J. Ed. Res., 1987, v.11, pp. 253–388) is the most outspoken advocate of the multidimensional view of evaluation based on factor scores. Because we disagree over such an important use of ratings, I explain the position I have adopted, summarize Marsh's most recent position, and respond to his concerns.An earlier version of this paper was presented at the annual meeting of the American Educational Research Association, New Orleans, LA, April, 1988.