Course ratings as measures of instructional effectiveness

This study assesses the validity of instructor ratings as measures of instructional effectiveness. Multiple regression analysis was used to assess the relationship between ratings of classroom practices, instructor characteristics and student commitment to the course and an overall rating of the instructor. Sixty-four percent of the variance in instructor ratings could be explained by variance in students' ratings for eleven course-related items. It was concluded that variance in instructor ratings reflected variance in classroom and instructional practices.     

Predicting student evaluations of teaching using decision tree analysis

Abstract

This study uses decision tree analysis to determine the most important variables that predict high overall teaching and course scores on a student evaluation of teaching (SET) instrument at a large public research university in the United States. Decision tree analysis is a more robust and intuitive approach for analysing and interpreting SET scores compared to more common parametric statistical approaches. Variables in this analysis included individual items on the SET instrument, self-reported student characteristics, course characteristics and instructor characteristics. The results show that items on the SET instrument that most directly address fundamental issues of teaching and learning, such as helping the student to better understand the course material, are most predictive of high overall teaching and course scores. SET items less directly related to student learning, such as those related to course grading policies, have little importance in predicting high overall teaching and course scores. Variables irrelevant to the construct, such as an instructor’s gender and race/ethnicity, were not predictive of high overall teaching and course scores. These findings provide evidence of criterion and discriminant validity, and show that high SET scores do not reflect student biases against an instructor’s gender or race/ethnicity.     

Faculty Perspectives on Course and Teacher Evaluations

Student ratings of instruction have been the subject of numerous studies with much of the research focusing on the validity and reliability of the ratings themselves. Comparatively little empirical investigation has been devoted to the perceptions of the individuals who are the subjects of the ratings, that is, the faculty. The current study explored faculty perspectives on the usefulness of student ratings for formative and summative purposes, and the actual use of student ratings for summative purposes. Contrary to what might have been deduced from the anecdotal literature, the results of this study do not portray a great deal of resistance to student ratings in general or to their use for formative and summative evaluation. It was also found that student ratings are actually being used for the latter purpose. The usefulness of the student feedback was viewed differentially by the faculty, with feedback on their interaction with students seen as most useful, followed by feedback on their grading practices, global ratings of instructor and course, and finally structural issues of the course.     

Assessing faculty performance using student evaluations of teaching in an uncontrolled setting

This paper provides new evidence on the disparity between student evaluation of teaching (SET) ratings when evaluations are conducted online versus in‐class. Using a multiple regression analysis, we show that after controlling for many of the class and student characteristics not under the direct control of the instructor, average SET ratings from evaluations conducted online are significantly lower than average SET ratings conducted in‐class. Further, we demonstrate the importance of controlling for the factors not under the instructor’s control when using SET ratings to evaluate faculty performance in the classroom. We do not suggest that moving to online evaluation is overly problematic, only that it is difficult to compare evaluations done online with evaluations done in‐class. While we do not suppose that one method is ‘more accurate’ than another, we do believe that institutions would benefit from either moving all evaluations online or by continuing to do all evaluations in‐class.     

Using a single score for summative teacher evaluation by students

In spite of the hundreds of studies done on teacher evaluation by student ratings, there are still several major controversial issues of how to construct evaluation instruments that best serve their purpose. The present study contributes to one of the ongoing debates of recent years that addresses the number of teaching dimensions to be considered in decision making, namely, whether to use a single score or multiple scores for teacher evaluation. The study demonstrates using a short form on which the global score overall teaching performance can almost perfectly predict the mean of all teacher-attribute items. The questionnaire used in this study was administered eight times—twice a semester for two years—for all faculty members and TAs in the departments of physics and chemistry at Tel Aviv University. The composition of teacher attribute items was different for the faculty members and TAs, reflecting their different teaching functions—lectures versus recitation problem solving. Results show that while for the faculty the global score can simply replace the mean of all instructor-attribute items and serve as a single score that faithfully represents all dimensions of teacher ratings, for the TAs, a linear transformation is needed.     

Following student ratings over time with a catalog-based system

First, the longitudinal nature of student ratings of instructors has not received deserved emphasis from researchers. Second, the use of item banks for designing student rating questionnaires, especially for instructor feedback, has needed attention. These two factors are investigated in this study, which tracks 304 instructors over a four semester period. It was found that the type of questionnaire generated from the item bank led to statistically significant differences among designated groups. The longitudinal analysis, however, indicated only minor improvement over time, regardless of whether or not an instructor chose to use items yielding specific feedback on the instructional components of a course. Additionally, although main effect differences were noted between teaching assistants and regular faculty, other results were very similar.     

The generalizability of student ratings of instructors: Item specificity and section effects

A number of recent studies have used generalizability theory to examine the dependability of student ratings of instruction. This study extends this line of research by examining the consistency of ratings between different sections of a course taught in a given semester by the same instructor, and by comparing the performance of global- and attribute-type instructor rating items. Five samples of physics instructors, varying in size from 5 to 12 instructors, were rated by their students on a form containing two global and eight attribute items. Each instructor taught two sections of a course. The study found that the section effect was small (ratings of instructors were consistent across different sections of the same course), and that the generalizability of ratings was substantially influenced by item specificity. For summary purposes, one global item seemed sufficient.     

Faculty Views of Student Evaluation of College Teaching

The literature abounds with psychometric studies of course evaluation measures and articles debating the merits of student ratings of instruction, but little research has focused on faculty perceptions of this procedure. In the present study faculty perceptions are explored at a teachers' college where evaluation is carried out annually on a sample of courses. The sample includes 101 instructors who completed the research questionnaire. Faculty attitudes reflected a broad range of responses towards validity of student ratings, and their usefulness for improving instruction. Although overall attitudes were mildly positive, few instructors reported changing instruction as a result of student ratings. Moreover, few supported sending evaluation results directly to college administrators or publishing them for student consumption.     

Administrative and structural changes in student evaluations of teaching and their effects on overall instructor scores

Using nine years of student evaluation of teaching (SET) data from a large US research university, we examine whether changes to the SET instrument have a substantial impact on overall instructor scores. Our study exploits four distinct natural experiments that arose when the SET instrument was changed. To maximise power, we compare the same course/instructor before and after each of the four changes occurred. We find that switching from in-class, paper course evaluations to online evaluations generates an average change of ?0.14 points on a five-point scale, or 0.25 standard deviations (SDs) in the overall instructor ratings. Changing labelling of the scale and the wording of the overall instructor question generates another decrease in the average rating: ?0.15 of a point (0.27 SDs). In contrast, extending the evaluation period to include the final examination and offering an incentive (early grade release) for completing the evaluations do not have a statistically significant effect on the overall instructor rating. The cumulative impact of these individual changes is ?0.29 points (0.52 SDs). This large decrease shows that SET scores are not comparable over time when instruments change. Therefore, administrators should measure and account for such changes when using historical benchmarks for evaluative purposes (e.g. appointments and compensation).     

Evaluating the Evaluations: A Factor Analysis of Student Ratings

Student evaluation of instruction and instructor performance in higher education are being accorded an increasingly prominent role in decisions concerned with faculty promotion and tenure, as well as with advisement on teaching strategies. In this context a number of factor analyses have been imposed upon student rating-instruments in order to facilitate meaningful interpretation of student judgments. The present paper reports the resul ts of a factor analysis of the rating scales used by graduate students enrolled in counselor education programs at Western Michigan University to evaluate perceived quality of instruction. A principal components factor analysis yielded three interpretable factors that were labeled Responsiveness-Rapport, Evaluation Process, and Instructional Merit. Implications of the factor structure for counselor educators are discussed.     

Predicting student ratings: the relationship between actual student ratings and instructors’ predictions

This study examined the relationship between student ratings and instructors’ predictions of these ratings, taking into account other instructor, student, and course characteristics. Participants in the study were 198 instructors in the School of Education at a major teacher training college in Israel. Data corresponding to one randomly selected course per instructor were collected using student and instructor questionnaires and college records. Results indicate a systematic positive relationship between instructors’ predictions and actual student ratings with respect to overall ratings and the ratings of three dimensions of teaching. Results also demonstrate a systematic trend whereby low‐rated instructors tend to overestimate their student ratings, high‐rated instructors underestimated ratings, and moderately rated instructors gave accurate predictions. Results have implications for using predictions to motivate teaching improvement.     

Academic discipline and generalizability of student evaluations of instruction

Previous research on the generalizability of student ratings of instruction has raised questions about the effects of academic discipline and item types on the generalizability of these data for making relative decisions about instructors and about courses. In particular, although student evaluation data appear to provide a reasonable basis for making decisions about instructors when generalizing across courses and students, when course is the object of measurement, the data appear to be less generalizable. It was suggested in the literature that this may be due to the type of evaluation items used or it may be due to academic discipline differences in the type of courses selected for study. This study used Biglan's (1973a) model for classifying disciplines along the dimensions of paradigmatic/preparadigmatic (hard/soft) and pure/applied. A nested sampling procedure yielded two sample types: courses within teachers, in which individual instructors taught more than one course; and teachers within courses, in which individual courses were taught by more than one instructor. For each sample type, evaluation forms for twenty courses within each discipline classification were sought. The evaluation items for this study were classified as measuring six dimensions of instruction: organization, breadth of coverage, group interaction, enthusiasm, grading, and individual rapport. Generalizability and decision studies were conducted in which, for one sample, teacher was the object of measurement, and for the second sample, course was the object of measurement. Results indicated that reliable decisions about instructors could reasonably be made from all six of the evaluation dimensions; however, reliability for course decisions varied greatly with the evaluation dimension, being highest for breadth of coverage and lowest for grading. The same general pattern was noted for the paradigmatic disciplines and the preparadigmatic-applied disciplines but not for the preparadigmatic-pure disciplines. It is suggested that a single evaluation instrument may not be uniformly applicable to all discipline areas.     

The impact of individual consultation on the teaching of younger versus older faculty

This study examines the effect of age of instructor on student ratings of teaching performance after individual consultation. Instructors self-presented over an 11-year period at a large Canadian university teaching service, where each received one of three interventions. End of term student ratings of the teaching of younger and older instructors are compared before consultation, immediately post consultation, and 1–3 years after year of consultation. Younger faculty obtained significantly improved ratings immediately after consultation, while older faculty achieved significant rating increases 1–3 years post consultation. Results from an earlier study on the impact of individual consultation on teacher ratings are reevaluated using this larger sample of faculty. Generally, results in this analysis parallel the original research. Consultation produced changes in student ratings both immediately after consultation as well as longitudinally, thus confirming the utility of intervention in producing enduring pedagogical improvements. Control analyses ensured that improvements were a result of the interventions and not an artefact of time.     

The effects of different oral directions as to disposition of results on student ratings of college instruction

A review of the literature is presented with particular reference to design flaws in previous studies of the effect of stated purpose of evaluation upon student ratings of instruction. Special attention in this study is paid to insuring the salience of the manipulated purpose-of-evaluation variable through using different oral instructions given to college students concerning the purposes of course evaluation. Students (N=516) within each of 18 classes were divided randomly into faculty use and administrative use conditions. Data were analyzed using global course and global instructor items as dependent variables on a standard short-form rating instrument. No statistically significant effect of the independent variable on either dependent variable was found. Recommendations regarding the design of this type of investigation are discussed as well as the implications of misconceptions about student ratings held by many college faculty.     

Student evaluation: The faculty responds

Results are presented of a survey of faculty members concerning their observations and opinions about the effects of student evaluation of instruction in a university where this practice had been a mandatory requirement for personnel evaluation purposes for several years. The respondents reported a definite reduction in morale and job satisfaction, changes in various instructional practices—mainly reduced course work demands on students and the occurrence of counterproductive actions as a result of this policy. Only limited expectations of improved ratings were indicated. There was also a clear division of opinion about the use of SEI ratings for personnel decisions that corresponded mainly with judgments about the validity of the ratings students provide.     

Improvement of instruction and teacher effectiveness: are student ratings reliable and valid?

This paper reviews the evidence on whether student ratings are reliable and valid enough to be used for the purpose of the improvement of instruction and teacher effectiveness. A consensus of evidence available in the literature tends to show a measure of consistency, stability and validity of student ratings. Several variables such as sex of rater and ratee, class size, mood of students, rank of instructors, grades students were expecting, time of the day courses are taught, to mention only a few, have been found to have a low to high positive relationship, with student ratings. It also appears that the use of student ratings leads to the improvement of instruction, provided the evaluation data are fed back to the instructor and that an expert or consultant provides assistance to the instructor.     

Student Evaluation of Teaching Effectiveness: An assessment of student perception and motivation

Over the past century, student ratings have steadily continued to take precedence in faculty evaluation systems in North America and Australia, are increasingly reported in Asia and Europe and are attracting considerable attention in the Far East. Since student ratings are the most, if not the only, influential measure of teaching effectiveness, active participation by and meaningful input from students can be critical in the success of such teaching evaluation systems. Nevertheless, very few studies have looked into students' perception of the teaching evaluation system and their motivation to participate. This study employs expectancy theory to evaluate some key factors that motivate students to participate in the teaching evaluation process. The results show that students generally consider an improvement in teaching to be the most attractive outcome of a teaching evaluation system. The second most attractive outcome was using teaching evaluations to improve course content and format. Using teaching evaluations for a professor's tenure, promotion and salary rise decisions and making the results of evaluations available for students' decisions on course and instructor selection were less important from the students' standpoint. Students' motivation to participate in teaching evaluations is also impacted significantly by their expectation that they will be able to provide meaningful feedback. Since quality student input is an essential antecedent of meaningful student evaluations of teaching effectiveness, the results of this study should be considered thoughtfully as the evaluation system is designed, implemented and operated.     

The influence of students' values and educational attitudes on their evaluation of faculty

The study determined the influence of students' values and educational attitudes on their evaluation of faculty. Sixty-five graduate students completed an evaluation form, a value survey, and an attitude scale at the end of an adult learning theory course. Multiple regression and path analysis showed that (a) educational attitudes and selected values accounted for 27% of the variance in student evaluation of faculty, (b) educational attitudes were responsible for 8% of the variance in student evaluation of faculty, and (c) values per se were not significantly related to student evaluation of faculty. A path model suggested that values influenced student evaluation of faculty through educational attitudes. The findings suggested that students' values and attitudes play a major role in their evaluation of faculty and should be recognized by teachers when interpreting the results of such ratings.     

Teaching Effectiveness: The relationship between reaction and learning evaluation criteria

This study used a pre/post design to assess student learning for the purposes of examining relationships among student grades, student learning, and student evaluations of teaching. These relationships were reframed in terms of reaction (Level I) and learning (Level II) evaluation criteria. Participants were 652 undergraduate students enrolled in seven sections of an introductory psychology course. Our results indicated a medium relationship between student grades and the pre/post learning measure. In addition, a small relationship was observed between student ratings of teaching effectiveness and a pre/post measure of learning. We conclude that student ratings and learning measures assess different aspects of teaching effectiveness and should not be used interchangeably. The most appropriate criterion for assessing teaching effectiveness is a function of the goal of evaluation. However, reaction and learning measures may be used in conjunction to obtain a more complete picture of instructor effectiveness.