A review of the research on student evaluation and a report on the effect of different sets of instructions on student course and instructor evaluation

An extensive review of the research concerning the effect of different variables on student ratings is presented. A study is then reported comparing the effects of different sets of instructions on student evaluations of the course and instructor. The results indicated that the students who were informed that the results of their ratings would be used for administrative decisions rated the course and instructor more favorably on all aspects than students who were informed that the results of their ratings would only be used by the instructor.The authors are indebted to Professor Robert A. Waller for cooperating in obtaining the data on his two history courses.     

Intimations of the 21st Century

By happenstance rather than design, what began as a single study intended to refute the validity of student ratings of instruction turned into a 30-year quest into the nature of college teaching. Although I found that student ratings can be affected by seemingly extraneous variables such as instructor reputation, I quickly realized that the validity issue was not as simple as it first appeared, raising further questions such as: Was the student ratings questionnaire used in my study valid to begin with? Is instructor reputation simply a nuisance variable, even though it affects the validity of student ratings? Are students seduced by instructor reputation only under certain conditions? and so on. Oddly enough, a number of researchers, including myself, initially became interested in studying college teaching by attempting to demonstrate the invalidity of student ratings despite a research.     

Relationships between student perceptions of teaching behavior and learning outcomes in college biology

This study examined relationships between student perceptions of teaching behavior and learning outcomes in college biology. One-hundred one students who enrolled in an introductory college biology course reported the frequencies of 20 overt, in-class teaching behaviors exhibited by the instructor and rated the instructor on 12 measures of teaching effectiveness. Achievement tests were administered periodically throughout the two-semester time interval. A series of simple correlations between teaching behavior factors and both achievement and student ratings revealed a complex pattern of relationships. Moderately strong relationships were found between student perceptions of “information-transmitting” behaviors and achievement, while generally stronger relationships linked behavior factors with student ratings of teaching effectiveness.     

Validity of student self-reported ratings of learning

This study examined the relationship between individual student self-reported ratings of progress on relevant learning objectives and performance on exams administered during a college course. Across three sections of the same course taught by a single instructor, 188 students rated themselves at the end of the course on two objectives identified by the instructor as either essential or important. They also rated themselves on 10 other objectives the instructor identified as having minor or no importance. Self-ratings on course-relevant objectives correlated significantly and positively with four out of five exams and the course total, whereas ratings on irrelevant objectives did not. Students who rated their progress as either exceptional or substantial generally performed better on course examinations than those who rated their progress as moderate or less. These findings support the validity of student self-reported ratings of learning.     

Students' ratings of instructors revisited: Interactions among class and instructor variables

Previous studies concerning students' ratings of instruction have traditionally used the class as the unit of analysis and the ratings have been analyzed in one of two ways: (1) regression analysis, wherein the amount of variability in instructor ratings can be attributed to a set of variables; or (2) analysis of variance, wherein the effect of some selected independent variable on instructor ratings is measured. While both approaches have provided valuable information about the evaluation of instruction, little attention has been given to the interactions among the variables selected. In order to determine how situational variables influence the student at the time an evaluation is performed, the present study used the individual student as the unit of analysis and focused principally on the interactions between three variables related to the class (type, level, and size) and three variables related to the instructor (reputation, rank, and sex). The data were analyzed through 15 two-way factorial analyses of variance, with 23 main effects and 12 interactions reaching significance. The implications of these findings are discussed in terms of their effect on the student rating process.     

The effect of Twitter posts on students’ perceptions of instructor credibility

Greater numbers of instructors are turning to social networking sites to communicate with students. This study examined whether posting social, scholarly, or a combination of social and scholarly information to Twitter has an impact on the perceived credibility of the instructor. Participants were assigned to one of three groups: a group that viewed social tweets, one that viewed scholarly tweets, and one that viewed a combination of social and scholarly tweets. Participants were then asked questions about the instructor’s perceived credibility. Results show that participants who viewed only the social tweets rated the instructor significantly higher in perceived credibility than the group that viewed only the scholarly tweets. No other significant differences were found among the groups. These results have implications for both teaching and learning, as there is an established link between perceived instructor credibility and positive learning outcomes.     

The effect of college course evaluation feedback on instructor and student perceptions of instructional climate and effectiveness

An experimental study was undertaken to examine (1) effect of early course evaluation feedback to instructor on subsequent end-of-course class evaluations by students (n=1484) and instructor (n=78), and (2) stability of course evaluations by students and instructor. These voluntary class-instructor units were randomly assigned to experimental (feedback) and control (no feedback) groups. After attrition the feedback group contained 37 units and the control group 41 units. The 26 item Faculty-Course Evaluation Form (FCEF) was administered approximately three weeks into the quarter and again eight weeks later. The FCEF reflected the frequency of occurrence of certain instructor behaviors and over-all opinions of course value and teaching ability.The five-factor structure of the FCEF can be described with the following labels: Subject Organization and Competency, Motivation-Stimulation, Instructor-Student Relations, Reasonable Work Load and Tests, and Clearness of Grading Procedures.Individual class, instructor, and total score results were fed back to instructors in the experimental groups from the early FCEF administrators. Results indicated significant differences in favor of the feedback group on the Subject Organization and Competency, and Motivation-Stimulation scale. In addition moderately high stability of ratings for both classes and instructors was noted, but little agreement between classes and instructors.     

The generalizability of student ratings of instructors: Item specificity and section effects

A number of recent studies have used generalizability theory to examine the dependability of student ratings of instruction. This study extends this line of research by examining the consistency of ratings between different sections of a course taught in a given semester by the same instructor, and by comparing the performance of global- and attribute-type instructor rating items. Five samples of physics instructors, varying in size from 5 to 12 instructors, were rated by their students on a form containing two global and eight attribute items. Each instructor taught two sections of a course. The study found that the section effect was small (ratings of instructors were consistent across different sections of the same course), and that the generalizability of ratings was substantially influenced by item specificity. For summary purposes, one global item seemed sufficient.     

Exploring graduate and undergraduate course evaluations administered on paper and online: a case study

The purpose of this study was to analyse the students’ evaluations of the course and instructor for all statistics courses offered during fall semester 2009 at a large university in the southern United States. Data were collected and analysed for course evaluations administered both online and on paper to students in both undergraduate and graduate courses. Unlike most previous studies on this subject, class section rather than student was treated as the unit of analysis. It was of specific interest to verify prior research findings that evaluation surveys administered online would not result in lower course and instructor ratings and lower response rates. The results showed that there is not sufficient evidence within the collected data to conclude that either course and instructor ratings or response rates are lower for evaluations administered online (online evaluations) than they are for evaluations administered on paper (paper evaluations). Of secondary interest was whether class ratings would be associated with students’ attendance and a comparison of variability among answers for undergraduate vs. graduate students. It was observed that class and teacher ratings were not related to students’ attendance and individual students did not tend to give the same answer for every question on their survey.     

Another look at college student’s ratings of course quality: data from Penn State student surveys in three settings

This study assessed the relationships of student attributes, course characteristics and course outcomes to college students’ ratings of course quality in three types of settings. The analysis utilised data from online surveys of samples of college students conducted in 2011 and 2012 at the Pennsylvania State University. Included in the analysis were: (1) 1805 students at the main campus; (2) 1453 students at 19 smaller satellite campus locations of the university scattered across the state; and (3) 522 students participating in online degree programmes through Penn State’s World Campus. Students were asked to rate the quality of instruction they received in a randomly selected course in which they had been enrolled during the previous semester, and to respond to a number of questions about the course, the instructor’s behaviour and themselves. The relationships of these factors to how students rated the course were assessed for subjects in the three study settings. In all three settings, student and course characteristics, course difficulty and amount of required work had little effect on course ratings. Grade received was modestly related to course rating. However, instructor’s use of selected recommended pedagogical practices and students’ perceptions of how much they felt they learned were by far the strongest correlates of students’ course evaluations. Implications of these findings are discussed.     

Predicting student ratings: the relationship between actual student ratings and instructors’ predictions

This study examined the relationship between student ratings and instructors’ predictions of these ratings, taking into account other instructor, student, and course characteristics. Participants in the study were 198 instructors in the School of Education at a major teacher training college in Israel. Data corresponding to one randomly selected course per instructor were collected using student and instructor questionnaires and college records. Results indicate a systematic positive relationship between instructors’ predictions and actual student ratings with respect to overall ratings and the ratings of three dimensions of teaching. Results also demonstrate a systematic trend whereby low‐rated instructors tend to overestimate their student ratings, high‐rated instructors underestimated ratings, and moderately rated instructors gave accurate predictions. Results have implications for using predictions to motivate teaching improvement.     

Student Psychological Need Satisfaction and College Teacher-Course Evaluations

Two studies examined student psychological need satisfaction as a predictor of positive teacher-course evaluations. In Study 1, 268 undergraduates recalled and rated the quality of a recent important college course, then rated their feelings of autonomy, competence, and relatedness within that course. Consistent with self-determination theory, all three ratings predicted instructor and/or course ratings. Study 2 found the same pattern in a sample of 179 introductory journalism students nested within 12 sections of a single course. Study 2 also evaluated instructor characteristics as predictors of mean levels of student need satisfaction across the 12 classes. Although instructor age and overall teaching experience were unrelated to students' need satisfaction, greater experience teaching their particular class negatively predicted student autonomy and relatedness need satisfaction. Implications for pedagogical practice are discussed.     

Evaluation of an Experimental Course in Race Relations

Universities and colleges have developed many new courses concerned with race relations. Objectives of these courses have been far-reaching and it has been particularly difficult to evaluate their effectiveness. Out-comes are often not clearly described. This study examined student experiences with minority groups before, during, and after taking an experimental course in race relations. We observed differences in the experiences with minority groups of open-minded, moderate, and closed-minded students who took the course, and differences in the experiences with minority groups of students who were more or less satisfied with the course. The sample was made up of 69 percent (N = 66) of the students who completed the course. Students said they had first heard of more of the well known minority individuals and had had more interpersonal relations with members of minority groups before taking the course. Most had visited or heard of more of the community agencies and councils while taking the course. At the conclusion of the course, a large number still had not read most of the books, magazines, and newspapers. Open-minded students had had more experiences with minority groups than had moderates before taking the course. The degree to which students were open or closed minded was also related to their number of non-experiences with both minority group people and minority literature at the conclusion of the course. Surprisingly, closed minded students were more satisfied with the course. More satisfied students read less about minority groups after the course was completed. Finally, the results indicated no significant differences in the experiences before, during, and after the course of students who were more and less satisfied with the course.     

The efficacy of directed studies courses as a form of undergraduate research experience: a comparison of instructor and student perspectives on course dynamics

Directed studies (DS) courses are widely touted for their ability to enhance research skills in undergraduate students—yet little is known about the dynamics, motivations, and perceived outcomes connected to these specific types of undergraduate research experiences. Building on earlier qualitative research, in this paper we report the results of a self-report survey designed to directly compare instructor and student perspectives on DS course dynamics at a small, liberal arts university. Samples of students who completed DS courses and instructors who supervised them completed a survey assessing their motives, perceived outcomes, and barriers encountered in their course work. Parallel wording of items in instructor and student surveys permitted comparison of perceptions of DS course dynamics. Results indicated that there were many similarities in how both groups approached DS courses but that there were also several important differences in motives and perceived outcomes pointing to the need for greater communication between instructors and students about their expectations for the course.     

The Meaning and Utility of Institutional Teaching Evaluations

This study analyzes end of course institutional teaching evaluations (ITEs) done anonymously by students in a pre-service teacher education course over a 15-year span. The purpose was to determine if and how the ITE findings might inform practice and relate to teaching metaphors as a tool of study. Analysis revealed: (a) teacher effectiveness ratings were more meaningful when examined by metaphor periods and when outliers, which skewed the trends, were removed, (b) patterns in students’ optional narrative comments on ITEs indicated that students consistently saw instructor strength in caring and weakness in clarity, (c) there was no clear correlation between narrative comments and teacher effectiveness ratings, and (d) alignment with identified teaching metaphors was strongest with student narrative comment categories. In addition, doing this analysis reframed the way I thought of myself as a teacher and allowed me to make positive changes in how I related to students. Comments from the course in the semester following the analysis support this contention. This work informs the field by considering the usefulness of metaphors for long-term study of practice as well as modeling how teacher educators subjected to accountability measures can use such institutional measures to create meaningful study of practice.     

Similarities and Differences in the Play Activities of Children in Two Public Schools with Contrasting Environment

Based on previous work by H. G. Murray, this study examined relationships between overt classroom teaching behaviors of college instructors and student ratings of teaching effectiveness. One hundred seven students who enrolled in six sections of an introductory psychology course rated instructors on 12 measures of teaching effectiveness and reported the frequencies of 20 observable, inclass teaching behaviors. Relationships between teaching behaviors and student ratings were examined by simple, multiple, and canonical correlation procedures. A complex pattern of relationships emerged. The canonical analysis revealed a two-dimensional structure consisting of instructor clarity and student-teacher rapport dimensions. Teaching behaviors loading on these dimensions were examined.     

Similarity of student ratings across instructors,courses, and time

This study raised three questions about the similarity or generalizability of student ratings of courses and instructors. First, how stable are student ratings of the same instructor giving the same course during twodifferent semesters? Second, how similar are student ratings of the same instructor in twodifferent courses? Third, how similar are student ratings of a given course being taught bydifferent instructors? Instances were identified in which student ratings on seven different factors were available for pairs of courses for each of these questions. For the case of the same instructor — same course—different semesters, student ratings were reasonably similar (median r for seven factors about 0.70). For the case of the same instructor—different courses, the median r was surprisingly low — about 0.40. For the case of the same course—different instructors, substantial correlations were obtained for some factors and insignificant correlations for other factors. Implications of these findings for practical use of student ratings and suggestions for further research in the area are discussed.     

Effects of administrative scheduling and directions upon student ratings of instruction

Using temporal proximity of instructor/course evaluations to critical in-course events, together with psychological set provided via directions, the present investigation sought to assess (1) the appropriateness of within-course time-series analyses for instructor ratings, and (2) the potency of inducing rater perceptions of the rating process as an interactive variable in course evaluations. A 2 × 2 (temporal proximity by psychological set) factorial design used students enrolled in two sections of an undergraduate course in educational psychology as subjects. Using both midterm and final examination time periods as data collection points, the results of this investigation suggested that preexamination instructor/course evaluations tend to be more favorable than postexamination evaluations, and that a specific positive psychological set prior to evaluations tends to produce more favorable evaluations than very general, brief instructions.     

Ratings of university teacher instruction: how much do student and course characteristics really matter?

Several student and course characteristics were examined in relation to student ratings of instruction. Students at a major Canadian university completed the Universal Student Ratings of Instruction instrument at the end of every course over a three‐year period, providing 371,131 student ratings. Analyses of between‐group differences indicate that students who attend class often and expect high grades provide high ratings of their instructors (p < .001). In addition, lab‐type courses receive higher ratings than lectures or tutorials, and courses in the social sciences receive higher ratings than courses in the natural sciences (p < .001). Regression analyses indicated, however, that student and course characteristics explain little variance in student ratings of their instructors (<7%). It is concluded that student ratings are more related to teaching instruction and behavior of the instructor than to these variables.     

Micro-Counseling: An Experimental Study of Pre-Practicum Training in Communicating Test Results

Micro-counseling techniques were used to train 10 beginning graduate students in communicating test results. Training emphasized listening and responding to cues given by the client, brief and concise statements about test-related information, and relating the information to the client's experiences and needs. Training aids included programmed materials, brief videotapes of positive and negative models, role-playing, and videotape feedback of the role-playing and test interpretation sessions. Judges' ratings of trainees in the experimental group were significantly higher than their ratings for the 10 trainees in the control group. Client ratings of counselor effectiveness did not differ significantly for the two groups. Changes in clients' pre- and post-test attitude scores and knowledge scores were statistically significant. An enthusiastic response to the micro-counseling approach was evident from both inexperienced and experienced graduate students.