Bias in student evaluations: Are my peers out to get me?

The use of peer evaluations to evaluate performance is commonplace, especially in higher education. Yet, researchers and educators have long expressed concerns about the accuracy of such ratings. While research has found student peer evaluations to be consistent with instructor evaluations, rater bias is inherent in all evaluations, and students are not exempt from this. This study examines task completion and identity threat to determine if experience and identity play a role in shaping student peer evaluations. A sample of MBA students evaluated their peers before and after completing a course presentation. Results showed that students’ peer evaluations were significantly “lower” or more critical after having completed the presentation themselves. This study illuminates the importance of experience and training in the implementation of performance evaluations.     

Determinants of teaching quality: What's important to students?

A method for using student evaluations to help faculty improve their teaching performance is presented. A survey of current methods of student evaluations of teaching identified a need to improve the statistical information obtained from these evaluations. An ordinary least squares framework is used to identify the factors that students feel are important in teacher and course ratings. This framework is used to estimate weights that students assign to various teacher and course attributes and to test whether students apply these weights consistently across teachers and courses. About 81 percent of the explained variation in teacher ratings was associated with attributes that contribute to student enjoyment of the learning process. Over 90 percent of the explained variation in course ratings was associated with attributes that measure how much a student learned in the course. Students were found to apply these attributes or weights consistently across teachers and courses. Implications for developing effective teaching strategies, faculty recruitment, and curriculum reform are discussed.     

Exploring graduate and undergraduate course evaluations administered on paper and online: a case study

The purpose of this study was to analyse the students’ evaluations of the course and instructor for all statistics courses offered during fall semester 2009 at a large university in the southern United States. Data were collected and analysed for course evaluations administered both online and on paper to students in both undergraduate and graduate courses. Unlike most previous studies on this subject, class section rather than student was treated as the unit of analysis. It was of specific interest to verify prior research findings that evaluation surveys administered online would not result in lower course and instructor ratings and lower response rates. The results showed that there is not sufficient evidence within the collected data to conclude that either course and instructor ratings or response rates are lower for evaluations administered online (online evaluations) than they are for evaluations administered on paper (paper evaluations). Of secondary interest was whether class ratings would be associated with students’ attendance and a comparison of variability among answers for undergraduate vs. graduate students. It was observed that class and teacher ratings were not related to students’ attendance and individual students did not tend to give the same answer for every question on their survey.     

An empirical test of the validity of student evaluations of teaching made on RateMyProfessors.com

The present article examined the validity of public web‐based teaching evaluations by comparing the ratings on RateMyProfessors.com for 126 professors at Lander University to the institutionally administered student evaluations of teaching and actual average assigned GPAs for these same professors. Easiness website ratings were significantly positively correlated with actual assigned grades. Further, clarity and helpfulness website ratings were significantly positively correlated with student ratings of overall instructor excellence and overall course excellence on the institutionally administered IDEA forms. The results of this study offer preliminary support for the validity of the evaluations on RateMyProfessors.com.     

Student evaluations of teaching: effects of the Big Five personality traits,grades and the validity hypothesis

The purpose of the current study was to examine whether the Big Five personality traits and expected student grades relate to student evaluations of teachers and courses at the college level. Extraversion, openness, agreeableness and conscientiousness were found to be personality traits favoured in instructors, whereas neuroticism was not. A significant correlation was found between the students’ expected grades in the course and student evaluations of the course, but not the evaluations of the instructor. When the effect of students’ perceived amount of learning was taken into account, no significant effect of grades was found on teacher ratings. Personality explained variance in teacher and course evaluations over and above grades and perceived learning.     

Student Evaluation of Faculty: Effects of Purpose on Pattern

Much of the existing research addressing student ratings of faculty has focused on psychometric properties of the rating process or on physical characteristics of courses and individuals. Less explored is the cognitive processes of students, as evaluators, and the administrative purposes for which outcomes from student evaluations are used. To determine if students used different cognitive sets as a function of the explicit purpose for which ratings are obtained, a field experiment was conducted. Explicit purposes for course evaluations were manipulated to include a control condition, a formative condition (instructional improvement), and a summative condition (salary increase). These conditions served as the classification variable in a discriminant analysis and six different dimensions of classroom performance, as assessed by students, served as the discriminating variables. Results indicate that the purpose of assessment fails to influence the pattern of faculty ratings provided by students. From an administrative perspective, these findings have practical implications both for the collection of student evaluations and for the use of student ratings.     

A review of the research on student evaluation and a report on the effect of different sets of instructions on student course and instructor evaluation

An extensive review of the research concerning the effect of different variables on student ratings is presented. A study is then reported comparing the effects of different sets of instructions on student evaluations of the course and instructor. The results indicated that the students who were informed that the results of their ratings would be used for administrative decisions rated the course and instructor more favorably on all aspects than students who were informed that the results of their ratings would only be used by the instructor.The authors are indebted to Professor Robert A. Waller for cooperating in obtaining the data on his two history courses.     

Correlations,trends and potential biases among publicly accessible web-based student evaluations of teaching: a large-scale study of RateMyProfessors.com data

Student evaluations of teaching are widely adopted across academic institutions, but there are many underlying trends and biases that can influence their interpretation. Publicly accessible web-based student evaluations of teaching are of particular relevance, due to their widespread use by students in the course selection process and the quantity of data available for analysis. In this study, data from the most popular of these websites, RateMyProfessors.com, is analysed for correlations between measures of instruction quality, easiness, physical attractiveness, discipline and gender. This study of 7,882,980 RateMyProfessors ratings (from 190,006 US professors with at least 20 student ratings) provides further insight into student perceptions of academic instruction and possible variables in student evaluations. Positive correlations were observed between ratings of instruction quality and easiness, as well as between instruction quality and attractiveness. On average, professors in science and engineering disciplines have lower ratings than in the humanities and arts. When looking at RateMyProfessors as a whole, the effect of a professor’s gender on rating criteria is small but statistically significant. When analysing the data as a function of discipline, however, the effects of gender are significantly more pronounced, albeit more complex. The potential implications are discussed.     

Comparing Student Evaluations of Distance Learning and Standard Classroom Courses in Graduate Social Work Education

Abstract

This study compared the student course evaluations of standard education and distance learning courses in the school of social work of one southern university. The sample included 14 distance learning (DL) and 122 standard education courses. A 20-item quantitative student course evaluation and a 7-item qualitative questionnaire were used to compare differences between social work courses taught in a standard classroom setting with those taught using distance learning technology. Comparisons of identical courses taught in both formats found that students rated distance learning higher than the standard classroom for a course having predominantly lecture content. Conversely, ratings for clinical practice courses were just the opposite with lower ratings for distance learning than the standard classroom. Results of a qualitative survey (n = 39) of students conducted at the end of a course taught by distance learning found that 73% of the students felt that distance learning technology interfered with class participation. However, 73% of the students felt the instructor handled technological problems in a professional and patient manner. Although 54% of the students reported that they would take another course by distance learning, more research is needed to determine what social work courses can be most effectively taught by distance and classroom formats.     

I liked your course because you taught me well: The influence of grades,workload, expectations and goals on students' evaluations of teaching

There has been considerable debate as to whether course evaluations are valid measures of teaching quality, or whether students instead reward tutors who give them high grades and assign low levels of work. To assess the factors that influence course evaluations, we measured university students' achievement goals and expectations at the beginning of the semester and also obtained information on grades and workload. Although grades and course difficulty did have a small influence on end‐of‐semester course ratings, structural modelling revealed that ratings were largely determined by how much students enjoyed or felt stimulated by the course content, which in turn depended on the perceived quality of teaching. Students with a mastery goal were more likely to look forward to the course, and this also contributed to positive course evaluations, but the effect was small. Overall, the results suggested that by far the largest determinant of student evaluation of courses is the quality of the teaching.     

Student Evaluation of Teaching Effectiveness: An assessment of student perception and motivation

Over the past century, student ratings have steadily continued to take precedence in faculty evaluation systems in North America and Australia, are increasingly reported in Asia and Europe and are attracting considerable attention in the Far East. Since student ratings are the most, if not the only, influential measure of teaching effectiveness, active participation by and meaningful input from students can be critical in the success of such teaching evaluation systems. Nevertheless, very few studies have looked into students' perception of the teaching evaluation system and their motivation to participate. This study employs expectancy theory to evaluate some key factors that motivate students to participate in the teaching evaluation process. The results show that students generally consider an improvement in teaching to be the most attractive outcome of a teaching evaluation system. The second most attractive outcome was using teaching evaluations to improve course content and format. Using teaching evaluations for a professor's tenure, promotion and salary rise decisions and making the results of evaluations available for students' decisions on course and instructor selection were less important from the students' standpoint. Students' motivation to participate in teaching evaluations is also impacted significantly by their expectation that they will be able to provide meaningful feedback. Since quality student input is an essential antecedent of meaningful student evaluations of teaching effectiveness, the results of this study should be considered thoughtfully as the evaluation system is designed, implemented and operated.     

“Top of the Mark”

The present study investigated differences in the effectiveness of instructors from a variety of departments who taught the same course in both intensive and traditional formats within the same year, while controlling for many confounding variables. Results indicated that intensive courses did not significantly differ from traditional courses in overall instructor ratings on student evaluations of teaching effectiveness when confounding variables were taken into account. Conversely, intensive courses received significantly higher overall course ratings on student evaluations than did traditional courses, even after controlling for class size and probable grade in course. These findings provide further evidence that negative beliefs concerning intensive courses may be unjustified, and intensive courses may be as or more effective than those presented in traditional formats.     

Stability and correlates of student evaluations of teaching at a Chinese university

This paper examines the stability and validity of a student evaluations of teaching (SET) instrument used by the administration at a university in the PR China. The SET scores for two semesters of courses taught by 435 teachers were collected. Total 388 teachers (170 males and 218 females) were also invited to fill out the 60‐item NEO Five‐Factor Inventory together with a demographic information questionnaire. The SET responses were found to have very high internal consistency and confirmatory factor analysis supported a one‐factor solution. The SET re‐test correlations were .62 for both the teachers who taught the same course (n = 234) and those who taught a different course in the second semester (n = 201). Linguistics teachers received higher SET scores than either social science or humanities or science and technology teachers. Student ratings were significantly related to Neuroticism and Extraversion. Regression results showed that the Big‐Five personality traits as a group explained only 2.6% of the total variance of student ratings and academic discipline explained 12.7% of the total variance of student ratings. Overall the stability and validity of SET was supported and future uses of SET scores in the PR China are discussed.     

Disciplinary,Gender and Course Year Influences on Student Perceptions of Teaching: Explorations and implications

The prevalent use of student ratings in teaching evaluations, particularly the reliability of such data, has been debated for many years. Reports in the literature indicate that there are many factors influencing student perceptions of teaching. Three of these factors were investigated at the University of Western Australia, namely the broad discipline group, course/unit year level and student gender. Data collected over 3 years were analysed. The outcomes of this study confirmed results reported by other workers in the field that there are differences in ratings of students in different discipline groups and at different year levels. It also provided a possible explanation for the mixed results reported in studies of student gender in relation to student ratings.     

Student Psychological Need Satisfaction and College Teacher-Course Evaluations

Two studies examined student psychological need satisfaction as a predictor of positive teacher-course evaluations. In Study 1, 268 undergraduates recalled and rated the quality of a recent important college course, then rated their feelings of autonomy, competence, and relatedness within that course. Consistent with self-determination theory, all three ratings predicted instructor and/or course ratings. Study 2 found the same pattern in a sample of 179 introductory journalism students nested within 12 sections of a single course. Study 2 also evaluated instructor characteristics as predictors of mean levels of student need satisfaction across the 12 classes. Although instructor age and overall teaching experience were unrelated to students' need satisfaction, greater experience teaching their particular class negatively predicted student autonomy and relatedness need satisfaction. Implications for pedagogical practice are discussed.     

Are Student Evaluations of Teaching Effectiveness Valid for Measuring Student Learning Outcomes in Business Related Classes? A Neural Network and Bayesian Analyses

In this study we investigate the underlying relational structure between student evaluations of teaching effectiveness (SETEs) and achievement of student learning outcomes in 116 business related courses. Utilizing traditional statistical techniques, a neural network analysis and a Bayesian data reduction and classification algorithm, we find little or no support for the validity of SETEs as a general indicator of teaching effectiveness or student learning. In fact, the underlying structure appears to be non-linear and possibly negatively bimodal where the most effective instructors are within the middle percentiles of student course ratings, while instructors receiving ratings in the top quintile or the bottom quintile are associated with significantly lower levels of student achievement.     

Understanding Student Evaluations of Teaching Quality: The Contributions of Class Attendance

Institutions of higher learning employ compulsory attendance policies with the expectation that these mandates enhance students' academic performance and perceptions of course quality. However, numerous empirical investigations demonstrate equivocal and often contradictory findings regarding the relationship between attendance and various markers of student achievement. The present investigation extends this research by exploring the utility of student ratings of the need to attend class in predicting their perceptions of teaching excellence after controlling for class size, instructor availability, and small-group interactions. As hypothesized, ratings of the need to attend predicted excellence while accounting for a significant 5.3% of the variance. Discussion and conclusions highlight the utility of class attendance in understanding students' evaluations of teaching and course quality.     

Correlates of monotonic response patterns in online ratings of a university course

This study examines the prevalence, contexts, and demographic correlates of monotonic response patterns (MRPs) in online student evaluations. Results of two-level hierarchical generalized linear models show evidence of careless monotonic responses to a survey administered to students enrolled in a university-level foreign language course in the Republic of Korea. All else being equal, freshmen and students in classes with fewer survey participants were more likely to choose monotonic response patterns in course evaluations. Possible factors at work in generating MRPs are identified and discussed. The severity of the MRP problem in online ratings underscores the importance for administrators to consider possible validity threats in student evaluations before using them as tools to inform instructional and administrative decisions. It is also important to design course evaluation surveys in such a way as to minimize careless responses and to identify means to induce more thoughtful responses from college students.     

Do examinations influence student evaluations?

This paper measures the impact of timing on student evaluations of teaching effectiveness, using a dataset of close to 3000 observations from Erasmus School of Economics. A special feature of the data is that students were able to complete on-line questionnaires during a time window ranging from one week before to one week after the final examination. This allows for the isolation of the effect of the examination on student evaluations. Among students who subsequently pass the exam, we find little difference between pre- and post-exam ratings. Among students who fail, evaluation scores are significantly lower after the exam on a number of items. Our evidence is compatible with a self-serving bias in student evaluations, but does not indicate that students seek revenge on instructors through lower ratings.     

Teaching Effectiveness: The relationship between reaction and learning evaluation criteria

This study used a pre/post design to assess student learning for the purposes of examining relationships among student grades, student learning, and student evaluations of teaching. These relationships were reframed in terms of reaction (Level I) and learning (Level II) evaluation criteria. Participants were 652 undergraduate students enrolled in seven sections of an introductory psychology course. Our results indicated a medium relationship between student grades and the pre/post learning measure. In addition, a small relationship was observed between student ratings of teaching effectiveness and a pre/post measure of learning. We conclude that student ratings and learning measures assess different aspects of teaching effectiveness and should not be used interchangeably. The most appropriate criterion for assessing teaching effectiveness is a function of the goal of evaluation. However, reaction and learning measures may be used in conjunction to obtain a more complete picture of instructor effectiveness.