An empirical test of the validity of student evaluations of teaching made on RateMyProfessors.com

The present article examined the validity of public web‐based teaching evaluations by comparing the ratings on RateMyProfessors.com for 126 professors at Lander University to the institutionally administered student evaluations of teaching and actual average assigned GPAs for these same professors. Easiness website ratings were significantly positively correlated with actual assigned grades. Further, clarity and helpfulness website ratings were significantly positively correlated with student ratings of overall instructor excellence and overall course excellence on the institutionally administered IDEA forms. The results of this study offer preliminary support for the validity of the evaluations on RateMyProfessors.com.     

Factorial Validity of Student Ratings of Instruction

This study tested the Systematic Distortion Hypothesis by examining the factorial validity of student ratings of university teaching. Factorial validity is defined as the degree to which covariance among judged traits resembles the actual or true covariation of observable behaviors underlying these traits. Although many studies have examined the factorial validity of ratings, results are inconsistent. The present study used a more complete methodology to address some of the limitations of previous studies. Student ratings of teaching and measurements of actual teaching behaviors were obtained for 32 instructors. Student ratings were compared to frequency counts of actual teaching behaviors obtained from videotape and to students’ similarity judgments of teacher characteristics. It was found, first, that the structure of student ratings showed a moderately strong relation to the structure of actual behaviors, and a somewhat stronger relation to the structure of conceptual associations; and second, that the effects of systematic distortion were more pronounced for low-inference student ratings than for high-inference ratings.     

Single comments or average ratings: which elements of RateMyProfessors.com™ shape university students’ judgments and course choice intentions?

The use and abuse of course and lecturer rating websites such as RateMyProfessors.com? is a highly relevant topic for universities’ evaluation and assessment policies and practice. However, only a few studies have paid attention to the actual influence of teaching evaluation websites on the students themselves—that is, their perceptions of a certain course and their course choice intention at university. Findings point to the fact that positive comments on the website about professors improve students’ evaluations. However, professor evaluation websites contain two types of information: single student comments and average ratings. Research on exemplification effects has shown that single cases often have a stronger influence on recipients than more valid base rate information. We test this assumption in an experiment (n?=?126) using a professor evaluation website stimulus. Results show that single comments strongly influence opinions and course choice intentions but that they are moderated by the valence of the average rating.     

Does the relationship between student ratings of course easiness and course quality vary across schools? The role of school academic rankings

Student ratings of teaching effectiveness are widely used to make judgments of faculty teaching performance. Research, however, has found that such ratings may not be accurate indicators of teaching performance because they are contaminated by course easiness. Using student ratings of 9855 professors employed at 79 different colleges and universities, the author hypothesized and found that the relationship between perceived course easiness and perceived course quality was moderated by school academic rankings. More specifically, easiness ratings were more strongly correlated with quality ratings among low‐ranked schools than among high‐ranked schools. Furthermore, the easiness–quality relationship was slightly stronger among public schools than among private schools. The article concludes by discussing the practical implications of these findings.     

Student Ratings and Professor Self-Ratings of College Teaching: Effects of Gender and Divisional Affiliation

Twenty female and 23 male professors at a liberal arts college participated along with their 803 undergraduate students in a questionnaire study of the effects of professor gender, student gender, and divisional affiliation on student ratings of professors and professor self-ratings. Students rated their professors on 26 questions tapping five teaching factors as well as overall teaching effectiveness. Professors rated themselves on the same questions as well as on nine exploratory ones. On student ratings, there were main effects for both professor gender (female professors were rated higher than male professors on the two interpersonal factors) and division (natural science courses were rated lowest on most factors). These patterns were qualified by significant interactions between professor gender and division. Although professor self-ratings varied by division, there were few significant correlations between professor self-ratings and students' ratings. Implications for future research are discussed.Appreciation goes to Laura Capotosto and Julie Phelan for their research assistance.Suzanne Montgomery is now at Widener Law School, Philadelphia, PA, USA.     

Correlations,trends and potential biases among publicly accessible web-based student evaluations of teaching: a large-scale study of RateMyProfessors.com data

Student evaluations of teaching are widely adopted across academic institutions, but there are many underlying trends and biases that can influence their interpretation. Publicly accessible web-based student evaluations of teaching are of particular relevance, due to their widespread use by students in the course selection process and the quantity of data available for analysis. In this study, data from the most popular of these websites, RateMyProfessors.com, is analysed for correlations between measures of instruction quality, easiness, physical attractiveness, discipline and gender. This study of 7,882,980 RateMyProfessors ratings (from 190,006 US professors with at least 20 student ratings) provides further insight into student perceptions of academic instruction and possible variables in student evaluations. Positive correlations were observed between ratings of instruction quality and easiness, as well as between instruction quality and attractiveness. On average, professors in science and engineering disciplines have lower ratings than in the humanities and arts. When looking at RateMyProfessors as a whole, the effect of a professor’s gender on rating criteria is small but statistically significant. When analysing the data as a function of discipline, however, the effects of gender are significantly more pronounced, albeit more complex. The potential implications are discussed.     

Student evaluation: The faculty responds

Results are presented of a survey of faculty members concerning their observations and opinions about the effects of student evaluation of instruction in a university where this practice had been a mandatory requirement for personnel evaluation purposes for several years. The respondents reported a definite reduction in morale and job satisfaction, changes in various instructional practices—mainly reduced course work demands on students and the occurrence of counterproductive actions as a result of this policy. Only limited expectations of improved ratings were indicated. There was also a clear division of opinion about the use of SEI ratings for personnel decisions that corresponded mainly with judgments about the validity of the ratings students provide.     

Validity and reliability of student perceptions of teaching quality in primary education

ABSTRACT

A Bayesian IRT-model approach was used to investigate the validity and reliability of student perceptions of teaching quality. Furthermore, the student perceptions were compared with ratings of teaching quality by external observers. Grade 4 students (n = 675) filled out a questionnaire that was used to measure their opinions about the lessons of their teachers. Three lessons of 39 teachers were recorded and rated by 4 raters. The analyses showed that student perception and lesson observation scales fit best in an 11-dimensional model, which was an indication of construct validity and discriminant validity. Student perception scales were reliable, although not all items contributed to the scales to the same extent. Student ratings and lesson observations scores generally correlated moderately (ranging from r = .18 to r = .50). Higher correlations were found for scales with a similar content; however, no clear pattern was apparent. Suggestions for future research are presented.     

Student assessment of teaching in higher education

Plans to introduce campus-wide assessments of college or university teaching which are largely dependent on student ratings are seen as a threat to academic freedom in those institutions with little or no experience of this form of evaluation. While regular student evaluations of teaching are very common in North America, their introduction is only now being considered in colleges and universities in a number of other countries. Research on the reliability and validity of student ratings indicate that they are capable of providing valuable information about the quality of teaching. Depending on the survey used, this type of evaluation may be used to provide evidence of teaching ability to staffing committees or to suggest ways of improving teaching. The paper concludes with a set of recommendations for higher education institutions which are considering the regular assessment of all teachers by their students.     

Perceptions of student engagement in the flipped classroom: a case study

ABSTRACT

One of the blended learning strategies that researchers and educators commonly use in higher education is Flipped Classroom. The purpose of this case study was to explore how both professors and students perceive student engagement in flipped classrooms. Three college professors who implemented flipped classrooms and their 14 students participated in the study. Three individual faculty interviews and three student focus group interviewers were conducted. Five components of the utilization-focused evaluation model were used in the data coding. The findings revealed both positive and negative opinions depending on how they implemented the flipped classroom including the engagement of students and the environment. Faculty participants who fully implemented the flipped model had positive opinions about the implementation and student engagement, finding the model helped increase student performance and grades. In addition, the majority of student participants liked the challenges offered by the flipped classroom, which increased their engagement.     

Examining the validity of office discipline referrals as an indicator of student behavior problems

Office discipline referral (ODR) data are increasingly used to monitor student behavior problems and the impact of interventions, but there has been limited research examining their validity. The current study examined the concordance of ODRs with teacher ratings of student behavior using data on 8,645 children in 335 classrooms at 21 elementary schools. The results of a variety of analyses (e.g., correlations, multivariate analysis of variance, receiver operating characteristics) suggested that ODRs are moderately valid and reliable. Multilevel analyses revealed that teacher ratings of disruptive behaviors were significantly associated with ODRs, even after controlling for other student‐, classroom‐, and school‐level factors. These findings suggest that ODRs are moderately valid indicators of student behavior problems and may be an efficient source of information for use in school‐based research and data‐based decision‐making. © 2011 Wiley Periodicals, Inc.     

Web-based student evaluations of professors: the relations between perceived quality,easiness and sexiness

College students critique their professors' teaching at RateMyProfessors.com, a web page where students anonymously rate their professors on Quality, Easiness, and Sexiness. Using the self-selected data from this public forum, we examine the relations between quality, easiness, and sexiness for 3190 professors at 25 universities. For faculty with at least ten student posts, the correlation between quality and easiness is 0.61, and the correlation between quality and sexiness is 0.30. Using simple linear regression, we find that about half of the variation in quality is a function of easiness and sexiness. When grouped into sexy and non-sexy professors, the data reveal that students give sexy-rated professors higher quality and easiness scores. If these findings reflect the thinking of American college students when they complete in-class student opinion surveys, then universities need to rethink the validity of student opinion surveys as a measure of teaching effectiveness. High student opinion survey scores might well be viewed with suspicion rather than reverence, since they might indicate a lack of rigor, little student learning, and grade inflation.     

University professor approaches to case studies: virtual field trips

Representatives from more than 20 institutions making up a research and teaching network met to discuss and investigate a web application designed to promote knowledge sharing and understanding. The web application, called Fieldtrip, enables students, professors and researchers to virtually share their experiences and findings from real field trips. This study investigates how professors from the network conceived Fieldtrip and their approach to designing and teaching student learning experiences using it. Results show that the professors had divergent understandings of the potential of the web application, and that qualitatively better ways of conceiving Fieldtrip were related to more robust approaches to design for learning and teaching. The implications of this study as an exploration into professor experiences of web applications are important if universities wish to make the best use of technologies in learning for their students and professors.

Intimations of the 21st Century

By happenstance rather than design, what began as a single study intended to refute the validity of student ratings of instruction turned into a 30-year quest into the nature of college teaching. Although I found that student ratings can be affected by seemingly extraneous variables such as instructor reputation, I quickly realized that the validity issue was not as simple as it first appeared, raising further questions such as: Was the student ratings questionnaire used in my study valid to begin with? Is instructor reputation simply a nuisance variable, even though it affects the validity of student ratings? Are students seduced by instructor reputation only under certain conditions? and so on. Oddly enough, a number of researchers, including myself, initially became interested in studying college teaching by attempting to demonstrate the invalidity of student ratings despite a research.     

The quality of student ratings of teacher behaviour

In a research project into the effectiveness of mathematics teaching in the first year of secondary education, external observers and students rated teachers' behaviour. The reliability and validity of both methods were established. The results show that teacher behaviour is assessed well when student ratings are aggregated at the classroom level. The quality of aggregated student ratings is as good as the quality of data from external observers. The predictive validity of aggregated student ratings is higher than the predictive validity of external observations when subject motivation is taken as a dependent variable.     

Meta-analysis of faculty's teaching effectiveness: Student evaluation of teaching ratings and student learning are not related

Student evaluation of teaching (SET) ratings are used to evaluate faculty's teaching effectiveness based on a widespread belief that students learn more from highly rated professors. The key evidence cited in support of this belief are meta-analyses of multisection studies showing small-to-moderate correlations between SET ratings and student achievement (e.g., Cohen, 1980, Cohen, 1981; Feldman, 1989). We re-analyzed previously published meta-analyses of the multisection studies and found that their findings were an artifact of small sample sized studies and publication bias. Whereas the small sample sized studies showed large and moderate correlation, the large sample sized studies showed no or only minimal correlation between SET ratings and learning. Our up-to-date meta-analysis of all multisection studies revealed no significant correlations between the SET ratings and learning. These findings suggest that institutions focused on student learning and career success may want to abandon SET ratings as a measure of faculty's teaching effectiveness.     

The changing role of the Japanese professor

In analyses of the professorial role, an opposition is often established between the professional or external components and the university or internal components. From this perspective, professors at leading universities in the West are said to emphasize the professional components of their role. In contrast, in the past Japanese professors tended to emphasize the local university-centered components. In recent years, many professors have shifted their energies away from these university-centered components. Some have become professional in the Western sense, but others have become showmen and still others politicians. No single phrase suffices to summarize the trend away from university centeredness.The authors wish to thank Kazuyuki Kitamura and the other members of the Research Institute for Higher Education of Hiroshima University for so generously supplying their unpublished data on university governance for their inspection. Also they acknowledge the stimulation they have experienced in reading Shigeru Nakayama's important comparative analysis of university traditions.     

Peer versus staff tutoring in problem-based learning

Effects of student versus staff tutoring on student learning in a problem-based, health sciences curriculum were studied. Academic achievement of 334 tutorial groups guided by staff tutors was compared with achievement of 400 groups guided by student tutors. In addition, students rated their tutor's performance on four behaviors considered critical to facilitating student learning. Overall, students guided by a staff tutor achieved somewhat better. In terms of practical significance, the difference was, however, fairly small. Staff tutors were rated as more knowledgeable and their contributions as more relevant. In addition, they asked stimulating questions to a larger extent. However, an interaction effect was found between the ratings and the year of study: Peer tutors displayed the supportive behaviors more extensively in the first year, whereas staff tutors' ratings were higher as the curriculum advanced. These results were interpreted in terms of the cognitive congruence framework.Parts of this article have been presented to the Annual Meeting of the American Educational Research Association, Atlanta, GA, April, 1993.     

The impact of individual consultation on student ratings of teaching

Abstract

The present study addressed the impact of individual consultation on teaching improvement as measured by changes in student ratings. Subjects included 91 professors who presented naturally for individual consultation services over a seven‐year period at the teaching centre of a Canadian university. Interventions by the consultant fell into three categories: 1) Feedback‐Consultation, 2) Feedback‐Consultation‐Class Observation, and 3) Feedback‐Consultation‐Class Observation and Student Consultation. End of term student ratings for the course that was the subject of the consultation were compared with student ratings for the same course taught between one and three years prior to the consultation service, and for the same course taught between one to three years following consultation. The results showed that, overall, consultation was effective in improving the quality of the consultees’ teaching, as evidenced by an increase in mean student ratings of instruction. This effect persisted post consultation. Not all intervention groups, however, showed the same pattern of results. Change was evident immediately after the intervention except in the case of brief consultation, although follow‐up data showed improved teaching for the latter group. Control data provided evidence that the change in student ratings post consultation could reasonably be attributed to consultation effects.