RateMyProfessors.com offers biased evaluations

RateMyProfessors.com (RMP) is becoming an increasingly popular tool among students, faculty and school administrators. The validity of RMP is a point of debate; many would argue that self‐selection bias obscures the usefulness of RMP evaluations. In order to test this possibility, we collected three types of evaluations: RMP evaluations that existed at the beginning of our study, traditional in‐class evaluations and RMP evaluations that were prompted after we collected in‐class evaluations. We found differences in the types of evaluations students provide for their professors for both perceptions of professor clarity and ratings of professor easiness. Based on these results, conclusions drawn from RMP are suspect and indeed may offer a biased view of professors.     

Using the sampling margin of error to assess the interpretative validity of student evaluations of teaching

We present an equation, derived from standard statistical theory, that can be used to estimate sampling margin of error for student evaluations of teaching (SETs). We use the equation to examine the effect of sample size, response rates and sample variability on the estimated sampling margin of error, and present results in four tables that allow users to assess the interpretative validity (IV) of a SET score for a specific evaluation context. In this framework, a small margin of error, e.g. 3% of the range, suggests a greater precision, or IV in a score, whereas a large margin of error, e.g. 10% of the range, suggests a lower IV. We review the SET literature and summarise five ways that low response rates may be ameliorated. We also compare our results to previous published studies. Our findings matched, but greatly extended, prior results.     

A study of the validity of four types of student ratings of college teaching assessed on a criterion of student achievement gains

The purpose of the study was to investigate relationships between student ratings of college teaching using four types of student rating instruments and pre- vs. post-student achievement gains in 36 sections of an undergraduate analytic geometry and calculus course. Student rating instruments used varied according to type of items (high vs. low inference) and focus (students rating their own perceived growth vs. rating the instructor). Data were collected on 799 students (66% freshmen; 16% sophomore; and 15% juniors) at the University of Florida, and relationships were analyzed using the Pearson product-moment correlation technique. Significant relationships were not found between student ratings and student achievement.     

Expectancy theory outcomes and student evaluations of teaching

As student evaluation of teaching (SET) instruments are increasingly administered online, research has found that the response rates have dropped significantly. Validity concerns have necessitated research that explores student motivation for completing SETs. This study uses Vroom's [(1964). Work and motivation (3rd ed.). New York, NY: John Wiley & Sons] expectancy theory to frame student focus group responses regarding their motivations for completing and not completing paper and online SETs. Results show that students consider the following outcomes when deciding whether to complete SETs: (a) course improvement, (b) appropriate instructor tenure and promotion, (c) accurate instructor ratings are available to students, (d) spending reasonable amount of time on SETs, (e) retaining anonymity, (f) avoiding social scrutiny, (g) earning points and releasing grades, and (h) being a good university citizen. Results show that the lower online response rate is largely due to students’ differing feelings of obligation in the 2 formats. Students also noted that in certain situations, students often answer SETs insincerely.     

Honesty on student evaluations of teaching: effectiveness,purpose, and timing matter!

Student evaluations of teaching (SETs) are an important point of assessment for faculty in curriculum development, tenure and promotion decisions, and merit raises. Faculty members utilise SETs to gain feedback on their classes and, hopefully, improve them. The question of the validity of student responses on SETs is a continuing debate in higher education. The current study uses data from two universities (n = 596) to determine whether and under what conditions students are honest on in-class and online SETs, while also assessing their knowledge and attitudes about SETs. Findings reveal that, while students report a high level of honesty on SETs, they are more likely to be honest when they believe that evaluations effectively measure the quality of the course, the results improve teaching and benefit students rather than the administration, and when they are given at the end of the term. Honesty on evaluations is not associated with socio-demographic characteristics.     

Correlations,trends and potential biases among publicly accessible web-based student evaluations of teaching: a large-scale study of RateMyProfessors.com data

Student evaluations of teaching are widely adopted across academic institutions, but there are many underlying trends and biases that can influence their interpretation. Publicly accessible web-based student evaluations of teaching are of particular relevance, due to their widespread use by students in the course selection process and the quantity of data available for analysis. In this study, data from the most popular of these websites, RateMyProfessors.com, is analysed for correlations between measures of instruction quality, easiness, physical attractiveness, discipline and gender. This study of 7,882,980 RateMyProfessors ratings (from 190,006 US professors with at least 20 student ratings) provides further insight into student perceptions of academic instruction and possible variables in student evaluations. Positive correlations were observed between ratings of instruction quality and easiness, as well as between instruction quality and attractiveness. On average, professors in science and engineering disciplines have lower ratings than in the humanities and arts. When looking at RateMyProfessors as a whole, the effect of a professor’s gender on rating criteria is small but statistically significant. When analysing the data as a function of discipline, however, the effects of gender are significantly more pronounced, albeit more complex. The potential implications are discussed.     

A path analysis of factors often found to be related to student ratings of teaching effectiveness

The relationships among several variables outside of the instructor's classroom control and student ratings of teaching effectiveness are investigated in a causal network. Student ratings are relatively independent of external variables. Students may be able to take into account more factors than generally assumed when they rate their instructors.     

大学生参与教师教学评价的调查研究

大学生参与教师教学评价是提升高校教育教学质量的重要手段,也是大学生发挥主体作用的具体体现。从问卷和访谈调查中发现,高校虽然普遍开展了学生评教活动,但在评价的时机、形式、内容、结果利用以及师生的认识和态度等方面,仍然存在着一些不容忽视的问题。高等学校应拓宽教学评价的内容,开辟学生参与评教的多种途径,开发适合各类课程的评价工具,使学生的评教真正有助于教师的教学改革。     

Academics' resistance to summative peer review of teaching: questionable rewards and the importance of student evaluations

This study draws from 30 semi-structured interviews with tenure-track faculty members in a research-intensive university to examine their lack of engagement in the summative peer review of teaching. Findings indicate that most academics in the study do not think peer review outcomes contribute meaningfully to decisions about career advancement and believe that, in comparison, student evaluation of teaching scores matter more. The findings suggest that faculty member resistance to summative peer reviews will persist unless academics are confident that the results will be seriously considered in decisions about tenure and promotion. This article also contends that senior administrators should provide further clarity about the purpose and use of peer review outcomes in high-stakes career decisions.     

Beyond Average: Contemporary statistical techniques for analysing student evaluations of teaching

Student evaluations of teaching (SETs) have been used to evaluate higher education teaching performance for decades. Reporting SET results often involves the extraction of an average for some set of course metrics, which facilitates the comparison of teaching teams across different organisational units. Here, we draw attention to ongoing problems with the naive application of this approach. Firstly, a specific average value may arise from data that demonstrates very different patterns of student satisfaction. Furthermore, the use of distance measures (e.g. an average) for ordinal data can be contested, and finally, issues of multiplicity increasingly plague approaches using hypothesis testing. It is time to advance the methodology of the field. We demonstrate how multinomial distributions and hierarchical Bayesian methods can be used to contextualise the SET scores of a course to different organisational units and student cohorts, and then show how this approach can be used to extract sensible information about how a distribution is changing.     

Framing student evaluations of university learning and teaching: discursive strategies and textual outcomes

Evaluation in higher education is an evolving social practice; that is, it involves what people, institutions and broader systems do and say, how they do and say it, what they value, the effects of these practices and values, and how meanings are ascribed. The textual products (verbal, written, visual and gestural) that inform and are produced by, for and through evaluative practices are important, as they promulgate particular kinds of meanings and values in specific contexts. This paper reports on an exploratory study that sought to investigate, using discourse analysis, the types of evaluative practices that were ascribed value, and the student responses that ensued, in different evaluative instruments. Findings indicate that when a reflective approach is taken to evaluation, students’ responses are more considered, they interrogate their own engagement in the learning context and they are more likely to demonstrate reconstructive thought. These findings have implications for reframing evaluation as reflective learning.     

Maximising resource allocation in the teaching laboratory: understanding student evaluations of teaching assistants in a team-based teaching format

Minimal research papers have investigated the use of student evaluations on the laboratory, a learning medium usually run by teaching assistants with little control of the content, delivery and equipment. Finding the right mix of teaching assistants for the laboratory can be an onerous task due to the many skills required including theoretical and practical know-how, troubleshooting, safety and class management. Using larger classes with multiple teaching assistants, a team-based teaching (TBT) format may be advantageous. A rigorous three-year study across twenty-five courses over repetitive laboratory classes is analysed using a multi-level statistical model considering students, laboratory classes and courses. The study is used to investigate the effectiveness of the TBT format, and quantify the influence each demonstrator has on the laboratory experience. The study found that TBT is effective and the lead demonstrator most influential, influencing up to 55% of the laboratory experience evaluation.     

论学生评教及高校教学质量保障体系的改善

学生评教是高校教学质量保障的一项根本制度,它的建立使学生拥有了表达教学意愿的常规渠道,有助于高校树立服务学生的办学宗旨。我国高校开展学生评教工作的时间不长,在实际工作中还存在学生代人评教、形式意义大于实质意义、评教结果被误用、制度设计存在漏洞等问题。为此,应从树立正确的学生评教观、构建科学的学生评教制度、借鉴国外的先进经验等方面,改进我国高校的学生评教制度,完善我国高校教学质量保障体系。     

Effects of anonymity and retaliatory potential on student evaluations of faculty performance

The present study, employing a 2×2 design, examines the independent and interactive effects of two experimentally manipulated variables, Anonymity (A) and Retaliatory Potential (RP), on students' ratings of faculty performance. Data were obtained from 188 students at a large midwestern university using an instrument that had been specially developed for the study. A hypothesized main effect was found for A: The professor was rated more positively by students completing signed ratings than those completing ratings anonymously. The data showed no support, however, for the hypothesized RP main effect or the A×RP interactive effect. Practical implications of the study's findings are discussed.     

Predicting student evaluations of teaching using decision tree analysis

Abstract

This study uses decision tree analysis to determine the most important variables that predict high overall teaching and course scores on a student evaluation of teaching (SET) instrument at a large public research university in the United States. Decision tree analysis is a more robust and intuitive approach for analysing and interpreting SET scores compared to more common parametric statistical approaches. Variables in this analysis included individual items on the SET instrument, self-reported student characteristics, course characteristics and instructor characteristics. The results show that items on the SET instrument that most directly address fundamental issues of teaching and learning, such as helping the student to better understand the course material, are most predictive of high overall teaching and course scores. SET items less directly related to student learning, such as those related to course grading policies, have little importance in predicting high overall teaching and course scores. Variables irrelevant to the construct, such as an instructor’s gender and race/ethnicity, were not predictive of high overall teaching and course scores. These findings provide evidence of criterion and discriminant validity, and show that high SET scores do not reflect student biases against an instructor’s gender or race/ethnicity.     

Stability and correlates of student evaluations of teaching at a Chinese university

This paper examines the stability and validity of a student evaluations of teaching (SET) instrument used by the administration at a university in the PR China. The SET scores for two semesters of courses taught by 435 teachers were collected. Total 388 teachers (170 males and 218 females) were also invited to fill out the 60‐item NEO Five‐Factor Inventory together with a demographic information questionnaire. The SET responses were found to have very high internal consistency and confirmatory factor analysis supported a one‐factor solution. The SET re‐test correlations were .62 for both the teachers who taught the same course (n = 234) and those who taught a different course in the second semester (n = 201). Linguistics teachers received higher SET scores than either social science or humanities or science and technology teachers. Student ratings were significantly related to Neuroticism and Extraversion. Regression results showed that the Big‐Five personality traits as a group explained only 2.6% of the total variance of student ratings and academic discipline explained 12.7% of the total variance of student ratings. Overall the stability and validity of SET was supported and future uses of SET scores in the PR China are discussed.     

Student evaluations of teaching effectiveness: a Nigerian investigation

An investigation is reported which tests the applicability of two American instruments designed to assess tertiary students' evaluations of teaching effectiveness with 158 Nigerian undergraduates. The scales were found to have generally high internal consistency reliability coefficients, most of the items were seen to be appropriate, and every item was considered of importance by at least some of the students. In addition, all but the Workload/Difficulty items clearly differentiated between good and poor lecturers. Factor analysis found a strong main factor of teaching effectiveness plus a minor factor referring to course workload and difficulty. Further analysis generally supported the convergent and discriminant validity of those scales hypothesized to measure similar or dissimilar components of effective teaching. However, this analysis supported the factor analytic results as more overlap between aspects of teaching skill and enthusiasm was found than has been evident in Western studies. Thus there must be doubt about the cross-cultural validity of a multidimensional model of teaching effectiveness.     

Exploring the impact of faculty reflection on weekly student evaluations of teaching

This exploratory study considered Larrivee’s assessment of teachers’ reflective practice levels by using a formative, weekly, online student evaluation of teaching (SET) tool through a virtual learning environment (VLE) as a means to encourage reflective practice. In‐depth interviews were conducted with six faculty members in three departments at a university college in the UK. The study found that: (a) faculty who experienced surface‐level reflection were more likely to have a reactive reflection style; and (b) faculty who experienced higher levels of reflection were more likely to have a proactive reflection style. Overall, the tool was found to be an efficient means of encouraging reflection by all participants and demonstrated that reflective practice could come about as a result of these weekly formative SETs. The study concludes with suggestions for academic development and future research on reflection that could be conducted using SETs via a VLE.