Correlations,trends and potential biases among publicly accessible web-based student evaluations of teaching: a large-scale study of RateMyProfessors.com data

Student evaluations of teaching are widely adopted across academic institutions, but there are many underlying trends and biases that can influence their interpretation. Publicly accessible web-based student evaluations of teaching are of particular relevance, due to their widespread use by students in the course selection process and the quantity of data available for analysis. In this study, data from the most popular of these websites, RateMyProfessors.com, is analysed for correlations between measures of instruction quality, easiness, physical attractiveness, discipline and gender. This study of 7,882,980 RateMyProfessors ratings (from 190,006 US professors with at least 20 student ratings) provides further insight into student perceptions of academic instruction and possible variables in student evaluations. Positive correlations were observed between ratings of instruction quality and easiness, as well as between instruction quality and attractiveness. On average, professors in science and engineering disciplines have lower ratings than in the humanities and arts. When looking at RateMyProfessors as a whole, the effect of a professor’s gender on rating criteria is small but statistically significant. When analysing the data as a function of discipline, however, the effects of gender are significantly more pronounced, albeit more complex. The potential implications are discussed.     

The impact of individual consultation on student ratings of teaching

Abstract

The present study addressed the impact of individual consultation on teaching improvement as measured by changes in student ratings. Subjects included 91 professors who presented naturally for individual consultation services over a seven‐year period at the teaching centre of a Canadian university. Interventions by the consultant fell into three categories: 1) Feedback‐Consultation, 2) Feedback‐Consultation‐Class Observation, and 3) Feedback‐Consultation‐Class Observation and Student Consultation. End of term student ratings for the course that was the subject of the consultation were compared with student ratings for the same course taught between one and three years prior to the consultation service, and for the same course taught between one to three years following consultation. The results showed that, overall, consultation was effective in improving the quality of the consultees’ teaching, as evidenced by an increase in mean student ratings of instruction. This effect persisted post consultation. Not all intervention groups, however, showed the same pattern of results. Change was evident immediately after the intervention except in the case of brief consultation, although follow‐up data showed improved teaching for the latter group. Control data provided evidence that the change in student ratings post consultation could reasonably be attributed to consultation effects.     

Student Ratings and Professor Self-Ratings of College Teaching: Effects of Gender and Divisional Affiliation

Twenty female and 23 male professors at a liberal arts college participated along with their 803 undergraduate students in a questionnaire study of the effects of professor gender, student gender, and divisional affiliation on student ratings of professors and professor self-ratings. Students rated their professors on 26 questions tapping five teaching factors as well as overall teaching effectiveness. Professors rated themselves on the same questions as well as on nine exploratory ones. On student ratings, there were main effects for both professor gender (female professors were rated higher than male professors on the two interpersonal factors) and division (natural science courses were rated lowest on most factors). These patterns were qualified by significant interactions between professor gender and division. Although professor self-ratings varied by division, there were few significant correlations between professor self-ratings and students' ratings. Implications for future research are discussed.Appreciation goes to Laura Capotosto and Julie Phelan for their research assistance.Suzanne Montgomery is now at Widener Law School, Philadelphia, PA, USA.     

Student Grades and Rating of Teaching Quality

As part of a Faculty Review, students currently enrolled at the University were asked to rate the quality of the teaching they had previously experienced during their first year. About 900 students in four separate entry cohorts were involved, yielding a total of 28 separate ratings. These ratings were correlated with students’ examination grades. Between classes, there is an indication that rating of teaching is related to examination performance. Within classes, there is a positive and significant relationship between student grades and their rating of teaching only when the teaching is rated above average. The results are discussed.     

Book Reviews

Good teaching should be inclusive of all students. There are very strong arguments for making courses/units/modules as inclusive as possible, based on issues of equity and access. Inclusive teaching has been a catch cry in recent times and most universities have policies related to this issue. However, research into the effectiveness of measures taken to ensure that teaching caters to all students is rare. The scarcity of such information may be due to the difficulty in finding an appropriate method of evaluation. One of the means to evaluate teaching of a unit or course is to obtain feedback from students. Although information collected through this method may be subjected to biases, student perceptions can still provide useful data that can be incorporated into a broader evaluation system. This paper discusses an investigation into the inclusive nature of a large number of units offered at The University of Western Australia over 3 years. Student ratings in relation to the issue of inclusivity were also explored for possible influences of the year level of courses, broad discipline areas and student gender. The results of this study indicate that these three factors could affect how students view the inclusive nature of particular units.     

Sources of variance in end-of-course student evaluations

Abstract

Evaluation of college instructors often centers on course ratings; however, there is little evidence that these ratings only reflect teaching. The purpose of this study was to assess the relative importance of three facets of course ratings: instructor, course and occasion. We sampled 2,459 fully-crossed dyads from a large university where two instructors taught the same two courses at least twice in a 3-year period. Generalizability theory was used to estimate unconfounded variance components for instructor, course and occasion, as well as their interactions. Meta-analysis was used to summarize those estimates. Results indicated that a three-way interaction between instructor, course and occasion that includes measurement error accounted for the most variance in student ratings (24%), with instructor accounting for the second largest amount (22%). While instructor - and presumably teaching - accounted for substantial variance in student course ratings, factors other than instructor quality had a larger influence on student ratings.     

Understanding innovative teaching practice in higher education: a framework for reflection

ABSTRACT

In recent years there has been a resurgence of interest in innovation in teaching and learning in higher education. Now more than ever, academics are advised that engaging in educational innovation for the purposes of improved student learning is supported, valued, and rewarded. However, embracing innovative teaching practice requires academics to develop new skills and understandings, take on extra work, risk failure and invite disapproval from staff and students. Ultimately, focusing upon educational innovation rather than discipline research can be a risky career move if it is not undertaken strategically. This paper provides insight into the characteristics of two innovative academics from the discipline of science and explores the dimensions of innovation as it pertains to teaching and learning. A framework emerged from an analysis of their experience which identifies different types of educational innovators in higher education, discriminating between three individuals with regards to the level at which they seek to influence practice.     

How Fair are Student Ratings in Assessing the Teaching Performance of University Teachers?

This paper examines the effects of course characteristics on student ratings of university teaching. The multivariate effects of various course characteristics on student ratings of university teachers were examined by a 5×4×3×2×2 multivariate analysis of variance (MANOVA) with academic discipline, class size, level of course, type of course, and mode of study as independent variables and a set of six composite student ratings as the dependent variable. Data were obtained from a university in Hong Kong and consisted of 4996 course average ratings collected from 25 departments over two academic years. Significant differences were found in course average student ratings across academic disciplines, class size groups, course levels, types of course, and modes of study. The effect sizes of the largest differences between sub‐groups within each of the factors were also estimated. Academic discipline differences and class size were found to have the largest effect on student ratings, with effect sizes of 0.43–0.86 and 0.55–1.01 respectively. Implications of the findings are discussed.     

Updating RIGs: including the systematic influence of online study on student evaluation of teaching

Based on student evaluation of teaching (SET) ratings from 1,432 units of study over a period of a year, representing 74,490 individual sets of ratings, and including a significant number of units offered in wholly online mode, we confirm the significant influence of class size, year level, and discipline area on at least some SET ratings. We also find online mode of offer to significantly influence at least some SET ratings. We reveal both the statistical significance and effect sizes of these influences, and find that the magnitudes of the effect sizes of all factors are small, but potentially cumulative. We also show that the influence of online mode of offer is of the same magnitude as the other 3 factors. These results support and extend the rating interpretation guides (RIGs) model proposed by Neumann and colleagues, and we present a general method for the development of a RIGs system.     

Interparental Differences of Opinion and Children’s Academic Achievement

Abstract

The purpose of this study was to determine the effectiveness of elementary school interns trained in a summer program by the Stanford microteaching approach as compared with interns who received conventional classroom observation and student teaching experience. Contrary to expectation based on previous research findings, microteaching was not found to result in significantly higher ratings of teacher effectiveness either immediately after or a year after training. However, it was concluded that microteaching is an effective training strategy since it achieves similar results when compared with conventional training methods but in only one-fifth the time and with fewer administrative problems. An incidental finding was that pretraining ratings of teaching performance based on a brief videotaped lesson were generally good predictors of later ratings of teaching effectiveness.     

Occupational values and employment: A follow-up study of graduate men and women

Undergraduate job values have been reported as reflecting both the sex and social origins of the student. The present study assesses whether such differences are of any occupational significance. Ratings on 30 job attributes were collected from 310 final year undergraduates at universities and colleges throughout Britain. The four factors extracted from these ratings were then related to the job held at 26 years. Although the proportions of men and women differed between jobs, this difference was adequately explained by the job values reported by the students before graduation. There was no evidence that women had different values (at college) than men who were, at 26, to do a similar type of job. The origins of sex and social class differences in students' values are explored and it is concluded that there is a need for a new approach to careers education for the highly intelligent student of limited aspiration.     

Ratings of university teacher instruction: how much do student and course characteristics really matter?

Several student and course characteristics were examined in relation to student ratings of instruction. Students at a major Canadian university completed the Universal Student Ratings of Instruction instrument at the end of every course over a three‐year period, providing 371,131 student ratings. Analyses of between‐group differences indicate that students who attend class often and expect high grades provide high ratings of their instructors (p < .001). In addition, lab‐type courses receive higher ratings than lectures or tutorials, and courses in the social sciences receive higher ratings than courses in the natural sciences (p < .001). Regression analyses indicated, however, that student and course characteristics explain little variance in student ratings of their instructors (<7%). It is concluded that student ratings are more related to teaching instruction and behavior of the instructor than to these variables.     

Stability and correlates of student evaluations of teaching at a Chinese university

This paper examines the stability and validity of a student evaluations of teaching (SET) instrument used by the administration at a university in the PR China. The SET scores for two semesters of courses taught by 435 teachers were collected. Total 388 teachers (170 males and 218 females) were also invited to fill out the 60‐item NEO Five‐Factor Inventory together with a demographic information questionnaire. The SET responses were found to have very high internal consistency and confirmatory factor analysis supported a one‐factor solution. The SET re‐test correlations were .62 for both the teachers who taught the same course (n = 234) and those who taught a different course in the second semester (n = 201). Linguistics teachers received higher SET scores than either social science or humanities or science and technology teachers. Student ratings were significantly related to Neuroticism and Extraversion. Regression results showed that the Big‐Five personality traits as a group explained only 2.6% of the total variance of student ratings and academic discipline explained 12.7% of the total variance of student ratings. Overall the stability and validity of SET was supported and future uses of SET scores in the PR China are discussed.     

Similarity of student ratings across instructors,courses, and time

This study raised three questions about the similarity or generalizability of student ratings of courses and instructors. First, how stable are student ratings of the same instructor giving the same course during twodifferent semesters? Second, how similar are student ratings of the same instructor in twodifferent courses? Third, how similar are student ratings of a given course being taught bydifferent instructors? Instances were identified in which student ratings on seven different factors were available for pairs of courses for each of these questions. For the case of the same instructor — same course—different semesters, student ratings were reasonably similar (median r for seven factors about 0.70). For the case of the same instructor—different courses, the median r was surprisingly low — about 0.40. For the case of the same course—different instructors, substantial correlations were obtained for some factors and insignificant correlations for other factors. Implications of these findings for practical use of student ratings and suggestions for further research in the area are discussed.     

Teacher behavior and student outcomes: Results of a European study

This study investigates the extent to which the factors included in the dynamic model of educational effectiveness are associated with student achievement gains in six different European countries. At classroom level, the dynamic model refers to eight factors relating to teacher behavior in the classroom: orientation, structuring, questioning, teaching-modeling, application, management of time, teacher role in making classroom a learning environment, and classroom assessment. This paper presents results concerned with the impact of the teacher factors on student achievement. In each participating country (i.e., Belgium/Flanders, Cyprus, Germany, Greece, Ireland, and Slovenia), a sample of at least 50 primary schools (n?=?334) was drawn. Written tests in mathematics and science were administered to all grade 4 students (n?=?10,742) at the beginning and at the end of the school year 2010–2011. Students were also asked to complete a questionnaire concerning the eight teacher factors of the dynamic model. Structural equation modeling techniques were used to test the construct validity of the student questionnaire. Both across and within country analyses revealed that student ratings are reliable and valid for measuring the functioning of the teacher factors included in the dynamic model. Multilevel analyses revealed that teacher factors are associated with student achievement gains in mathematics and science. Implications for the development of educational effectiveness research and for improving quality of teaching are drawn.     

Is distance teaching more work or less work?

Abstract

This article presents data collected in a year‐long study in which the author and his assistants kept detailed records of their time spent teaching and maintaining two comparable university courses. One course was offered online to adult professionals away from campus, the other offered to undergraduates in traditional classrooms on campus. The courses had similar learning objectives, similar student activities, and equally favorable ratings by students. Both were mature courses that required only routine maintenance and revision. The data do not support the widely held belief that teaching an asynchronous online course requires more effort than teaching a comparable synchronous classroom course. Although the distance course required more frequent attention, the total teaching and maintenance time spent per student was less than that required to teach and maintain the classroom course. Categories of teaching tasks are also compared.     

Salary increases and teaching effectiveness

The relationship between salary increases and student ratings of teaching effectiveness was studied for a sample of 266 faculty members at Kansas State University. Three measures of teaching effectiveness (student progress in meeting relevant course objectives, liking the instructor, and appreciation of the field of study) and two salary criteria (percent and dollar increases) were used. Several measures of the emphasis on the teaching function were used to determine if this variable moderated the relationship between salaries and student ratings. The effect of discipline was also examined.In general, there was a modest but significant correlation between ratings of teaching effectiveness and percent salary increase. The amount of emphasis given to the teaching function was a significant determinant of the strength of this relationship. Correlations were generally more pronounced in social science and humanities than in the science areas. Student motivation was highly correlated with effectiveness ratings, but was not regularly taken into account in salary recommendations.     

Improving quality and equity in schools in socially disadvantaged areas

Background: Recent effectiveness studies have investigated the relationship between two dimensions of effectiveness – namely, quality and equity. Specifically, the question of whether effective schools can also reduce the initial differences in student outcomes attributed to student background factors has been examined. In this context, the Dynamic Approach to School Improvement (DASI) makes use of theory and the research findings of effectiveness studies to try to improve school effectiveness in terms of quality and equity.

Purpose: This study aimed to examine whether the implementation of DASI in primary schools in socially disadvantaged areas in four European countries (Cyprus, England, Greece and Ireland) was able to promote student learning outcomes in mathematics and to reduce the impact of student background factors on student achievement in mathematics.

Design and methods: A sample of 72 primary schools across the four countries was randomly split into experimental and control groups. At the beginning and at the end of the school year, mathematics tests were administered to all students of Grades 4–6 (n = 5560; student ages 9–12 years). The experimental group made use of DASI. Within-country multilevel regression analyses were conducted to evaluate the impact of the intervention and search for interaction effects between the use of DASI and student background factors on final achievement.

Results: In each country, the experimental group achieved better results in mathematics than the control group. At the beginning of the intervention, the achievement gap based on socio-economic status (SES) was equally large in the experimental and the control groups. Only in the experimental group did the achievement gap based on SES become smaller. However, DASI was not found to have an effect on equity when the equity dimension was examined by focusing on the achievement gap based on either gender or ethnicity.

Conclusions: Implications of findings are drawn and the importance of measuring equity in terms of student achievement gaps based on different background factors, rather than only on SES, is emphasised. We propose the evaluation of the impact of interventions on promoting equity by the use of various criteria.     

The impact of individual consultation on the teaching of younger versus older faculty

This study examines the effect of age of instructor on student ratings of teaching performance after individual consultation. Instructors self-presented over an 11-year period at a large Canadian university teaching service, where each received one of three interventions. End of term student ratings of the teaching of younger and older instructors are compared before consultation, immediately post consultation, and 1–3 years after year of consultation. Younger faculty obtained significantly improved ratings immediately after consultation, while older faculty achieved significant rating increases 1–3 years post consultation. Results from an earlier study on the impact of individual consultation on teacher ratings are reevaluated using this larger sample of faculty. Generally, results in this analysis parallel the original research. Consultation produced changes in student ratings both immediately after consultation as well as longitudinally, thus confirming the utility of intervention in producing enduring pedagogical improvements. Control analyses ensured that improvements were a result of the interventions and not an artefact of time.     

Initial Academic Momentum and Student Success: Comparing 4- and 2-Year Students

ABSTRACT

Today, many full-time students spend one, two, or more additional years completing a 2-year associate or 4-year bachelor degree. In order to reduce this time frame, many institutions are utilizing a 15 to Finish campaign to encourage full-time students to enroll in 15 credits each semester. The first purpose of this study was to examine if there is a relationship between the number of credit hours enrolled in the first semester and the number of credits earned at the end of the first year. Concurrently, it examined how a student’s initial academic momentum (i.e., the number of credit hours earned in the first year) relates to student persistence and degree completion. The second purpose was to determine if there was a relationship between a student’s socioeconomic status and the number of credit hours enrolled in the first semester—and if it is feasible for low-income students, who primarily attend community colleges, to afford additional credit hours. The results showed that more students who earned 30 credit hours in the first academic year persisted to the second year and earned a degree than students who did not earn 30 credit hours. Also, the results showed that the majority of low-income community college students had enough Federal Pell Grant funds to pay for the additional tuition cost. Given these results, practical and policy implications are discussed.