Correlations,trends and potential biases among publicly accessible web-based student evaluations of teaching: a large-scale study of RateMyProfessors.com data

Student evaluations of teaching are widely adopted across academic institutions, but there are many underlying trends and biases that can influence their interpretation. Publicly accessible web-based student evaluations of teaching are of particular relevance, due to their widespread use by students in the course selection process and the quantity of data available for analysis. In this study, data from the most popular of these websites, RateMyProfessors.com, is analysed for correlations between measures of instruction quality, easiness, physical attractiveness, discipline and gender. This study of 7,882,980 RateMyProfessors ratings (from 190,006 US professors with at least 20 student ratings) provides further insight into student perceptions of academic instruction and possible variables in student evaluations. Positive correlations were observed between ratings of instruction quality and easiness, as well as between instruction quality and attractiveness. On average, professors in science and engineering disciplines have lower ratings than in the humanities and arts. When looking at RateMyProfessors as a whole, the effect of a professor’s gender on rating criteria is small but statistically significant. When analysing the data as a function of discipline, however, the effects of gender are significantly more pronounced, albeit more complex. The potential implications are discussed.     

Updating RIGs: including the systematic influence of online study on student evaluation of teaching

Based on student evaluation of teaching (SET) ratings from 1,432 units of study over a period of a year, representing 74,490 individual sets of ratings, and including a significant number of units offered in wholly online mode, we confirm the significant influence of class size, year level, and discipline area on at least some SET ratings. We also find online mode of offer to significantly influence at least some SET ratings. We reveal both the statistical significance and effect sizes of these influences, and find that the magnitudes of the effect sizes of all factors are small, but potentially cumulative. We also show that the influence of online mode of offer is of the same magnitude as the other 3 factors. These results support and extend the rating interpretation guides (RIGs) model proposed by Neumann and colleagues, and we present a general method for the development of a RIGs system.     

Stability and correlates of student evaluations of teaching at a Chinese university

This paper examines the stability and validity of a student evaluations of teaching (SET) instrument used by the administration at a university in the PR China. The SET scores for two semesters of courses taught by 435 teachers were collected. Total 388 teachers (170 males and 218 females) were also invited to fill out the 60‐item NEO Five‐Factor Inventory together with a demographic information questionnaire. The SET responses were found to have very high internal consistency and confirmatory factor analysis supported a one‐factor solution. The SET re‐test correlations were .62 for both the teachers who taught the same course (n = 234) and those who taught a different course in the second semester (n = 201). Linguistics teachers received higher SET scores than either social science or humanities or science and technology teachers. Student ratings were significantly related to Neuroticism and Extraversion. Regression results showed that the Big‐Five personality traits as a group explained only 2.6% of the total variance of student ratings and academic discipline explained 12.7% of the total variance of student ratings. Overall the stability and validity of SET was supported and future uses of SET scores in the PR China are discussed.     

Honesty on student evaluations of teaching: effectiveness,purpose, and timing matter!

Student evaluations of teaching (SETs) are an important point of assessment for faculty in curriculum development, tenure and promotion decisions, and merit raises. Faculty members utilise SETs to gain feedback on their classes and, hopefully, improve them. The question of the validity of student responses on SETs is a continuing debate in higher education. The current study uses data from two universities (n = 596) to determine whether and under what conditions students are honest on in-class and online SETs, while also assessing their knowledge and attitudes about SETs. Findings reveal that, while students report a high level of honesty on SETs, they are more likely to be honest when they believe that evaluations effectively measure the quality of the course, the results improve teaching and benefit students rather than the administration, and when they are given at the end of the term. Honesty on evaluations is not associated with socio-demographic characteristics.     

Beyond Average: Contemporary statistical techniques for analysing student evaluations of teaching

Student evaluations of teaching (SETs) have been used to evaluate higher education teaching performance for decades. Reporting SET results often involves the extraction of an average for some set of course metrics, which facilitates the comparison of teaching teams across different organisational units. Here, we draw attention to ongoing problems with the naive application of this approach. Firstly, a specific average value may arise from data that demonstrates very different patterns of student satisfaction. Furthermore, the use of distance measures (e.g. an average) for ordinal data can be contested, and finally, issues of multiplicity increasingly plague approaches using hypothesis testing. It is time to advance the methodology of the field. We demonstrate how multinomial distributions and hierarchical Bayesian methods can be used to contextualise the SET scores of a course to different organisational units and student cohorts, and then show how this approach can be used to extract sensible information about how a distribution is changing.     

Eliciting student feedback for course development: the application of a qualitative course evaluation tool among business research students

Student evaluations of teaching and learning are playing an increasingly important role in the delivery of high-quality, student-centred education. Insights into student perceptions of their learning experience provide important information that can be used to inform course design and development. The majority of course evaluations take the form of quantitative surveys, but research suggests that a reliance on survey data alone can be problematic from a teaching and learning perspective. Qualitative course evaluations have been cited as a viable alternative to quantitative evaluations, but less research has been conducted into their efficacy when compared to quantitative evaluations. The study on which this article reports attempted to contribute to addressing this shortcoming by describing and assessing a novel approach to eliciting qualitative feedback from students in a research methodology course at a higher education institution in South Africa. Conventional content analysis was used to analyse the qualitative feedback received from students. The qualitative course evaluation approach was then appraised in terms of the degree to which it has the potential to overcome the shortcomings associated with quantitative course evaluations and the extent to which the information gathered could be used to improve the design and delivery of the academic programme.     

Student evaluations of teaching: combining the meta-analyses and demonstrating further evidence for effective use

There is a plethora of research on student evaluations of teaching (SETs) regarding their validity, susceptibility to bias, practical use and effective implementation. Given that there is not one study summarising all these domains of research, a comprehensive overview of SETs was conducted by combining all prior meta-analyses related to SETs. Eleven meta-analyses were identified, and nine meta-analyses covering 193 studies were included in the analysis, which yielded a small-to-medium overall weighted mean effect size (r = .26) between SETs and the variables studied. Findings suggest that SETs appear to be valid, have practical use that is largely free from gender bias and are most effective when implemented with consultation strategies. Research, teaching and policy implications are discussed.     

美国高等教育扩张时期的学生评价研究

学生评价肇始于 2 0世纪 2 0年代的哈佛大学 ,兴盛于美国高等教育大众化后期。大量研究表明 ,学生评价完全可以有效地评价高校教师的教学质量 ,但班级大小、学生期望、学科领域、课程类型、教龄、教师个性等对评分也存在一定影响。学生评价的偏差也值得引起我国高教界的关注。     

Assessment & Evaluation in Higher Education Using the margin of error statistic to examine the effects of aggregating student evaluations of teaching

We proposed an extended form of the Govindarajulu and Barnett margin of error (MOE) equation and used it with an analysis of variance experimental design to examine the effects of aggregating student evaluations of teaching (SET) ratings on the MOE statistic. The interpretative validity of SET ratings can be questioned when the number of students enrolled in a course is low or when the response rate is low. A possible method of improving interpretative validity is to aggregate SET ratings data from two or more courses taught by the same instructor. Based on non-parametric comparisons of the generated MOE, we found that aggregating course evaluation data from two courses reduced the MOE in most cases. However, significant improvement was only achieved when combining course evaluation data for the same instructor for the same course. Significance did not hold when combining data from different courses. We discuss the implications of our findings and provide recommendations for practice.     

An empirical test of the validity of student evaluations of teaching made on RateMyProfessors.com

The present article examined the validity of public web‐based teaching evaluations by comparing the ratings on RateMyProfessors.com for 126 professors at Lander University to the institutionally administered student evaluations of teaching and actual average assigned GPAs for these same professors. Easiness website ratings were significantly positively correlated with actual assigned grades. Further, clarity and helpfulness website ratings were significantly positively correlated with student ratings of overall instructor excellence and overall course excellence on the institutionally administered IDEA forms. The results of this study offer preliminary support for the validity of the evaluations on RateMyProfessors.com.     

Framing student evaluations of university learning and teaching: discursive strategies and textual outcomes

Evaluation in higher education is an evolving social practice; that is, it involves what people, institutions and broader systems do and say, how they do and say it, what they value, the effects of these practices and values, and how meanings are ascribed. The textual products (verbal, written, visual and gestural) that inform and are produced by, for and through evaluative practices are important, as they promulgate particular kinds of meanings and values in specific contexts. This paper reports on an exploratory study that sought to investigate, using discourse analysis, the types of evaluative practices that were ascribed value, and the student responses that ensued, in different evaluative instruments. Findings indicate that when a reflective approach is taken to evaluation, students’ responses are more considered, they interrogate their own engagement in the learning context and they are more likely to demonstrate reconstructive thought. These findings have implications for reframing evaluation as reflective learning.     

Teacher support,student motivation,student need satisfaction,and college teacher course evaluations: testing a sequential path model

Self‐determination theory posits that individuals who have basic psychological needs satisfied while engaging in an activity will be more likely to value and persist in that activity. Scholars in this area have also posited that autonomy‐supportive social contexts are top‐down determinants of individual need satisfaction. To understand better the progression from social to motivational to outcome variables, we tested a four‐step path model within a classroom setting. Data were collected from 220 students in an introductory journalism course, which was subdivided into 14 lab sections. Students were surveyed regarding their feelings about the course, the autonomy support of the instructor, the motivation they felt to engage in the course, and the levels of need satisfaction they experienced. Theoretically‐guided structural equation modeling produced a best‐fitting model in which teacher autonomy support predicted more self‐determined student motivation, which along with teacher autonomy support predicted greater student psychological need satisfaction, which led finally to higher predicted course grades and higher teacher‐course evaluations. Overall, the results replicate and significantly extend previous findings regarding need satisfaction and teacher‐course evaluations. Implications for pedagogy and educational interventions are discussed.     

Psychometric quality of a student evaluation of teaching survey in higher education

Student evaluations of teaching (SET) are used globally by higher education institutions for performance assessment of academic staff and evaluation of course quality. Higher education institutions commonly develop their own SETs to measure variables deemed relevant to them. However, ‘home-grown’ SETs are rarely assessed psychometrically. One potential consequence of this limitation is that an invalidated instrument may not provide accurate information for the intended purposes. Moreover, in the absence of psychometric assessment, the students’ voices collected by the SETs often fail to provide insight relative to their intended purpose. The present study evaluates a ‘home-grown’ SET using a Rasch model and confirmatory factor analysis. Our results identified weaknesses in two areas: the rating categories and the number of items used to measure the intended constructs. Suggestions are provided to address these weaknesses. This work provides an additional tool set for critical analysis of SET that is generally applicable for a variety of institutions, including those in Asia.     

新经济地理学产业集群理论发展及其评析

文章回顾了新经济地理学产业集群理论发展的历史过程，介绍其主要观点，探讨了其研究的新进展，并作出简要评析。     

Applying the scholarship of teaching and learning: student perceptions,behaviours and success online and face-to-face

This study compares student perceptions, learning behaviours and success in online and face-to-face versions of a Principles of Microeconomics course. It follows a Scholarship of Teaching and Learning (SoTL) approach by using a cycle of empirical analysis, reflection and action to improve the learning experience for students. The online course design involves 58 interactive narrated online modules, interactive online quizzes and biweekly online meetings with the instructor via video and voice-over-IP technology. Findings indicate that schedule flexibility motivates students to choose the online course format. Students in both learning environments felt they had high-quality communication with the instructor, while online students indicated limited peer-to-peer communication. Online students report studying more at home than face-to-face students, but not enough to compensate for the time face-to-face students spend in class. Reflecting on the findings, the authors suggest actions to improve the online course experience.     

The relationship between student engagement with online content and achievement in a blended learning anatomy course

Blended learning has become increasingly common in higher education. Recent findings suggest that blended learning achieves better student outcomes than traditional face‐to‐face teaching in gross anatomy courses. While face‐to‐face content is perceived as important to learning there is less evidence for the significance of online content in improving student outcomes. Students enrolled in a second‐year anatomy course from the physiotherapy (PT), exercise physiology (EP), and exercise science (ES) programs across two campuses were included (n = 500). A structural equation model was used to evaluate the relationship of prior student ability (represented by grade in prerequisite anatomy course) and final course grade and whether the relationship was mediated by program, campus or engagement with the online elements of the learning management system (LMS; proportion of documents and video segments viewed and number of interactions with discussion forums). PT students obtained higher grades and were more likely to engage with online course materials than EP and ES students. Prerequisite grade made a direct contribution to course final grade (P < 0.001) but was also mediated by engagement with LMS videos and discussion forums (P < 0.001). Student learning outcomes in a blended anatomy course can be predicted the by level of engagement with online content. Anat Sci Educ 11: 471–477. © 2017 American Association of Anatomists.     

Evaluation of a pedagogical model for student engagement in learning activities

Current research suggests that student engagement with academic schoolwork in secondary school classrooms is often insufficient. This issue is of relevance because student engagement is a prerequisite for acquiring knowledge and skills and is also a mediator of achievement and important life outcomes. In response to this, the present study evaluates a pedagogical model for stimulating student engagement in learning activities in secondary school contexts through an action research approach. This model is composed of five facilitators of engagement that, when considered in both the design and implementation of learning activities, can potentially contribute to stimulating behavioural, emotional and cognitive engagement in such activities. Nineteen Year 9 students of Spanish from a state school in England, their teacher and the researcher participated in two long-term, learning activities designed using the proposed pedagogical model. Data were collected over two school terms through qualitative student self-reports, interviews with students and their teacher and the researcher’s journal. Thematic analyses were used to examine the effects of the learning activities on student behavioural, emotional and cognitive engagement. The results seem to confirm that the application of the proposed pedagogical model could help promote thorough student engagement in learning activities. Findings also suggest that failing to support the model’s facilitators of engagement may have detrimental effects on overall student engagement in learning activities. The study contributes to the improvement of student engagement with academic schoolwork internationally by proposing an assessed pedagogical solution that could be implemented by secondary school teachers with relative ease.     

Advancing text-analysis to tap into the student voice: a proof-of-concept study

Abstract

Student evaluations of teaching and courses (SETs) are part of the fabric of tertiary education and quantitative ratings derived from SETs are highly valued by tertiary institutions. However, many staff do not engage meaningfully with SETs, especially if the process of analysing student feedback is cumbersome or time-consuming. To address this issue, we describe a proof-of-concept study to automate aspects of analysing student free text responses to questions. Using Quantext text analysis software, we summarise and categorise student free text responses to two questions posed as part of a larger research project which explored student perceptions of SETs. We compare human analysis of student responses with automated methods and identify some key reasons why students do not complete SETs. We conclude that the text analytic tools in Quantext have an important role in assisting teaching staff with the rigorous analysis and interpretation of SETs and that keeping teachers and students at the centre of the evaluation process is key.     

Correlation between grade point averages and student evaluation of teaching scores: taking a closer look

One of the most contentious potential sources of bias is whether instructors who give higher grades receive higher ratings from students. We examined the grade point averages (GPAs) and student ratings across 2073 general education religion courses at a large private university. A moderate correlation was found between GPAs and student evaluations of teaching (SETs); however, this global correlation did not hold true for individual teachers and courses. In fact, there was a large variance in the correlations between GPAs and SETs, including some teachers with a negative correlation and a large variance between courses.