Student evaluations of teaching: combining the meta-analyses and demonstrating further evidence for effective use

There is a plethora of research on student evaluations of teaching (SETs) regarding their validity, susceptibility to bias, practical use and effective implementation. Given that there is not one study summarising all these domains of research, a comprehensive overview of SETs was conducted by combining all prior meta-analyses related to SETs. Eleven meta-analyses were identified, and nine meta-analyses covering 193 studies were included in the analysis, which yielded a small-to-medium overall weighted mean effect size (r = .26) between SETs and the variables studied. Findings suggest that SETs appear to be valid, have practical use that is largely free from gender bias and are most effective when implemented with consultation strategies. Research, teaching and policy implications are discussed.     

Student evaluations of teaching: perceptions of faculty based on gender,position, and rank

The current study explores the feelings and thoughts that faculty have about their student evaluations of teaching (SET). To assess the perceptions of SETs, all teaching faculty in one college at a western Land Grant University were asked to complete an anonymous online survey. The survey included demographic questions (i.e. gender; rank such as assistant, associate, and full professor; and positions like non-tenure track, tenure track, and tenured) as well as questions related to faculty's feelings while reading their SETs. While minimal differences were found in responses based on rank or position, several differences were found based on faculty gender. Overall, female faculty appear to be more negatively impacted by student evaluations than male faculty. These gender differences support previous research that suggests males and females receive and react differently to personal evaluation. Resultant suggestions include modifying surveys from anonymous to confidential and offering professional development training for faculty.     

How reliable are students’ evaluations of teaching quality? A variance components approach

The inter-rater reliability of university students’ evaluations of teaching quality was examined with cross-classified multilevel models. Students (N = 480) evaluated lectures and seminars over three years with a standardised evaluation questionnaire, yielding 4224 data points. The total variance of these student evaluations was separated into the variance components of courses, teachers, students and the student/teacher interaction. The substantial variance components of teachers and courses suggest reliability. However, a similar proportion of variance was due to students, and the interaction of students and teachers was the strongest source of variance. Students’ individual perceptions of teaching and the fit of these perceptions with the particular teacher greatly influence their evaluations. This casts some doubt on the validity of student evaluations as indicators of teaching quality and suggests that aggregated evaluation scores should be used with caution.     

Prediction-Based Student Evaluations of Teaching as an Alternative to Traditional Opinion-Based Evaluations

Abstract

The validity of traditional opinion-based student evaluations of teaching (SETs) may be compromised by inattentive responses and low response rates due to evaluation fatigue, and/or by personal response bias. To reduce the impact of evaluation fatigue and personal response bias on SETs, this study explores peer prediction-based SETs as an alternative to opinion-based SETs in a multicultural environment. The results suggest that statistically significant fewer respondents are needed to reach stable average outcomes when peer prediction-based SETs are used than when opinion-based SETs are used. This implies that peer prediction-based SETs could reduce evaluation fatigue, as not all students would need to do each evaluation. The results also report that the peer prediction-based method significantly reduces the bias evident in the opinion-based method, in respect of gender and prior academic performance. However, in respect of the cultural variables, race and home language, bias was identified in the peer prediction-based method, where none was evident in the opinion-based method. These observations, interpreted through the psychology literature on the formulation of perceptions of others, imply that although peer prediction-based SETs may in some instances reduce some personal response bias, it may introduce the perceived bias of others.     

The effect of incentives and other instructor-driven strategies to increase online student evaluation response rates

Course evaluations (often termed student evaluations of teaching or SETs) are pervasive in higher education. As SETs increasingly shift from pencil-and-paper to online, concerns grow over the lower response rates that typically accompany online SETs. This study of online SET response rates examined data from 678 faculty respondents and student response rates from an entire semester. The analysis focused on those tactics that faculty employ to raise response rates for their courses, and explored instructor and course characteristics as contributing factors. A comprehensive regression model was evaluated to determine the most effective tactics and characteristics. Using incentives had the most impact on response rates. Other effective tactics that increase response rates include reminding students to take the evaluation, explaining how the evaluations would be used to improve instruction, sending personal emails and posting reminders on Blackboard^®. Incentives are not widely used; however, findings suggest that non-point incentives work as well as point-based ones, as do simple-to-administer minimum class-wide response rate expectations (compared to individual completion).     

Evaluate to improve: useful approaches to student evaluation

Many teachers in higher education use feedback from students to evaluate their teaching, but only some use these evaluations to improve their teaching. One important factor that makes the difference is the teacher’s approach to their evaluations. In this article, we identify some useful approaches for improving teaching. We conducted focus groups with award-winning university teachers who use student evaluations to improve their teaching, and we identified how they approach their evaluation data. We found that these teachers take a reflective approach, aiming for constant improvement, and see their evaluation data as formative feedback, useful for improving learning outcomes for their students. We summarise this as the improvement approach, and we offer it for other teachers to emulate. We argue that if teachers take this reflective, formative, student-centred approach, they can also use student evaluations to improve their teaching, and this approach should be fostered by institutions to encourage more teachers to use student evaluations to improve their teaching.     

Expectancy theory outcomes and student evaluations of teaching

As student evaluation of teaching (SET) instruments are increasingly administered online, research has found that the response rates have dropped significantly. Validity concerns have necessitated research that explores student motivation for completing SETs. This study uses Vroom's [(1964). Work and motivation (3rd ed.). New York, NY: John Wiley & Sons] expectancy theory to frame student focus group responses regarding their motivations for completing and not completing paper and online SETs. Results show that students consider the following outcomes when deciding whether to complete SETs: (a) course improvement, (b) appropriate instructor tenure and promotion, (c) accurate instructor ratings are available to students, (d) spending reasonable amount of time on SETs, (e) retaining anonymity, (f) avoiding social scrutiny, (g) earning points and releasing grades, and (h) being a good university citizen. Results show that the lower online response rate is largely due to students’ differing feelings of obligation in the 2 formats. Students also noted that in certain situations, students often answer SETs insincerely.     

Student evaluations of teaching: the impact of faculty procedures on response rates

In the last 10–15 years, many institutions of higher education have switched from paper-and-pencil methods to online methods of administering student evaluations of teaching (SETs). One consequence has been a significant reduction in the response rates to such instruments. The current study was conducted to identify whether offering in-class time to students to complete online SETs would increase response rates. A quasi-experiment (nonequivalent group design) was conducted in which one group of tenured faculty instructed students to bring electronic devices with internet capabilities on a specified day and offered in-class time to students to complete online SETs. A communication protocol for faculty members’ use was developed and implemented. A comparison group of tenured faculty who did not offer in-class time for SET completion was identified and the difference-in-differences method was used to compare the previous year’s response rates for the same instructor teaching the same course across the two groups. Response rates were substantially higher when faculty provided in-class time to students to complete SETs. These results indicate that high response rates can be obtained for online SETs submitted by students in face-to-face classes if faculty communicate the importance of SETs in both their words and actions.     

Advancing text-analysis to tap into the student voice: a proof-of-concept study

Abstract

Student evaluations of teaching and courses (SETs) are part of the fabric of tertiary education and quantitative ratings derived from SETs are highly valued by tertiary institutions. However, many staff do not engage meaningfully with SETs, especially if the process of analysing student feedback is cumbersome or time-consuming. To address this issue, we describe a proof-of-concept study to automate aspects of analysing student free text responses to questions. Using Quantext text analysis software, we summarise and categorise student free text responses to two questions posed as part of a larger research project which explored student perceptions of SETs. We compare human analysis of student responses with automated methods and identify some key reasons why students do not complete SETs. We conclude that the text analytic tools in Quantext have an important role in assisting teaching staff with the rigorous analysis and interpretation of SETs and that keeping teachers and students at the centre of the evaluation process is key.     

Rate my attitude: research agendas and RateMyProfessor scores

The literature on student evaluations of teaching (SETs) generally presents two opposing camps: those who believe in the validity and usefulness of SETs, and those who do not. Some researchers have suggested that ‘SET deniers’ resist SETs because of their own poor SET results. To test this hypothesis, I analysed essays by 230 SET researchers (170 lead authors) and classified the researchers as having negative, neutral or positive attitudes towards SETs. I retrieved their RateMyProfessors.com (RMP) scores and, using logistic regression, found that lead authors with negative attitudes towards SETs were 14 times more likely to score below an estimated RMP average than lead authors with positive attitudes towards SETs. Co-authors and researchers with neutral attitudes, on the other hand, did not significantly differ from the RMP average. These results suggest that personal attitudes towards SETs may drive research findings.     

Research Productivity and Student Evaluation of Teaching in Social Science Classes: A Research Note

Research on the relationship between research productivity and student evaluations of teaching (SETs) has been marked by several shortcomings. First, research typically fails to check and adjust for nonlinear distributions in research productivity. Since approximately 15% of researchers account for most articles and citations (e.g., Zuckerman, H., Handbook of Sociology, Sage Publications, Newbury Park, CA, pp. 511–574, 1988), this failure might explain weak or nonsignificant findings in some of the past research. Second, the unit of analysis is typically the instructor, not the class. Since top researchers might disproportionately teach small classes at the graduate level, and that SETs are usually higher in such classes, the small relationships between research excellence and SETS found in previous research may be spurious. The present study corrects for each of these issues. It analyzes data from 167 classes in the social sciences and on 65 faculty. The quality of research productivity (raw citations/post-PhD year) is not related to SETs. However, when the distribution of citations is corrected for skewness, a significant positive relationship between research productivity and SETs emerges. This relationship survives controls for course and instructor characteristics, and holds for both the faculty member and the class as units of analysis. This is the first systematic investigation to demonstrate a significant relationship between the quality of research (citations) and SETs.     

Student personality differences are related to their responses on instructor evaluation forms

The relation of student personality to student evaluations of teaching (SETs) was determined in a sample of 144 undergraduates. Student Big Five personality variables and core self-evaluation (CSE) were assessed. Students rated their most preferred instructor (MPI) and least preferred instructor (LPI) on 11 common evaluation items. Pearson and partial correlations simultaneously controlling for six demographic variables, Extraversion, Conscientiousness and Openness showed that SETs were positively related to Agreeableness and CSE and negatively related to Neuroticism, supporting the three hypotheses of study. Each of these significant relations was maintained when MPI, LPI or a composite of MPI and LPI served as the SET criterion. For example, the MPI-LPI composite correlated .28 with Agreeableness, .35 with CSE and –.28 with Neuroticism. Similar correlations resulted for MPI and LPI. Hierarchical multiple regression demonstrated that the CSE was an independent predictor of MPI ratings, Agreeableness was an independent predictor of LPI ratings, and both the CSE and Agreeableness were independent predictors of MPI-LPI composite ratings. Neuroticism did not emerge as an independent predictor because of the substantial correlation between CSE and Neuroticism (r = .53) and because CSE had greater predictive capacity. This is the first study to incorporate the CSE construct into the SET literature.     

The three-tiered support system and the special education teachers’ role in Swedish-speaking schools in Finland

ABSTRACT

In 2010, new amendments regarding special education were made to the Finnish Basic Education Act (642/2010), and they were officially adopted in 2011. The three-tiered support system that was introduced can be considered the Finnish approach to moving education toward a more inclusive system since it emphasises all teachers’ responsibility to deliver support within the regular educational setting, representing a new feature in the policy documents. This has brought about new expectations for special education teachers’ (SETs’) roles. Our research aims to contribute to knowledge about the implementation of the three-tiered support system and SETs’ roles in Swedish-speaking schools in Finland. The data were collected using a questionnaire (N = 158). The results indicate that the SETs have an important role in the three-tiered support system, both as those with the knowledge and those who share this knowledge. The SETs’ role is more evident when it comes to pupils receiving support on the second and third tiers. Although inclusive values are emphasised in the policy documents, the SETs still use most of their time teaching pupils in educational settings that are often relatively segregated (individual or small-group teaching), and for example, co-teaching seems to be a less frequent approach to collaboration.     

Improving SET Response Rates: Synchronous Online Administration as a Tool to Improve Evaluation Quality

Institutions of higher education continue to migrate student evaluations of teaching (SET) from traditional, in-class paper forms to online SETs. Online SETs would favorably compare to paper-and-pencil evaluations were it not for widely reported response rate decreases that cause SET validity concerns stemming from possible nonresponse bias. To combat low response rates, one institution introduced a SET application for mobile devices and piloted formal synchronous classroom time for SET completion. This paper uses the Leverage Salience Theory to estimate the impact of these SET process changes on overall response rates, open-ended question response rates, and open end response word counts. Synchronous class time best improves SET responses when faculty encourage completion on keyboarded devices and provide students SET completion time in the first 15 min of a class meeting. Full support from administrators requires sufficient wireless signal strength, IT infrastructure, and assuring student access to devices for responses clustering around meeting times.     

The Relative Merits of Different Types of Overall Evaluations of Teaching Effectiveness

An unresolved issue in student evaluations of teaching effectiveness (SETE) literature is what type of overall evaluation of teaching effectiveness should be used in personnel decisions. The objective of this study is to compare the merits of: (a) an overall evaluation made by students, (b) a weighted average overall evaluation with the weights determined by students, (c) a weighted average overall evaluation with the weights being determined by the individual instructors teaching their respective classes, (d) an unweighted average overall evaluation, and (e) a second-order factor that proxies for an overall evaluation. Our results indicate that: (a) all of these overall evaluations were very highly intercorrelated, (b) the unweighted and weighted average overall evaluations measured virtually the same thing, and (c) the second-order factor that served as an overall evaluation was most highly correlated with the other overall evaluations and had the advantage of being most understandable to the faculty.     

Are they paying attention? Students’ lack of motivation and attention potentially threaten the utility of course evaluations

Student evaluations are a common source of information used by instructors and administrators, but their utility depends on students’ motivation and attention. This paper presents evidence from two studies indicating that insufficient effort responding and lack of motivation may be problems in course evaluations. In the first study, approximately one in four students responded in an improbable way to ‘catch items’ embedded in actual course evaluation instruments, suggesting a lack of attention. In the second study, students’ responses to an online survey indicated that they doubted that their responses to course evaluations would be used by instructors or administrators. The majority of students admitted to only occasionally putting sufficient effort into their responses. Potential means for identifying insufficient effort responding and mechanisms for increasing student attention and motivation during the course evaluation process are discussed.     

“Top of the Mark”

The present study investigated differences in the effectiveness of instructors from a variety of departments who taught the same course in both intensive and traditional formats within the same year, while controlling for many confounding variables. Results indicated that intensive courses did not significantly differ from traditional courses in overall instructor ratings on student evaluations of teaching effectiveness when confounding variables were taken into account. Conversely, intensive courses received significantly higher overall course ratings on student evaluations than did traditional courses, even after controlling for class size and probable grade in course. These findings provide further evidence that negative beliefs concerning intensive courses may be unjustified, and intensive courses may be as or more effective than those presented in traditional formats.     

Eliciting student feedback for course development: the application of a qualitative course evaluation tool among business research students

Student evaluations of teaching and learning are playing an increasingly important role in the delivery of high-quality, student-centred education. Insights into student perceptions of their learning experience provide important information that can be used to inform course design and development. The majority of course evaluations take the form of quantitative surveys, but research suggests that a reliance on survey data alone can be problematic from a teaching and learning perspective. Qualitative course evaluations have been cited as a viable alternative to quantitative evaluations, but less research has been conducted into their efficacy when compared to quantitative evaluations. The study on which this article reports attempted to contribute to addressing this shortcoming by describing and assessing a novel approach to eliciting qualitative feedback from students in a research methodology course at a higher education institution in South Africa. Conventional content analysis was used to analyse the qualitative feedback received from students. The qualitative course evaluation approach was then appraised in terms of the degree to which it has the potential to overcome the shortcomings associated with quantitative course evaluations and the extent to which the information gathered could be used to improve the design and delivery of the academic programme.     

Psychometric quality of a student evaluation of teaching survey in higher education

Student evaluations of teaching (SET) are used globally by higher education institutions for performance assessment of academic staff and evaluation of course quality. Higher education institutions commonly develop their own SETs to measure variables deemed relevant to them. However, ‘home-grown’ SETs are rarely assessed psychometrically. One potential consequence of this limitation is that an invalidated instrument may not provide accurate information for the intended purposes. Moreover, in the absence of psychometric assessment, the students’ voices collected by the SETs often fail to provide insight relative to their intended purpose. The present study evaluates a ‘home-grown’ SET using a Rasch model and confirmatory factor analysis. Our results identified weaknesses in two areas: the rating categories and the number of items used to measure the intended constructs. Suggestions are provided to address these weaknesses. This work provides an additional tool set for critical analysis of SET that is generally applicable for a variety of institutions, including those in Asia.     

Measuring Teaching Effectiveness: Correspondence Between Students’ Evaluations of Teaching and Different Measures of Student Learning

Relating students?? evaluations of teaching (SETs) to student learning as an approach to validate SETs has produced inconsistent results. The present study tested the hypothesis that the strength of association of SETs and student learning varies with the criteria used to indicate student learning. A multisection validity approach was employed to investigate the association of SETs and two different criteria of student learning, a multiple-choice test and a practical examination. Participants were N?=?883 medical students, enrolled in k?=?32 sections of the same course. As expected, results showed a strong positive association between SETs and the practical examination but no significant correlation between SETs and multiple-choice test scores. Furthermore, students?? subjective perception of learning significantly correlated with the practical examination score whereas no relation was found for subjective learning and the multiple choice test. It is discussed whether these results might be due to different measures of student learning varying in the degree to which they reflect teaching effectiveness.