Acquiescence,instructor’s gender bias and validity of student evaluation of teaching

Abstract

The validity of student evaluation of teaching (SET) scores depends on minimum effect of extraneous response processes or biases. A bias may increase or decrease scores and change the relationship with other variables. In contrast, SET literature defines bias as an irrelevant variable correlated with SET scores, and among many, a relevant biasing factor in literature is the instructor’s gender. The study examines the extent to which acquiescence, the tendency to endorse the highest response option across items and bias in the first sense affects students’ responses to a SET rating scale. The study also explores how acquiescence affects the difference in teaching quality (TQ) by instructor’s gender, a bias in the latter sense. SET data collected at a faculty of education in Ontario, Canada were analysed using the Rasch rating scale model. Findings provide empirical support for acquiescence affecting students’ responses. Latent regression analyses show how acquiescence reduces the difference in TQ by instructor’s gender. Findings encourage greater attention to the response process quality as a way to better defend the utility of SET and prevent potentially misleading conclusions from the analysis of SET data.     

Whose feedback? A multilevel analysis of student completion of end-of-term teaching evaluations

Student evaluation of teaching (SET) is now common practice across higher education, with the results used for both course improvement and quality assurance purposes. While much research has examined the validity of SETs for measuring teaching quality, few studies have investigated the factors that influence student participation in the SET process. This study aimed to address this deficit through the analysis of an SET respondent pool at a large Canadian research-intensive university. The findings were largely consistent with available research (showing influence of student gender, age, specialisation area and final grade on SET completion). However, the study also identified additional influential course-specific factors such as term of study, course year level and course type as statistically significant. Collectively, such findings point to substantively significant patterns of bias in the characteristics of the respondent pool. Further research is needed to specify and quantify the impact (if any) on SET scores. We conclude, however, by recommending that such bias does not invalidate SET implementation, but instead should be embraced and reported within standard institutional practice, allowing better understanding of feedback received, and driving future efforts at recruiting student respondents.     

高校学生评教效果的影响因素研究

学生评教作为评价教师教学的一种重要手段受到了高校的普遍重视,并被广泛运用到高校的各项管理之中.但是学生评教是否真正能从本质上反映教学质量一直存在争议,教师因素、学生因素及课程自身因素都有可能影响到学生评教的效果.对学生评教效果影响因素的分析和探讨,有助于高等院校优化学生评教机制,促进教学质量的提高.     

Assessment & Evaluation in Higher Education Using the margin of error statistic to examine the effects of aggregating student evaluations of teaching

We proposed an extended form of the Govindarajulu and Barnett margin of error (MOE) equation and used it with an analysis of variance experimental design to examine the effects of aggregating student evaluations of teaching (SET) ratings on the MOE statistic. The interpretative validity of SET ratings can be questioned when the number of students enrolled in a course is low or when the response rate is low. A possible method of improving interpretative validity is to aggregate SET ratings data from two or more courses taught by the same instructor. Based on non-parametric comparisons of the generated MOE, we found that aggregating course evaluation data from two courses reduced the MOE in most cases. However, significant improvement was only achieved when combining course evaluation data for the same instructor for the same course. Significance did not hold when combining data from different courses. We discuss the implications of our findings and provide recommendations for practice.     

Making sense of module feedback: accounting for individual behaviours in student evaluations of teaching

Quantitative student evaluations of teaching (SET) and assessments are widely used in higher education as a proxy for teaching quality. However, SET are a function of individual rating behaviours resulting from student background, knowledge and personalities, as well as the learning experience being rated. SET from three years of data from a science department at a Russell Group University in the UK were analysed to highlight issues of sample size in relation to variable perceptions of modules, and develop a statistical model of feedback incorporating individual rating behaviours across modules. Key results are that sample size and individual rating behaviours have the potential to significantly affect summary module ratings, especially for <20 respondents or if individuals have heterogeneous views. A new approach is suggested, to interpret and compare quantitative module ratings, acknowledging uncertainty, variability and individual rating behaviours. This has implications for the interpretation of SET in many aspects of academic life, including university league table positions, the identification of good teaching practice with respect to student satisfaction, and the weight given to SET in individual academics’ promotion applications.     

Stability and correlates of student evaluations of teaching at a Chinese university

This paper examines the stability and validity of a student evaluations of teaching (SET) instrument used by the administration at a university in the PR China. The SET scores for two semesters of courses taught by 435 teachers were collected. Total 388 teachers (170 males and 218 females) were also invited to fill out the 60‐item NEO Five‐Factor Inventory together with a demographic information questionnaire. The SET responses were found to have very high internal consistency and confirmatory factor analysis supported a one‐factor solution. The SET re‐test correlations were .62 for both the teachers who taught the same course (n = 234) and those who taught a different course in the second semester (n = 201). Linguistics teachers received higher SET scores than either social science or humanities or science and technology teachers. Student ratings were significantly related to Neuroticism and Extraversion. Regression results showed that the Big‐Five personality traits as a group explained only 2.6% of the total variance of student ratings and academic discipline explained 12.7% of the total variance of student ratings. Overall the stability and validity of SET was supported and future uses of SET scores in the PR China are discussed.     

Teachers’ perceptions of student evaluations of teaching

Evaluation of teaching in higher education has drawn much attention due to the need for greater accountability and improvement in student learning. Our review of literature on Student Evaluation of Teaching (SET) surveys suggests that considerable controversy and criticism have surrounded its use, fairness, and validity. Yet, many universities in Hong Kong are depending primarily on student feedback in SET surveys to provide evidence for their improvement of teaching and learning, and for their decisions made on teachers' pay-rise and tenure. If universities are to continue to use SET surveys as the main source of mechanism for measuring teaching effectiveness, teachers will have to understand and be aware of its purposes. But how much do we know about teachers' perceptions of teaching and learning and, consequently, their opinions of using SET surveys as the primary standardized evaluation mechanism of teaching for promotion and tenure and for teaching effectiveness?     

Academics’ Perceptions of Good Teaching: Assessing the Degree of Parity with Student Evaluation of Teaching Questionnaires

ABSTRACT

There is a dearth of studies on academics’ perceptions of good teaching in transitional economies such as Uganda and the degree of parity between academics’ conceptions of good teaching and the items in the student evaluation of teaching (SET) questionnaires. Against this backdrop, the article reports on a study that explored how academics at Makerere University, Uganda, perceive good teaching and compared the resultant perceptions with the items in the SET questionnaires. The study employed a qualitative approach and data was collected by using semi-structured interviews and reviewing documents. Thematic analysis was employed to analyse the data from the interviews while the data from the documents was analysed using content analysis. The findings showed that academics perceive good teaching as: being knowledgeable; being student-centred; demonstrating good communication skills; undertaking research-based teaching; demonstrating professionalism; being approachable; and being organised. Finally, the findings demonstrated a convergence between academics’ perceptions of good teaching and most of the items in the SET questionnaires.     

Predicting student evaluations of teaching using decision tree analysis

Abstract

This study uses decision tree analysis to determine the most important variables that predict high overall teaching and course scores on a student evaluation of teaching (SET) instrument at a large public research university in the United States. Decision tree analysis is a more robust and intuitive approach for analysing and interpreting SET scores compared to more common parametric statistical approaches. Variables in this analysis included individual items on the SET instrument, self-reported student characteristics, course characteristics and instructor characteristics. The results show that items on the SET instrument that most directly address fundamental issues of teaching and learning, such as helping the student to better understand the course material, are most predictive of high overall teaching and course scores. SET items less directly related to student learning, such as those related to course grading policies, have little importance in predicting high overall teaching and course scores. Variables irrelevant to the construct, such as an instructor’s gender and race/ethnicity, were not predictive of high overall teaching and course scores. These findings provide evidence of criterion and discriminant validity, and show that high SET scores do not reflect student biases against an instructor’s gender or race/ethnicity.     

高等学校学生评教行为偏差研究——基于南京仙林大学城若干大学的分析

学生评教是高校教学质量保障体系的重要组成部分.目前,学生在评教过程中存在评教行为偏差,主要表现为引致评教、恶意评教、随意评教和放弃评教.其直接原因在于高校在学生评教管理中的若干强制规定和院系对学生评教的不当引导;根本原因则是由学生评教定位偏差以及学生评教运作无关学生利益造成的.因此,高校需要科学定位学生评教,极力完善学生评教的制度设计,构建学生评教和学生利益紧密关联的基点,以便促成教师根据学生评教改进教学和服务学生、满足学生学习需要这一根本目的的实现.     

Multiple dimensions of university teacher self-concept

Despite the recognised importance of positive self-concepts inmany settings, surprisingly little attention has been paid to teacherself-concepts – teachers' self-perceptions of their own teachingeffectiveness. We integrate research literatures on self-concept and onstudents' evaluations of teaching effectiveness (SETs), develop amultidimensional university teacher self-concept instrument, andevaluate its psychometric properties (factor structure, reliability,validity). A multitrait-multimethod analysis of relations betweenmultiple dimensions of teacher self-concept and corresponding SET ratingdimensions provides good support for the construct validity of teacherself-concept responses. In support of a priori hypotheses based onself-concept theory, agreement between teacher self-concepts and SETswas moderate (median r = 0.20) for teachers who had notpreviously received SET feedback, but substantially higher (median r= 0.40) for teachers who had previously received SET feedback.Implications for further research on teacher self-reflection and forimproving teaching effectiveness in higher education arediscussed.     

Rate my attitude: research agendas and RateMyProfessor scores

The literature on student evaluations of teaching (SETs) generally presents two opposing camps: those who believe in the validity and usefulness of SETs, and those who do not. Some researchers have suggested that ‘SET deniers’ resist SETs because of their own poor SET results. To test this hypothesis, I analysed essays by 230 SET researchers (170 lead authors) and classified the researchers as having negative, neutral or positive attitudes towards SETs. I retrieved their RateMyProfessors.com (RMP) scores and, using logistic regression, found that lead authors with negative attitudes towards SETs were 14 times more likely to score below an estimated RMP average than lead authors with positive attitudes towards SETs. Co-authors and researchers with neutral attitudes, on the other hand, did not significantly differ from the RMP average. These results suggest that personal attitudes towards SETs may drive research findings.     

A Bootstrap Experiment on the Statistical Properties of Students' Ratings of Teaching Effectiveness

This paper examines the effects of two background variables in students' ratings of teaching effectiveness (SETs): class size and students' motivation (as surrogated by students' likelihood to respond randomly). Resampling simulation methodology has been employed to test the sensitivity of the SET scale for three hypothetical instructors (excellent, average, and poor). In an ideal scenario without confounding factors, SET statistics unmistakably distinguish the instructors. However, at different class sizes and levels of random responses, SET class averages are significantly biased. Results suggest that evaluations based on SET statistics should look at more than class averages. Resampling methodology (bootstrap simulation) is useful for SET research for scale sensitivity study, research results validation, and actual SET score analyses. Examples will be given on how bootstrap simulation can be applied to real-life SET data comparison.     

基于学生视角的中小学评教工具系统构建

教师专业发展需要倾听学生的声音,但是当前国内中小学学生评教系统主要是满意度测评或基于专家评教改编而来的学生评教问卷。由于其设计理念是基于经验,编制视角是成人中心,使用目的是绩效评估,导致学生评教结果不能指向教师专业发展、教师不愿被评教、师生关系异化等后果。因此,良好的学生评教工具系统的建构原则应该是基于实证的理论建构、学生中心的编制视角、实用主义的使用目的。以此为指导尝试建构了三个模块共九个维度的学生评教工具系统。     

Student evaluations of teaching: perceptions of faculty based on gender,position, and rank

The current study explores the feelings and thoughts that faculty have about their student evaluations of teaching (SET). To assess the perceptions of SETs, all teaching faculty in one college at a western Land Grant University were asked to complete an anonymous online survey. The survey included demographic questions (i.e. gender; rank such as assistant, associate, and full professor; and positions like non-tenure track, tenure track, and tenured) as well as questions related to faculty's feelings while reading their SETs. While minimal differences were found in responses based on rank or position, several differences were found based on faculty gender. Overall, female faculty appear to be more negatively impacted by student evaluations than male faculty. These gender differences support previous research that suggests males and females receive and react differently to personal evaluation. Resultant suggestions include modifying surveys from anonymous to confidential and offering professional development training for faculty.     

Analysis of Differences in Nonteaching Factors Influencing Student Evaluation of Teaching between Engineering and Business Classrooms

In recent years, there have been increasing calls from the government and other organizations to provide easy public access to student evaluations of teaching. Indeed, the increasing ease of displaying and viewing large quantities of information, and competition among universities and majors for students, makes it likely that an era of greater transparency of this type of information is at hand. While students’ evaluation of teaching (SET) is one quantitative metric that rates the instructor, it may be influenced by factors that are often beyond the instructor's control. In this study, we analyze a longitudinal data set from both engineering and business schools of a large public university, and identify factors that influence SET. We show which factors have the highest influence on overall SET scores, and contrast these between engineering and business colleges. Colleges within the same university may have differences in the factors affecting SET, and recognition of this is important in effectively and fairly evaluating SET scores. We also provide recommendations regarding information that should be displayed along with the SET, particularly when SET scores are made public, so that instructors are not unduly penalized when their evaluations can be influenced by factors over which they have no control.     

Achievement targets and funding equity in the zimbabwe secondary education system: The case of harare

Currently a lecturer in economics in the School of Economics and Finance at the Queensland University of Technology. Previously at the University of New England, Macquarie University and Deakin University in Australia. Taught in Zimbabwe after its independence in 1980. Since returning to Australia, his research into the economics of education in Zimbabwe has been published in a number of international journals, the most recent article, ‘Effective and ineffective secondary schools in Zimbabwe: a preliminary study’ (with John Fisher) will appear inInternational studies in educational administration. His other research interests include the economics of crime and corrections and correctional education.     

试论信息不对称与高等学校学生评教

学生评教是高等学校教学质量管理的关键举措.在其运作过程中存在诸多信息不时称现象,它会造成学生和教师的逆选择及败德行为.根据信息不对称理论对学生评教的分析,学生、教师的逆选择和败德行为的发生不仅与评教过程中的信息不对称有关,还与目前学生评教制度运作的不科学有关.所以,完善学生评教制度,需要促成教学信息在管理方、教师和学生之间的对称配置,同时应使学生评教制度回归到服务学生和实现学生利益这一基点上.     

Standardised module evaluation surveys in UK higher education: Establishing students’ perspectives

Standardised module evaluation surveys have recently been implemented or extensively redesigned at many different HEIs across the UK in response to an evolving national context, notwithstanding a body of scholarship that has called student evaluation of teaching (SET) into question. Through a focussed single-institution study, this mixed-methods research fills a notable gap in the current literature in establishing students’ perspectives on standardised module evaluation by means of a paper-based questionnaire. Its participants (N=40) recognised some general advantages of a university-wide system, such as facilitating comparison between different modules; but they also acknowledged several shortcomings relating to its lack of sensitivity to individual module contexts and schedules, yielding the overall view that standardised surveys are only partially effective as a means of teaching evaluation. The conclusion considers the wider implications of these distinctive findings, and suggests that the perceived limitations of SET point to the need to triangulate its results with data obtained through alternative evaluation mechanisms.