The performance of a student evaluation of teaching system

Student evaluation of teaching (SET) is now commonplace in many universities internationally. While much effort has been devoted to examining the statistical validity of SET instruments, there has been limited examination of the methodological and consequential validity (together referred to as ‘utility’) of the ways in which SET data are used. This paper examines the SET system at Deakin University from the perspective of utility. It draws on publicly available SET results for an entire annual cycle of unit offerings. Consideration is given to the representativeness of the data produced, and to the utility of the data reported, by the system. While this investigation focuses on the SET system currently employed at Deakin University, it offers both an analysis methodology and conclusions that can be applied more generally.     

大学生参与教师教学评价的调查研究

大学生参与教师教学评价是提升高校教育教学质量的重要手段,也是大学生发挥主体作用的具体体现。从问卷和访谈调查中发现,高校虽然普遍开展了学生评教活动,但在评价的时机、形式、内容、结果利用以及师生的认识和态度等方面,仍然存在着一些不容忽视的问题。高等学校应拓宽教学评价的内容,开辟学生参与评教的多种途径,开发适合各类课程的评价工具,使学生的评教真正有助于教师的教学改革。     

Quality lies in the eyes of the beholder: A mismatch between student evaluation and peer observation of teaching

Abstract

The study described in this article was prompted by the poor performance of students in an ‘at risk subject’ in a science faculty at a university in South Africa. Teacher performance could contribute to poor performance among students, therefore the performance of one of the science teachers whose students were performing poorly was evaluated by his students and through peer observation of his teaching. The article draws on a merger between Bernstein's ideas on framing as well as deep and surface approaches to learning to form a theoretical framework that underpinned the study. Peer observation showed that the teacher employed predominantly teacher-centred, passive approaches to teaching, and the facilitation of active learning was minimal, that is, framing was strong. Students, however, evaluated their teacher positively, indicating that he was an ‘effective’ teacher. Therefore, the perception of what constitutes ‘quality teaching’ is viewed differently by peers and students. This is most likely due to the incompatibility between peers' conception of teaching and students' conception of learning. Therefore, students' feedback on teaching is not necessarily accurate or useful.     

论学生评教及高校教学质量保障体系的改善

学生评教是高校教学质量保障的一项根本制度,它的建立使学生拥有了表达教学意愿的常规渠道,有助于高校树立服务学生的办学宗旨。我国高校开展学生评教工作的时间不长,在实际工作中还存在学生代人评教、形式意义大于实质意义、评教结果被误用、制度设计存在漏洞等问题。为此,应从树立正确的学生评教观、构建科学的学生评教制度、借鉴国外的先进经验等方面,改进我国高校的学生评教制度,完善我国高校教学质量保障体系。     

Updating RIGs: including the systematic influence of online study on student evaluation of teaching

Based on student evaluation of teaching (SET) ratings from 1,432 units of study over a period of a year, representing 74,490 individual sets of ratings, and including a significant number of units offered in wholly online mode, we confirm the significant influence of class size, year level, and discipline area on at least some SET ratings. We also find online mode of offer to significantly influence at least some SET ratings. We reveal both the statistical significance and effect sizes of these influences, and find that the magnitudes of the effect sizes of all factors are small, but potentially cumulative. We also show that the influence of online mode of offer is of the same magnitude as the other 3 factors. These results support and extend the rating interpretation guides (RIGs) model proposed by Neumann and colleagues, and we present a general method for the development of a RIGs system.     

Expectancy theory outcomes and student evaluations of teaching

As student evaluation of teaching (SET) instruments are increasingly administered online, research has found that the response rates have dropped significantly. Validity concerns have necessitated research that explores student motivation for completing SETs. This study uses Vroom's [(1964). Work and motivation (3rd ed.). New York, NY: John Wiley & Sons] expectancy theory to frame student focus group responses regarding their motivations for completing and not completing paper and online SETs. Results show that students consider the following outcomes when deciding whether to complete SETs: (a) course improvement, (b) appropriate instructor tenure and promotion, (c) accurate instructor ratings are available to students, (d) spending reasonable amount of time on SETs, (e) retaining anonymity, (f) avoiding social scrutiny, (g) earning points and releasing grades, and (h) being a good university citizen. Results show that the lower online response rate is largely due to students’ differing feelings of obligation in the 2 formats. Students also noted that in certain situations, students often answer SETs insincerely.     

Student evaluation of teaching and matters of reliability

The student evaluation of teaching process is generally thought to produce reliable results. The consistency is found within class and instructor averages, while a considerable amount of inconsistency exists with individual student responses. This paper reviews these issues along with a detailed examination of common measures of reliability that are utilised with the instruments. While inter-item consistency of the evaluations has been shown to be high, the agreement between students was shown to be no better than what would be expected by chance, indicating that students do not agree on what they are being asked to evaluate. The reliability measures generated by the student evaluations of teaching are an insufficient foundation for establishing validity. Further, the pattern of reliability indicates that the instruments are generally providing information about students, not instructors.     

学生评教对课堂教学质量影响的研究

文章首先分析了我国学生评教产生的宏观背景,然后根据学校学生评教情况,通过数据分析来说明学生评教对课堂教学质量的影响。同时,针对学校学生评教中存在的问题,参考其他高校的成功经验,对如何改进学校学生评教,发挥其在促进教学改革,提高教学质量方面的作用提出建议。     

Strengthening internal quality assurance processes: facilitating student evaluation committees to contribute

Student evaluation committees play a crucial role in internal quality assurance processes as representatives of the student body. However, the students on these committees sometimes experience difficulty in providing constructive and structured feedback to faculty in an environment characterised by a strong power differential between student and staff. This study aimed to evaluate a leadership and quality assurance training implemented for students involved in internal quality assurance. Furthermore, we explored how students give shape to their internal quality assurance role. Students from three health sciences programmes participated in a mixed methods study to evaluate the training and reflect on their internal quality assurance role. Overall, the students were very enthusiastic about the training. Qualitative data analysis indicated that in their internal quality assurance role, students: (1) harnessed the power of the entire student population; (2) tried to create focus and take charge; (3) searched for a common ground with staff, and (4) they explained how they dealt with the power differential. Providing training for students with internal quality assurance roles is a valuable endeavour. Future research needs to investigate further ways to help students to contribute to internal quality assurance processes in higher education.     

Meta-analysis of faculty's teaching effectiveness: Student evaluation of teaching ratings and student learning are not related

Student evaluation of teaching (SET) ratings are used to evaluate faculty's teaching effectiveness based on a widespread belief that students learn more from highly rated professors. The key evidence cited in support of this belief are meta-analyses of multisection studies showing small-to-moderate correlations between SET ratings and student achievement (e.g., Cohen, 1980, Cohen, 1981; Feldman, 1989). We re-analyzed previously published meta-analyses of the multisection studies and found that their findings were an artifact of small sample sized studies and publication bias. Whereas the small sample sized studies showed large and moderate correlation, the large sample sized studies showed no or only minimal correlation between SET ratings and learning. Our up-to-date meta-analysis of all multisection studies revealed no significant correlations between the SET ratings and learning. These findings suggest that institutions focused on student learning and career success may want to abandon SET ratings as a measure of faculty's teaching effectiveness.     

Maximising resource allocation in the teaching laboratory: understanding student evaluations of teaching assistants in a team-based teaching format

Minimal research papers have investigated the use of student evaluations on the laboratory, a learning medium usually run by teaching assistants with little control of the content, delivery and equipment. Finding the right mix of teaching assistants for the laboratory can be an onerous task due to the many skills required including theoretical and practical know-how, troubleshooting, safety and class management. Using larger classes with multiple teaching assistants, a team-based teaching (TBT) format may be advantageous. A rigorous three-year study across twenty-five courses over repetitive laboratory classes is analysed using a multi-level statistical model considering students, laboratory classes and courses. The study is used to investigate the effectiveness of the TBT format, and quantify the influence each demonstrator has on the laboratory experience. The study found that TBT is effective and the lead demonstrator most influential, influencing up to 55% of the laboratory experience evaluation.     

Framing student evaluations of university learning and teaching: discursive strategies and textual outcomes

Evaluation in higher education is an evolving social practice; that is, it involves what people, institutions and broader systems do and say, how they do and say it, what they value, the effects of these practices and values, and how meanings are ascribed. The textual products (verbal, written, visual and gestural) that inform and are produced by, for and through evaluative practices are important, as they promulgate particular kinds of meanings and values in specific contexts. This paper reports on an exploratory study that sought to investigate, using discourse analysis, the types of evaluative practices that were ascribed value, and the student responses that ensued, in different evaluative instruments. Findings indicate that when a reflective approach is taken to evaluation, students’ responses are more considered, they interrogate their own engagement in the learning context and they are more likely to demonstrate reconstructive thought. These findings have implications for reframing evaluation as reflective learning.     

Faculty attitudes toward student evaluation of teaching

This study dealt with university faculty attitudes toward the validity of and varying uses for student evaluations of teaching. It was hypothesized that faculty attitudes toward the potential uses of student evaluations would be systematically related to a professor's a priori frame of reference regarding the validity of these instruments. On the basis of responses to selected questionnaire items, faculty were divided into those whose frame of reference (1) opposed the idea that student evaluations could be made valid, (2) supported the idea that existing evaluation forms were valid, and (3) were uncommitted regarding this validity question. A fourth group, administrators at the university, were also surveyed. Large and consistent differences among these groups were found on policy matters such as the use of student evaluations in tenure and promotion decisions, in providing information to students to help them choose courses, and on a variety of other issues. In each case the results supported the hypothesis.Authors' names are ordered randomly.     

The influence of teacher emotion on grading practices: a preliminary look at the evaluation of student writing

The evaluation of student work is a central aspect of the teaching profession that can affect students in significant ways. Although teachers use multiple criteria for assessing student work, it is not yet known if emotions are a factor in their grading decisions as has been found in other instances of professional evaluations. Reason to believe that this might be the case not only comes from emotion-congruent judgment research, but also from the emerging field of teacher emotion research which is beginning to reveal that emotion may be an important element in various teacher cognitions and practices. The present study provides initial evidence that emotions may bias the grades teachers assign to their students, such that positive and negative emotions influence grade assignment in emotion-congruent ways. This research offers empirical contributions to research on emotion and cognition as well as on teacher emotion.     

Development and validation of a theoretically based,multidimensional questionnaire of student evaluation of university teaching

The authors describe the development and validation of a multidimensional instrument of students’ evaluation of university teaching (the Pedagogical Questionnaire of the University of Porto). The goal was to develop an instrument based on a sound psychometric analysis and simultaneously supported by the learning theory. Based on the data from 4875 questionnaires, the present study examined the fit of the proposed multidimensional model of university teaching to the whole sample and to various specific university programmes of study using confirmatory factor analysis (AMOS 5.0). Results clearly supported the applicability of the proposed model for the whole university and across programmes of study, demonstrating the validity and reliability of the instrument in evaluating several distinct dimensions of the quality of university teaching.     

Academics' resistance to summative peer review of teaching: questionable rewards and the importance of student evaluations

This study draws from 30 semi-structured interviews with tenure-track faculty members in a research-intensive university to examine their lack of engagement in the summative peer review of teaching. Findings indicate that most academics in the study do not think peer review outcomes contribute meaningfully to decisions about career advancement and believe that, in comparison, student evaluation of teaching scores matter more. The findings suggest that faculty member resistance to summative peer reviews will persist unless academics are confident that the results will be seriously considered in decisions about tenure and promotion. This article also contends that senior administrators should provide further clarity about the purpose and use of peer review outcomes in high-stakes career decisions.     

Student evaluation of instruction: comparison between in-class and online methods

This study compares student evaluations of instruction that were collected in-class with those gathered through an online survey. The two modes of administration were compared with respect to response rate, psychometric characteristics and mean ratings through different statistical analyses. Findings indicated that in-class evaluations produced a significantly higher response rate than online evaluation. In addition, Rasch analysis showed that mean ratings obtained in in-class evaluation were significantly higher than those obtained in online evaluation. Finally, the distributions of student attendance and expected grade in both modes were compared via chi-square tests, and were found to differ in the two modes of administration.     

Negotiating accountability during student teaching: the influence of an inquiry-based student teaching seminar

Drawing on the work of Russian literary critic, Mikhail Bakhtin, this article explores how an inquiry-based social studies student teaching seminar helped three preservice teachers negotiate the pressures of standards-based reforms during student teaching. The author first examines how initial perceptions of standardization and high-stakes testing corroded images of powerful teaching and created an ex post facto relationship with teaching social studies. The author then explores how an inquiry-based seminar mitigated these initial impressions by (1) suspending the authority of accountability; (2) creating contact through collaborative inquiry; and (3) refracting practice.     

Student evaluations of teaching effectiveness: a Nigerian investigation

An investigation is reported which tests the applicability of two American instruments designed to assess tertiary students' evaluations of teaching effectiveness with 158 Nigerian undergraduates. The scales were found to have generally high internal consistency reliability coefficients, most of the items were seen to be appropriate, and every item was considered of importance by at least some of the students. In addition, all but the Workload/Difficulty items clearly differentiated between good and poor lecturers. Factor analysis found a strong main factor of teaching effectiveness plus a minor factor referring to course workload and difficulty. Further analysis generally supported the convergent and discriminant validity of those scales hypothesized to measure similar or dissimilar components of effective teaching. However, this analysis supported the factor analytic results as more overlap between aspects of teaching skill and enthusiasm was found than has been evident in Western studies. Thus there must be doubt about the cross-cultural validity of a multidimensional model of teaching effectiveness.