Honesty on student evaluations of teaching: effectiveness,purpose, and timing matter!

Student evaluations of teaching (SETs) are an important point of assessment for faculty in curriculum development, tenure and promotion decisions, and merit raises. Faculty members utilise SETs to gain feedback on their classes and, hopefully, improve them. The question of the validity of student responses on SETs is a continuing debate in higher education. The current study uses data from two universities (n = 596) to determine whether and under what conditions students are honest on in-class and online SETs, while also assessing their knowledge and attitudes about SETs. Findings reveal that, while students report a high level of honesty on SETs, they are more likely to be honest when they believe that evaluations effectively measure the quality of the course, the results improve teaching and benefit students rather than the administration, and when they are given at the end of the term. Honesty on evaluations is not associated with socio-demographic characteristics.     

Framing student evaluations of university learning and teaching: discursive strategies and textual outcomes

Evaluation in higher education is an evolving social practice; that is, it involves what people, institutions and broader systems do and say, how they do and say it, what they value, the effects of these practices and values, and how meanings are ascribed. The textual products (verbal, written, visual and gestural) that inform and are produced by, for and through evaluative practices are important, as they promulgate particular kinds of meanings and values in specific contexts. This paper reports on an exploratory study that sought to investigate, using discourse analysis, the types of evaluative practices that were ascribed value, and the student responses that ensued, in different evaluative instruments. Findings indicate that when a reflective approach is taken to evaluation, students’ responses are more considered, they interrogate their own engagement in the learning context and they are more likely to demonstrate reconstructive thought. These findings have implications for reframing evaluation as reflective learning.     

Stability and correlates of student evaluations of teaching at a Chinese university

This paper examines the stability and validity of a student evaluations of teaching (SET) instrument used by the administration at a university in the PR China. The SET scores for two semesters of courses taught by 435 teachers were collected. Total 388 teachers (170 males and 218 females) were also invited to fill out the 60‐item NEO Five‐Factor Inventory together with a demographic information questionnaire. The SET responses were found to have very high internal consistency and confirmatory factor analysis supported a one‐factor solution. The SET re‐test correlations were .62 for both the teachers who taught the same course (n = 234) and those who taught a different course in the second semester (n = 201). Linguistics teachers received higher SET scores than either social science or humanities or science and technology teachers. Student ratings were significantly related to Neuroticism and Extraversion. Regression results showed that the Big‐Five personality traits as a group explained only 2.6% of the total variance of student ratings and academic discipline explained 12.7% of the total variance of student ratings. Overall the stability and validity of SET was supported and future uses of SET scores in the PR China are discussed.     

Sources of variance in end-of-course student evaluations

Abstract

Evaluation of college instructors often centers on course ratings; however, there is little evidence that these ratings only reflect teaching. The purpose of this study was to assess the relative importance of three facets of course ratings: instructor, course and occasion. We sampled 2,459 fully-crossed dyads from a large university where two instructors taught the same two courses at least twice in a 3-year period. Generalizability theory was used to estimate unconfounded variance components for instructor, course and occasion, as well as their interactions. Meta-analysis was used to summarize those estimates. Results indicated that a three-way interaction between instructor, course and occasion that includes measurement error accounted for the most variance in student ratings (24%), with instructor accounting for the second largest amount (22%). While instructor - and presumably teaching - accounted for substantial variance in student course ratings, factors other than instructor quality had a larger influence on student ratings.     

Assessing the reliability of student evaluations of teaching: choosing the right coefficient

Many of the studies used to support the claim that student evaluations of teaching are reliable measures of teaching effectiveness have frequently calculated inappropriate reliability coefficients. This paper points to three coefficients that would be appropriate depending on if student evaluations were used for formative or summative purposes. Results from the present study indicated that students had very low absolute inter-rater reliability, but somewhat higher consistency inter-rater reliability.     

Informing Institutional Management: institutional strategies and student retention

European universities are facing demands for better student retention, especially in countries where state funding is no longer based on the number of students, but on the number of graduates. An extensive literature on retention focuses on the characteristics of students who leave higher education without a degree. Much less is known about the measures taken by higher education institutions to encourage study progress. This article examines whether retention strategies are incorporated into the institutions’ overall strategies and the policies and activities developed by universities to foster study progress and enhance retention rates. The results suggest that the universities’ overall strategies and retention activities are loosely coupled and that the universities tend to use blanket measures addressing retention, regardless of why students are leaving.     

Using a Capability Maturity Model to build on the generational approach to student engagement practices

The generational approach to conceptualising first-year student learning behaviour has made a useful contribution to understanding student engagement. It has an explicit focus on student behaviour and we suggest that a Capability Maturity Model interpretation may provide a complementary extension of that understanding as it builds on the generational approach by allowing an assessment of institutional capability to initiate, plan, manage, evaluate and review institutional student engagement practices. The development of a Student Engagement, Success and Retention Maturity Model is discussed along with its application in an Australian higher education institution. In this case study, the model identified first-, second- and third-generation approaches and, in addition, achieved a ‘complementary extension’ of the generational approach, building on it by identifying additional practices not normally considered within the generational concept and indicating the capability of the institution to provide and implement the practices.     

Beyond Average: Contemporary statistical techniques for analysing student evaluations of teaching

Student evaluations of teaching (SETs) have been used to evaluate higher education teaching performance for decades. Reporting SET results often involves the extraction of an average for some set of course metrics, which facilitates the comparison of teaching teams across different organisational units. Here, we draw attention to ongoing problems with the naive application of this approach. Firstly, a specific average value may arise from data that demonstrates very different patterns of student satisfaction. Furthermore, the use of distance measures (e.g. an average) for ordinal data can be contested, and finally, issues of multiplicity increasingly plague approaches using hypothesis testing. It is time to advance the methodology of the field. We demonstrate how multinomial distributions and hierarchical Bayesian methods can be used to contextualise the SET scores of a course to different organisational units and student cohorts, and then show how this approach can be used to extract sensible information about how a distribution is changing.     

Enhancing student engagement in one institution

Student engagement is important to further and higher education institutions: it is understood to be a proxy for quality teaching and governments attach a proportion of funding to student retention and completion. Many institutions are taking part in student engagement surveys, using the data generated to initiate changes to policies and practice. This article presents an overview of literature on student engagement and quality teaching. It then briefly describes three projects in one institution in New Zealand that were designed to improve teaching and student engagement, and outlines key findings from them. The projects are then related to four of Chalmers’ five levels of quality teaching and Leach and Zepke’s six perspectives on engagement, enabling some conclusions about what Massey University is doing successfully and what more it could do to enhance student engagement. It is suggested that other institutions could use this different approach to review their current pedagogy and student engagement strategies.     

The uncertainty and fragility of learning to teach: a Britzmanian lens on a student teacher story

This paper turns the kaleidoscopic lens of Professor Deborah Britzman’s writings on a student teacher story of learning to teach. It begins by briefly setting the context for the research. It then explains the study’s ontological and epistemological focus on experience and how student teacher and researchers create expressions of experience through an interpretative process. This leads to the design for interpretation in this study and to an explanation of why the work of Britzman became part of the interpretative layering for understanding student teacher Ciara’s texts. It continues by explicating Britzman’s concept of ‘struggle for voice’ as a metaphor for learning to teach and briefly makes comparison with the work of Bakhtin and his concept of the dialogical imagination. It then presents the layered interpretation in the form of indicative excerpts in a chronological story. Finally it focuses on the salient and pervading theme of affect and offers a brief commentary on its significance in initial teacher education and the need for developing an understanding for the feeling world of learning to teach.     

Physical classroom environment and student satisfaction with courses

The higher education market in South Korea has matured over recent decades. Higher education institutions have invested in many areas to boost the student experience. Student satisfaction has been identified as a key variable in explaining the holistic evaluation of the course. This study aims to fulfil a void in research by developing a study model consisting of cognitive evaluation, affective evaluation and student satisfaction with the course. Furthermore, two constructs of classroom physical environments were added to measure the moderating effect of satisfaction’s antecedents. Results indicated that student satisfaction with a course was significantly explained by cognitive and affective evaluations. Affective evaluation played a mediating role between cognitive evaluation and students’ satisfaction with the course. In addition, the impact of cognitive evaluation was greater than that of affective evaluation. The cognitive and affective evaluation constructs in the proposed theoretical framework accounted for 56% of the variance explained in student satisfaction with the course. Moreover, results of the invariance test revealed that ambient conditions of the classroom and its spatial layout and functionality significantly enhanced the effects of cognitive evaluation and affective evaluation on student satisfaction with the course.     

Relationship of changes in student motivation to student evaluations of instruction

Research has demonstrated that student evaluations of instruction are influenced by variables extraneous to the instructional procedures being evaluated. One of the most important of these is the student's motivation to take the course. The Instructional Development and Effectiveness Assessment (IDEA) system controls this variable by comparing a course evaluation to a norm group of courses having students with similar motivation. The present study examined the possibility that the IDEA procedure of having students rate their precourse motivation at theend of a course might be unacceptable, because the rating would be influenced by experiences within the course itself. The data indicated that postcourse ratings of precourse motivation do deviate somewhat from actual precourse ratings, but the deviation is not of an order of magnitude which would seriously distort the interpretation of the ratings.     

The emergence of contesting motives for student feedback-based evaluation in Australian higher education

Student feedback-based evaluation performs a significant social role in framing perceptions of the quality of teaching in contemporary Australian higher education. Yet its emergence is a relatively recent phenomenon, having only been in widespread application since the mid-1980s. The early manifestations of student feedback-based evaluation came with newly emerging academic development units with a motive to enhance the quality of local teaching and to afford student retention. However, new motives for assailing student feedback evolved with the rapid growth in student numbers, the introduction of student fees and heightened levels of inter-institutional competition for students. As a result, student feedback-based evaluation progressively became also a powerful proxy measure of teaching and curricula quality assurance at an individual, institutional and sectoral level [Blackmore, J. (2009). Academic pedagogies, quality logics and performative universities: Evaluating teaching and what students want. Studies in Higher Education, 34(8), 857–872. doi:10.1080/03075070902898664]. This generated critical tensions between the seminal motive of student feedback around quality improvement, and the rising quality assurances discourses, academic performance management demands and institutional marketing. In this paper, the complex social origins of these competing motives for student feedback-based evaluation in Australian higher education will be explored and analysed. It is argued this provides an important means of understanding the polarising effects of student feedback-based evaluation in Australian universities.     

Understanding student evaluations of teaching in online learning

In this paper, we have developed a classification model for online learning environments that relates the Instructors Overall Performance (IOP) rating (according to students’ perceptions) with the course characteristics, students’ demographics and the effectiveness of the instructor in his/her teaching roles. To that end, a comprehensive Student Evaluation of Teaching (SET) instrument is proposed, which includes not only conventional teaching elements, but also items that encourage twenty-first century skills. The goal of the study is twofold: (i) to quantify the extent to which the selected variables explain the IOP rating, and (ii) determine which teaching and non-teaching variables most affect the IOP rating. The best performing classifier achieved a competitive accuracy, highlighting that the selected variables mainly determine the IOP values. Other important findings include: (i) the IOP value is mainly influenced by the effectiveness of the instructor in his/her teaching roles; (ii) teaching strategies that involve the cooperation between the technical and pedagogical roles should be promoted; (iii) the pedagogical role has the highest impact on the final IOP value; and (iv) the most influential demographic variable is the student’s status (working commitments and family responsibilities).     

Widening participation in higher education with a view to implementing institutional change

In this research informed perspective, I discuss some of the barriers students face during progression to higher education. A crucial role can be played by higher education institutions (HEIs) and other public bodies. I discuss some of the measures taken and critically evaluate these to show how these can be improved. In the absence of a centralised admission system and autonomy exercised by HEIs, it is not clear yet how these targets will be achieved. HEFCE and OFFA play a vital role, but there is further scope towards addressing equality and diversity. This paper discusses some of the ways forward.     

Assessment & Evaluation in Higher Education Using the margin of error statistic to examine the effects of aggregating student evaluations of teaching

We proposed an extended form of the Govindarajulu and Barnett margin of error (MOE) equation and used it with an analysis of variance experimental design to examine the effects of aggregating student evaluations of teaching (SET) ratings on the MOE statistic. The interpretative validity of SET ratings can be questioned when the number of students enrolled in a course is low or when the response rate is low. A possible method of improving interpretative validity is to aggregate SET ratings data from two or more courses taught by the same instructor. Based on non-parametric comparisons of the generated MOE, we found that aggregating course evaluation data from two courses reduced the MOE in most cases. However, significant improvement was only achieved when combining course evaluation data for the same instructor for the same course. Significance did not hold when combining data from different courses. We discuss the implications of our findings and provide recommendations for practice.     

Evaluate to improve: useful approaches to student evaluation

Many teachers in higher education use feedback from students to evaluate their teaching, but only some use these evaluations to improve their teaching. One important factor that makes the difference is the teacher’s approach to their evaluations. In this article, we identify some useful approaches for improving teaching. We conducted focus groups with award-winning university teachers who use student evaluations to improve their teaching, and we identified how they approach their evaluation data. We found that these teachers take a reflective approach, aiming for constant improvement, and see their evaluation data as formative feedback, useful for improving learning outcomes for their students. We summarise this as the improvement approach, and we offer it for other teachers to emulate. We argue that if teachers take this reflective, formative, student-centred approach, they can also use student evaluations to improve their teaching, and this approach should be fostered by institutions to encourage more teachers to use student evaluations to improve their teaching.     

Exploring the impact of faculty reflection on weekly student evaluations of teaching

This exploratory study considered Larrivee’s assessment of teachers’ reflective practice levels by using a formative, weekly, online student evaluation of teaching (SET) tool through a virtual learning environment (VLE) as a means to encourage reflective practice. In‐depth interviews were conducted with six faculty members in three departments at a university college in the UK. The study found that: (a) faculty who experienced surface‐level reflection were more likely to have a reactive reflection style; and (b) faculty who experienced higher levels of reflection were more likely to have a proactive reflection style. Overall, the tool was found to be an efficient means of encouraging reflection by all participants and demonstrated that reflective practice could come about as a result of these weekly formative SETs. The study concludes with suggestions for academic development and future research on reflection that could be conducted using SETs via a VLE.     

Bridging the gap: supporting student transitions into higher education

This research focused on the early experience of students entering an undergraduate course in a post‐1992 university that is committed to widening participation. Using Bourdieu’s concept of cultural capital and habitus as a theoretical framework, data were collected from students using an online questionnaire and small‐group discussions during the critical first days and weeks when they need to fit in to their new environment. The research was designed to consider whether there is a ‘new student’ in higher education (HE) and to consider the possible influence of cultural capital and habitus on a student’s transition. Data were collected using an online questionnaire with a response rate of 52% (n=180), and this was followed up with five small‐group discussions with 25 of the respondents. Participants self‐selected to take part in the small‐group discussions but the sample did reflect the cohort in relation to ethnicity, age and gender. The data collected from the questionnaire provided a snapshot of the students’ early experience within the university, and data from the small‐group discussions were used to triangulate this and allow emerging themes to be explored in greater depth. The results showed that the majority of the students (70%) were combining work with study and most students spent a minimal amount of time on campus, perhaps supporting the concept that there is a ‘new student’ in HE. Perceptions about their transition varied, but most of the students expressed concern about the perceived need to be an independent learner. Students stated that they needed more structured activities on campus to encourage them to fit in, and more support from academic staff, with clear instructions about what was expected.