Prediction-Based Student Evaluations of Teaching as an Alternative to Traditional Opinion-Based Evaluations

Abstract

The validity of traditional opinion-based student evaluations of teaching (SETs) may be compromised by inattentive responses and low response rates due to evaluation fatigue, and/or by personal response bias. To reduce the impact of evaluation fatigue and personal response bias on SETs, this study explores peer prediction-based SETs as an alternative to opinion-based SETs in a multicultural environment. The results suggest that statistically significant fewer respondents are needed to reach stable average outcomes when peer prediction-based SETs are used than when opinion-based SETs are used. This implies that peer prediction-based SETs could reduce evaluation fatigue, as not all students would need to do each evaluation. The results also report that the peer prediction-based method significantly reduces the bias evident in the opinion-based method, in respect of gender and prior academic performance. However, in respect of the cultural variables, race and home language, bias was identified in the peer prediction-based method, where none was evident in the opinion-based method. These observations, interpreted through the psychology literature on the formulation of perceptions of others, imply that although peer prediction-based SETs may in some instances reduce some personal response bias, it may introduce the perceived bias of others.     

Honesty on student evaluations of teaching: effectiveness,purpose, and timing matter!

Student evaluations of teaching (SETs) are an important point of assessment for faculty in curriculum development, tenure and promotion decisions, and merit raises. Faculty members utilise SETs to gain feedback on their classes and, hopefully, improve them. The question of the validity of student responses on SETs is a continuing debate in higher education. The current study uses data from two universities (n = 596) to determine whether and under what conditions students are honest on in-class and online SETs, while also assessing their knowledge and attitudes about SETs. Findings reveal that, while students report a high level of honesty on SETs, they are more likely to be honest when they believe that evaluations effectively measure the quality of the course, the results improve teaching and benefit students rather than the administration, and when they are given at the end of the term. Honesty on evaluations is not associated with socio-demographic characteristics.     

An empirical study on gender differences in the perception of support during transition to university

Research has highlighted the importance of members of staff knowing their students and one way of achieving that is to ask them to predict students' responses to questions about the teaching process. To this end, student responses to open-ended questionnaires were content analysed to produce a questionnaire about factors that contribute to productive seminars and lectures. A total of 148 psychology students ranked the factors in order of importance and 13 members of academic staff predicted student responses. Results indicated that, despite some differences, members of staff were reasonably accurate in their predictions of student responses. It was also seen that student views were relatively consistent across student levels. The importance of the findings in relation to teaching practice are discussed.     

Expectancy theory outcomes and student evaluations of teaching

As student evaluation of teaching (SET) instruments are increasingly administered online, research has found that the response rates have dropped significantly. Validity concerns have necessitated research that explores student motivation for completing SETs. This study uses Vroom's [(1964). Work and motivation (3rd ed.). New York, NY: John Wiley & Sons] expectancy theory to frame student focus group responses regarding their motivations for completing and not completing paper and online SETs. Results show that students consider the following outcomes when deciding whether to complete SETs: (a) course improvement, (b) appropriate instructor tenure and promotion, (c) accurate instructor ratings are available to students, (d) spending reasonable amount of time on SETs, (e) retaining anonymity, (f) avoiding social scrutiny, (g) earning points and releasing grades, and (h) being a good university citizen. Results show that the lower online response rate is largely due to students’ differing feelings of obligation in the 2 formats. Students also noted that in certain situations, students often answer SETs insincerely.     

Student evaluations of teaching: combining the meta-analyses and demonstrating further evidence for effective use

There is a plethora of research on student evaluations of teaching (SETs) regarding their validity, susceptibility to bias, practical use and effective implementation. Given that there is not one study summarising all these domains of research, a comprehensive overview of SETs was conducted by combining all prior meta-analyses related to SETs. Eleven meta-analyses were identified, and nine meta-analyses covering 193 studies were included in the analysis, which yielded a small-to-medium overall weighted mean effect size (r = .26) between SETs and the variables studied. Findings suggest that SETs appear to be valid, have practical use that is largely free from gender bias and are most effective when implemented with consultation strategies. Research, teaching and policy implications are discussed.     

Student evaluations of teaching: perceptions of faculty based on gender,position, and rank

The current study explores the feelings and thoughts that faculty have about their student evaluations of teaching (SET). To assess the perceptions of SETs, all teaching faculty in one college at a western Land Grant University were asked to complete an anonymous online survey. The survey included demographic questions (i.e. gender; rank such as assistant, associate, and full professor; and positions like non-tenure track, tenure track, and tenured) as well as questions related to faculty's feelings while reading their SETs. While minimal differences were found in responses based on rank or position, several differences were found based on faculty gender. Overall, female faculty appear to be more negatively impacted by student evaluations than male faculty. These gender differences support previous research that suggests males and females receive and react differently to personal evaluation. Resultant suggestions include modifying surveys from anonymous to confidential and offering professional development training for faculty.     

Student satisfaction with short learning programmes at a distance learning institution

Abstract

The relevance of short learning programmes as an alternative to full time tertiary degree courses at open and distance learning institutions forms the primary focus of this article. A new kind of student is on the increase in higher education: the mature student who prefers to study part-time in order to combine his or her studies with the advancement of his or her professional life. This kind of student is often interested in short duration, non-degree courses, such as short learning programmes, which have a high degree of specificity in order to update or expand the knowledge needed to enhance their occupational lives. Course applicability and effectiveness, evaluated as student satisfaction with these programmes, are vital in terms of the strategic planning of institutions. This article reports on an analysis of the dimensions that determine satisfaction with short learning programmes at a distance learning institution. Data was obtained by an electronic survey of students who had completed short learning programmes. Findings indicate that teaching staff, teaching methods and course administration are key elements to achieving satisfaction, and students’ loyalty to the institution is expressed in their intention to continue with their studies and to recommend the programmes to other prospective students.     

What motivates higher education educators to innovate? Exploring competence,autonomy, and relatedness – and connections with wellbeing

ABSTRACT

Background: High quality tertiary teaching is important for maximising the impact of tertiary education for students, employers and society, as well as for institutional reputation and accountability. Varied interpretations exist regarding what counts as ‘innovative’ tertiary teaching.

Purpose: This study sought to explore the deeper drivers of motivations of tertiary educators to initiate and implement teaching innovations.

Method: Drawing from the data from semi-structured interviews from a wider study into innovative teaching in a tertiary education institution in Aotearoa New Zealand, we analysed the rationale of 13 expert tertiary educators for teaching in novel ways. The framework for analysis was based on Self-Determination Theory: data were analysed in relation to three basic psychological needs known to impact on motivation and wellbeing – competence, autonomy and relatedness.

Findings: Our analysis suggests that tertiary educator motivation to innovate in their teaching is related to feelings of all three needs. Findings suggest that innovative pedagogical change was linked to educators’ need for feelings of competence in relation to subject area content, pedagogy and developing student competence. Enhancing autonomy for the educators themselves, or for their students, was also an important factor motivating innovative practice. Developing relatedness between students, between themselves and students, and between students and society were reported as motivations for innovative teaching. Some teaching innovations were motivated by the educator’s desire for greater personal or student wellbeing.

Conclusions Educators’ motivations for being innovative in their tertiary pedagogical practice encompassed content, pedagogical, personal and social dimensions. None reported that their innovation was motivated by institutional policy or expectations. The reported challenges to innovative teaching practice, including institutional structures and processes, lack of access to specific technologies, and the time needed to develop and implement changed practice, indicate that educators must be strongly committed to making change in their practice to sustain the effort required.     

The effect of incentives and other instructor-driven strategies to increase online student evaluation response rates

Course evaluations (often termed student evaluations of teaching or SETs) are pervasive in higher education. As SETs increasingly shift from pencil-and-paper to online, concerns grow over the lower response rates that typically accompany online SETs. This study of online SET response rates examined data from 678 faculty respondents and student response rates from an entire semester. The analysis focused on those tactics that faculty employ to raise response rates for their courses, and explored instructor and course characteristics as contributing factors. A comprehensive regression model was evaluated to determine the most effective tactics and characteristics. Using incentives had the most impact on response rates. Other effective tactics that increase response rates include reminding students to take the evaluation, explaining how the evaluations would be used to improve instruction, sending personal emails and posting reminders on Blackboard^®. Incentives are not widely used; however, findings suggest that non-point incentives work as well as point-based ones, as do simple-to-administer minimum class-wide response rate expectations (compared to individual completion).     

Correlation between grade point averages and student evaluation of teaching scores: taking a closer look

One of the most contentious potential sources of bias is whether instructors who give higher grades receive higher ratings from students. We examined the grade point averages (GPAs) and student ratings across 2073 general education religion courses at a large private university. A moderate correlation was found between GPAs and student evaluations of teaching (SETs); however, this global correlation did not hold true for individual teachers and courses. In fact, there was a large variance in the correlations between GPAs and SETs, including some teachers with a negative correlation and a large variance between courses.     

Professional staff contributions to student retention and success in higher education

ABSTRACT

Student attrition remains a persistent problem within the Australian higher education sector. Contributing factors include financial, reputational and quality issues, which can pose significant risks for a university’s sustainability. Institutional culture is fundamental to decisions student make about withdrawing or remaining in higher education. Therefore, student retention requires a sustained, deeply embedded commitment from all parts of the institution, placing student experience at the forefront of all activities in the student lifecycle. Applying a lifecycle approach to the way in which institutions think about student retention benefits students by providing comprehensive and consistent support. Given that less is known about how professional staff contribute to student academic learning journeys as part of whole of institution responses to student retention, this paper focusses on the contributions that professional staff make within the student lifecycle and how they can most readily map their contributions to student retention and success.     

In search of the feeling of ‘belonging’ in higher education: undergraduate students transition into higher education

ABSTRACT

Previous research that evaluated first year students’ transition into university found that the values of ‘being, belonging and becoming’ were important in particular within the first few months and within the first year of university. From our previous work, we reported that three things matter to students: the academic staff they work with, the nature of their academic study and the feeling of belonging. This paper provides a further illumination to our work by reporting on the qualitative data collected in the same study. The study included 530 students from five cohorts over a five-year period. As part of the Student Experience Evaluation instrument, open-ended questions probed students about their early experiences of belonging and transition into university. This original research uses rich data to illuminate the scales and items from previous quantitative data analysis to explore ‘belonging’, triangulated with research from the field. This paper is timely due to increased emphasis placed on learning and teaching with the introduction of the Teaching Excellence Framework. Student satisfaction is not a simplistic measure and this study articulates the complexity of student belonging in Higher Education.     

Motivations,barriers, & understandings: how students at four universities perceive student–faculty partnership programs

ABSTRACT

Extracurricular student–staff partnership activities are increasingly common in higher education institutions, but concerns have been expressed about their relative inclusivity. In order to contribute to understanding of why students do and do not take part in such initiatives, this study draws on data from a survey of students at four institutions, in three different countries, with established extracurricular partnership programs. Though the findings indicate some preliminary differences between participants at the four institutions, they also demonstrate a number of shared perceptions, including similar understandings of key features of partnership and shared perceptions of barriers to participation. Implications for partnership practice and research are discussed.     

Personal Backgrounds of Effective Teaching

Planning time for giving students effective feedback is an important and challenging aspect of the teaching and learning process. In our article we describe and analyze how we engage students as partners in providing formative feedback in time for students to modify their own thinking or behavior to improve learning. We have found ways to provide formative feedback more frequently and to involve students in providing effective formative feedback to each other. The four techniques we describe are the following: a) three-color group quiz with feedback on product, process, and progress; b) midterm student conferencing; c) shared revision of student generated questions and statements; and d) timely feedback using collaborative assignment blogs. These techniques give feedback in time for revisions to occur, provide scaffolding for learners, inform instruction, and most importantly, involve students as partners in assessment. These pedagogical strategies show that the resulting benefits of improved instruction, enhanced student learning, and better student products are worth the time and effort and contribute to a productive classroom climate where the focus is on learning more than on grading. Formative feedback involving students as partners is a key strategy to enhance the teaching and learning process.     

Part-time faculty status and student evaluation of teaching: An analysis of criminal justice classes

Although part-time (p/t) faculties constitute a growing proportion of college instructors, there is little work on their level of teaching effectiveness relative to full-time (f/t) faculty. Previous work on a key indicator of perceived teaching effectiveness, student evaluation of teaching (SET), and faculty status (p/t/ vs f/t) is marked by a series of shortcomings including lack of a systematic theoretical framework and lack of multivariate statistical analysis techniques to check for possible spuriousness. The present study corrects for these shortcomings. Data consist of SETs from 175 sections of criminal justice classes taught at a Midwestern urban university. Controls are introduced for variables drawn from the literature and include ascribed characteristics of the professor, grade distribution, and structural features of the course (e.g., level, size). The results of a multivariate regression analysis indicate that even after controlling for the other predictors of SETs, p/t faculty receive significantly higher student evaluation scores than f/t faculty. Further, faculty status was the most important predictor of SETs. The results present the first systematic evidence on faculty status and SETs.     

Measuring Teaching Effectiveness: Correspondence Between Students’ Evaluations of Teaching and Different Measures of Student Learning

Relating students?? evaluations of teaching (SETs) to student learning as an approach to validate SETs has produced inconsistent results. The present study tested the hypothesis that the strength of association of SETs and student learning varies with the criteria used to indicate student learning. A multisection validity approach was employed to investigate the association of SETs and two different criteria of student learning, a multiple-choice test and a practical examination. Participants were N?=?883 medical students, enrolled in k?=?32 sections of the same course. As expected, results showed a strong positive association between SETs and the practical examination but no significant correlation between SETs and multiple-choice test scores. Furthermore, students?? subjective perception of learning significantly correlated with the practical examination score whereas no relation was found for subjective learning and the multiple choice test. It is discussed whether these results might be due to different measures of student learning varying in the degree to which they reflect teaching effectiveness.     

Staff student partnership in assessment: enhancing assessment literacy through democratic practices

In recent years, research and practice focused on staff and students working in partnership to co-design learning and teaching in higher education has increased. However, within staff–student partnerships a focus on assessment is relatively uncommon, with fewer examples evident in the literature. In this paper, we take the stance that all assessment can be oriented for learning, and that students’ learning is enhanced by improving their level of assessment literacy. A small study in a Scottish university was undertaken that involved a range of different adaptations to assessment and feedback, in which students were invited to become partners in assessment. We argue that a partnership approach, designed to democratise the assessment process, not only offered students greater agency in their own and their peers’ learning, but also helped students to enhance their assessment literacy. Although staff and students reported experiencing a sense of risk, there was immense compensation through increased motivation, and a sense of being part of an engaged learning community. Implications for partnership in assessment are discussed and explored further. We assert that adopting staff–student partnership in assessment and more democratic classroom practices can have a wide range of positive benefits.     

Laboratory Approach to Secondary Teacher Professional Development: Impacting Teacher Behavior and Student Engagement

Abstract

The Literacy Lab Professional Development provided a laboratory approach to professional development for 42 high school teachers in two schools. Three main activities included: (1) planning and professional development days, (2) lab and professional development days, and (3) individual coaching. The targets of the Literacy Lab Professional Development and its associated guiding questions focused on student engagement and understanding, teaching students how to make meaning from text, teaching students to write, and the structures, systems, routines, and rituals needed to create a classroom culture of thinking and learning. Results from this study point to positive differences between participating and matched-nonparticipating teachers in their instruction, student grouping, nature of tasks, task demands, types of knowledge expectations, personalization, tone, and authority structures.