Teachers’ perceptions of student evaluations of teaching

Evaluation of teaching in higher education has drawn much attention due to the need for greater accountability and improvement in student learning. Our review of literature on Student Evaluation of Teaching (SET) surveys suggests that considerable controversy and criticism have surrounded its use, fairness, and validity. Yet, many universities in Hong Kong are depending primarily on student feedback in SET surveys to provide evidence for their improvement of teaching and learning, and for their decisions made on teachers' pay-rise and tenure. If universities are to continue to use SET surveys as the main source of mechanism for measuring teaching effectiveness, teachers will have to understand and be aware of its purposes. But how much do we know about teachers' perceptions of teaching and learning and, consequently, their opinions of using SET surveys as the primary standardized evaluation mechanism of teaching for promotion and tenure and for teaching effectiveness?     

Why bother? Student motivation to participate in student evaluations of teaching

When response rates on student evaluations of teaching (SETs) are low, inability to properly interpret and use responses from the students who do participate is a big problem. Where does the motivation to participate break down, and where and how does it make sense to invest efforts in rectifying that? In this study, we examined 641 university students’ reported behaviours and motivation related to their SET participation. In terms of behaviour, students who seldom or never participate in online SET tools reported a willingness to invest, at most, five minutes in the process, though the majority never even open the online evaluation links when they receive them. In terms of motivation, they significantly differed from students who always participate with distinctly lower levels of: (1) willingness to participate at all, (2) perception of autonomy and competence, (3) meaningfulness, (4) personal value, (5) engagement in others’ participation, and (6) understanding of the value of their own participation for others’ benefit. Based on these findings, we propose a strategy for increasing future response rates, particularly among recalcitrant students, in order to be able to gather sufficient and reliable results for the work of improving teaching.     

Online student evaluations of teaching: what are we sacrificing for the affordances of technology?

In the context of increased emphasis on quality assurance of teaching, it is crucial that student evaluations of teaching (SET) methods be both reliable and workable in practice. Online SETs particularly tend to raise criticisms with those most reactive to mechanisms of teaching accountability. However, most studies of SET processes have been conducted with convenience, small and cross-sectional samples. Longitudinal studies are rare, as comparison studies on SET methodological approaches are generally pilot studies followed shortly after by implementation. The investigation presented here significantly contributes to the debate by examining the impact of the online administration method of SET on a very large longitudinal sample at the course level rather than attending to the student unit, thus compensating for the inter-dependency of students’ responses according to the instructor variable. It explores the impact of the administration method of SET (paper based in-class vs. out-of-class online collection) on scores, with a longitudinal sample of over 63,000 student responses collected over a total period of 10 years. Having adjusted for the confounding effect of class size, faculty, year of evaluation, years of teaching experience and student performance, it is observed that the actual effect of the administration method exists, but is insignificant.     

What response rates are needed to make reliable inferences from student evaluations of teaching?

This paper addresses the determination of statistically desirable response rates in students’ surveys, with emphasis on assessing the effect of underlying variability in the student evaluation of teaching (SET). We discuss factors affecting the determination of adequate response rates and highlight challenges caused by non-response and lack of randomization. Estimates of underlying variability were obtained for a period of 4?years, from online evaluations at the University of British Columbia (UBC). Simulations were used to examine the effect of underlying variability on desirable response rates. The UBC response rates were compared to those reported in the literature. Results indicate that small differences in underlying variability may not impact desired rates. We present acceptable response rates for a range of variability scenarios, class sizes, confidence level, and margin of error. The stability of estimates observed at UBC, over a 4-year period, indicates that valid model-based inferences of SET could be made.     

How do student and classroom characteristics affect attitude toward mathematics? A multivariate multilevel analysis

This study investigated the effects of student and classroom characteristics on math self-confidence, perceived usefulness, and enjoyment of mathematics as multiple outcomes. A sample of 7th-grade students from 78 classes of 49 schools was studied. The data were collected using, among other instruments, an attitude questionnaire. The results of the multivariate multilevel analysis showed that the variance of the 3 indicators was situated mostly at the student level, and that the indicators correlated strongly at the class level. Higher prior mathematics achievement and positive parental beliefs and attitudes were significant predictors of higher scores across the 3 indicators. Each of the baseline indicators was significantly associated with its corresponding final indicator. At the classroom level, classroom assessment was significantly associated with less endorsement of all 3 indicators, higher levels of classroom modeling with greater endorsement of perceived usefulness of mathematics, and classroom questioning with greater enjoyment of mathematics.     

If you build it will they come?; Exploring the student perspective of weekly student evaluations of teaching

This exploratory study investigated the students’ use of formative, weekly, online evaluations of teaching through a virtual learning environment. Results were based on in‐depth interviews of seven students at a rural university college in the UK. Students from different genders, education levels and backgrounds volunteered for the study. The students thought it was a good tool and useful for providing anonymous feedback. However, their motivation to fill in the evaluations every week varied throughout the period of study, and the weekly feedback soon became routine and too onerous a task, and thus had a tendency towards being superficially conducted. Students were more inclined to comment on negative issues, rather than critically analyse positive ones. They also tended to be more positive towards conducting the evaluation if the lecturer discussed them and/or made changes to their future lectures.     

Is experience the best teacher? A multilevel analysis of teacher characteristics and student achievement in low performing schools

The study investigated several teacher characteristics, with a focus on two measures of teaching experience, and their association with second grade student achievement gains in low performing, high poverty schools in a Mid-Atlantic state. Value-added models using three-level hierarchical linear modeling were used to analyze the data from 1,544 students, 154 teachers, and 53 schools. Results indicated that traditional teacher qualification characteristics such as licensing status and educational attainment were not statistically significant in producing student achievement gains. Total years of teaching experience was also not a significant predictor but a more specific measure, years of teaching experience at a particular grade level, was significantly associated with increased student reading achievement. We caution researchers and policymakers when interpreting results from studies that have used only a general measure of teacher experience as effects are possibly underestimated. Policy implications are discussed.

Once good teaching,always good teaching? The differential stability of student perceptions of teaching quality

In many countries, students are asked about their perceptions of teaching in order to make decisions about the further development of teaching practices on the basis of this feedback. The stability of this measurement of teaching quality is a prerequisite for the ability to generalize the results to other teaching situations. The present study aims to expand the extant empirical body of knowledge on the effects of situational factors on the stability of students’ perceptions of teaching quality. Therefore, we investigate whether the degree of stability is moderated by three situational factors: time between assessments, subjects taught by teachers, and students’ grade levels. To this end, we analyzed data from a web-based student feedback system. The study involved 497 teachers, each of whom conducted two student surveys. We examined the differential stability of student perceptions of 16 teaching constructs that were operationalized as latent correlations between aggregated student perceptions of the same teacher’s teaching. Testing metric invariance indicated that student ratings provided measures of teaching constructs that were invariant across time, subjects, and grade levels. Stability was moderated to some extent by grade level but not by subjects taught nor time spacing between surveys. The results provide evidence of the extent to which situational factors may affect the stability of student perceptions of teaching constructs. The generalizability of the students’ feedback results to other teaching situations is discussed.     

How reliable are students’ evaluations of teaching quality? A variance components approach

The inter-rater reliability of university students’ evaluations of teaching quality was examined with cross-classified multilevel models. Students (N = 480) evaluated lectures and seminars over three years with a standardised evaluation questionnaire, yielding 4224 data points. The total variance of these student evaluations was separated into the variance components of courses, teachers, students and the student/teacher interaction. The substantial variance components of teachers and courses suggest reliability. However, a similar proportion of variance was due to students, and the interaction of students and teachers was the strongest source of variance. Students’ individual perceptions of teaching and the fit of these perceptions with the particular teacher greatly influence their evaluations. This casts some doubt on the validity of student evaluations as indicators of teaching quality and suggests that aggregated evaluation scores should be used with caution.     

More or better student teaching?

Drawing from data on over 1000 prospective teachers in a large urban district including pre and post-student teaching survey data, this study investigates whether lengthening student teaching improves teachers’ perceptions of instructional preparedness, efficacy, and career plans. The findings suggest that the duration of student teaching has little effect on teacher outcomes; however, the quality of student teaching has significant and positive effects. Moreover, the magnitude of the effects of student teaching quality are greater when student teaching is shorter and in schools with more historically underserved racial groups. The authors discuss policy implications and directions for further research.     

Mathematics student teachers’ views on tutor feedback during teaching practice

A group of students studying to become mathematics teachers were asked to comment on the tutor feedback they received during teaching practice (TP) and to offer suggestions aimed at improving this feedback. Analysis of the written data – which was collected through emails – suggests the need for: (i) all TP tutors to provide good quality feedback; (ii) action aimed at deepening students’ understanding of feedback; and (iii) feedback improvement strategies that take into consideration the students’ sense of practicality. The ensuing discussion serves to promote the idea that teachers-to-be can learn a lot about the constructive role of feedback when they observe their tutors managing successfully during TP the inherent tensions between the formative and summative dimensions of assessment. This would require however a shift in pre-service teacher education from simply ‘teaching by telling’ to also ‘teaching by example’.     

How do we rate? An evaluation of online student evaluations

This paper analyses the popular RateMyProfessors (RMP) website where students evaluate instructors in higher education. A study was designed to measure (1) the awareness and utilisation of the RMP website, (2) the internal and external validity of the RMP ratings in measuring teaching effectiveness, and (3) variation in the above across disciplines. It is concluded that the category of ratings, created by the website, establishes an anti‐intellectual tone that manifests itself in comments about instructors’ personality, easiness of workload and entertainment value rather than knowledge attained.     

Understanding and using feedback – A review of empirical studies concerning feedback from external evaluations to teachers

To improve the quality of teaching, educational accountability needs to include periodic external evaluations of students’ performance. This requires evaluation formats which support the development of the educational process and provide information which is understandable for teachers. The aims of this study were to review: (i) how teachers understand the feedback they receive from external evaluations; (ii) how they use the feedback; and (iii) how teachers’ understanding and use of such feedback affects the achievement of their students. None of the papers included contained simultaneous a study with all three of these aspects of external evaluations; the review shows that teachers have many problems understanding feedback and mainly focus their use of it on developing strategic teaching tactics. Research that focuses on teachers’ understanding of external evaluation, their use of it, and how the use of the feedback can foster student achievement is needed.     

Who gets more financial aid in China? A multilevel analysis

The Chinese financial aid system intends to increase the affordability of postsecondary education and provide access to college for disadvantaged students. However, the research base for access to aid in China is extremely thin. Using data from a large cross-sectional survey in Beijing, this study found that attending selective institutions with high-ability peers was positively correlated with the amount of aid awarded and the probability of receiving aid. Female students, students with college-educated fathers, and students from poorer households were expected to receive more aid. Junior and senior students along with more able individuals in science-related majors obtained significantly more financial assistance.     

Development of low-stakes mathematics and literacy test scores during lower secondary school – A multilevel pattern-centered analysis of student and classroom differences

The development of students’ learning and test-taking behavior may derive from the social context and the group of peers they associate with daily for years. Consequently, it is assumed that students’ academic achievements are to some degree affected by their classmates and the composition of the classroom. The present study provides evidence on how Finnish students (N = 5071) from different classrooms (N = 435) develop distinct patterns regarding their mathematics and literacy achievement during lower secondary school. We analysed longitudinal large-scale educational assessment data using a multilevel latent profile analysis (MLPA) to investigate the impact of classroom effect on students’ achievement patterns, that is, on the development of students’ low-stakes mathematics and literacy test scores from 7th to 9th grade. The results demonstrated the added value of modelling the multilevel structure inherent in educational assessment data: we identified four student achievement patterns that displayed different distributions across the school classes. More precisely, besides individual characteristics, the development of students’ low-stakes mathematics and literacy test scores was associated with class-level factors and some of the classrooms seemed to have a stronger effect on students’ test scores. These results suggest that classroom context is associated with students’ achievement patterns, especially regarding the worst achieving students. The findings may reflect a combination of class placement practices as well as classroom and peer effect. Although the differences between Finnish schools have been one of the lowest in the OECD countries, the findings of the present study suggest that the classroom membership may create class level quality differences in both the preconditions and the development of learning.     

Testing a developmental–ecological model of student engagement: a multilevel latent growth curve analysis

US schools fail to engage a significant proportion of adolescent students. Although student engagement is significantly related to academic achievement, there is a dearth of longitudinal research simultaneously examining the impact of personal and contextual factors on student engagement at both individual and school levels. Using a nationally‐representative sample, multilevel growth curve analyses found significant factors related to adolescents’ student engagement both in initial status and rate of growth. Significant factors at the student level were students’ locus of control, self‐esteem, peer academic value, parental expectation and parent–child communication along with the students’ socio‐economic status, previous grades and friend dropout history. Significant factors at the school level were teacher rules on homework, teacher support, and school safety. Implications for research and practice are discussed.     

Anatomy of a teacher–student feedback encounter

This study examines the anatomy of a teacher–student feedback encounter by qualitatively analysing live recordings of feedback between a teacher and her student in upper secondary school in Norway. By conducting a conversation analysis, this research identifies the overall organisation of the encounter. The results reveal that, first, the teacher used questions to establish a basis to promote her own agenda and worked to optimise students' contributions by providing positive feedback and minimising critiques and disagreement and second, the student approached the teacher's feedback with resistance. The study concludes with pedagogical implications for practice.     

Beliefs about teaching: Persistent or malleable? A longitudinal study of prospective student teachers' beliefs

This longitudinal study focused on change in university students' beliefs about the role of teachers. The students (n = 80) were Estonian undergraduates, whose beliefs were investigated in the first and third years of their studies and followed up to the point at which the students either entered teacher education or chose other paths. Beliefs about teaching were neither unambiguously persistent nor malleable. Students who continued in teacher education exhibited stronger beliefs about the teacher as pedagogue and aligned less with the belief that the teacher's role is to be a subject matter expert than peers who did not choose teacher education.