Predicting student ratings: the relationship between actual student ratings and instructors’ predictions

This study examined the relationship between student ratings and instructors’ predictions of these ratings, taking into account other instructor, student, and course characteristics. Participants in the study were 198 instructors in the School of Education at a major teacher training college in Israel. Data corresponding to one randomly selected course per instructor were collected using student and instructor questionnaires and college records. Results indicate a systematic positive relationship between instructors’ predictions and actual student ratings with respect to overall ratings and the ratings of three dimensions of teaching. Results also demonstrate a systematic trend whereby low‐rated instructors tend to overestimate their student ratings, high‐rated instructors underestimated ratings, and moderately rated instructors gave accurate predictions. Results have implications for using predictions to motivate teaching improvement.     

The generalizability of student ratings of instructors: Item specificity and section effects

A number of recent studies have used generalizability theory to examine the dependability of student ratings of instruction. This study extends this line of research by examining the consistency of ratings between different sections of a course taught in a given semester by the same instructor, and by comparing the performance of global- and attribute-type instructor rating items. Five samples of physics instructors, varying in size from 5 to 12 instructors, were rated by their students on a form containing two global and eight attribute items. Each instructor taught two sections of a course. The study found that the section effect was small (ratings of instructors were consistent across different sections of the same course), and that the generalizability of ratings was substantially influenced by item specificity. For summary purposes, one global item seemed sufficient.     

Sources of variance in end-of-course student evaluations

Abstract

Evaluation of college instructors often centers on course ratings; however, there is little evidence that these ratings only reflect teaching. The purpose of this study was to assess the relative importance of three facets of course ratings: instructor, course and occasion. We sampled 2,459 fully-crossed dyads from a large university where two instructors taught the same two courses at least twice in a 3-year period. Generalizability theory was used to estimate unconfounded variance components for instructor, course and occasion, as well as their interactions. Meta-analysis was used to summarize those estimates. Results indicated that a three-way interaction between instructor, course and occasion that includes measurement error accounted for the most variance in student ratings (24%), with instructor accounting for the second largest amount (22%). While instructor - and presumably teaching - accounted for substantial variance in student course ratings, factors other than instructor quality had a larger influence on student ratings.     

Similarity of student ratings across instructors,courses, and time

This study raised three questions about the similarity or generalizability of student ratings of courses and instructors. First, how stable are student ratings of the same instructor giving the same course during twodifferent semesters? Second, how similar are student ratings of the same instructor in twodifferent courses? Third, how similar are student ratings of a given course being taught bydifferent instructors? Instances were identified in which student ratings on seven different factors were available for pairs of courses for each of these questions. For the case of the same instructor — same course—different semesters, student ratings were reasonably similar (median r for seven factors about 0.70). For the case of the same instructor—different courses, the median r was surprisingly low — about 0.40. For the case of the same course—different instructors, substantial correlations were obtained for some factors and insignificant correlations for other factors. Implications of these findings for practical use of student ratings and suggestions for further research in the area are discussed.     

Following student ratings over time with a catalog-based system

First, the longitudinal nature of student ratings of instructors has not received deserved emphasis from researchers. Second, the use of item banks for designing student rating questionnaires, especially for instructor feedback, has needed attention. These two factors are investigated in this study, which tracks 304 instructors over a four semester period. It was found that the type of questionnaire generated from the item bank led to statistically significant differences among designated groups. The longitudinal analysis, however, indicated only minor improvement over time, regardless of whether or not an instructor chose to use items yielding specific feedback on the instructional components of a course. Additionally, although main effect differences were noted between teaching assistants and regular faculty, other results were very similar.     

Course ratings as measures of instructional effectiveness

This study assesses the validity of instructor ratings as measures of instructional effectiveness. Multiple regression analysis was used to assess the relationship between ratings of classroom practices, instructor characteristics and student commitment to the course and an overall rating of the instructor. Sixty-four percent of the variance in instructor ratings could be explained by variance in students' ratings for eleven course-related items. It was concluded that variance in instructor ratings reflected variance in classroom and instructional practices.     

Ratings of university teacher instruction: how much do student and course characteristics really matter?

Several student and course characteristics were examined in relation to student ratings of instruction. Students at a major Canadian university completed the Universal Student Ratings of Instruction instrument at the end of every course over a three‐year period, providing 371,131 student ratings. Analyses of between‐group differences indicate that students who attend class often and expect high grades provide high ratings of their instructors (p < .001). In addition, lab‐type courses receive higher ratings than lectures or tutorials, and courses in the social sciences receive higher ratings than courses in the natural sciences (p < .001). Regression analyses indicated, however, that student and course characteristics explain little variance in student ratings of their instructors (<7%). It is concluded that student ratings are more related to teaching instruction and behavior of the instructor than to these variables.     

The use of evaluative feedback for instructional improvement: A longitudinal perspective

Both the available experimental evidence and recent reviews are unclear regarding the effectiveness of student ratings feedback for instructional improvement. Provision of feedback has not consistently produced improvement. Feedback accompanied by consultation has resulted in more consistent instructor improvement. However, the literature on interventions for instructional improvements has failed to address the longitudinal nature of improvement and development. The present study examined follow-up data over a ten-year period on instructors who had participated in a quasi-experimental study of the effectiveness of consultation and feedback conducted in 1971 and 1972. The results obtained indicate that instructors who had originally received feedback with consultation maintained higher student ratings and used instructional resources more during follow-up than instructors who originally received feedback only. This result must be tempered by several interpretative considerations regarding sample size, power, and experimental control. A number of concerns were also expressed regarding the adequacy of the current literature for describing a developmental event, instructional improvement, through the use of short-term studies. Recommendations are made for increased durations of study and alternative tactics of experimental design and control.     

Two-way and one-way video: when is ‘no difference’ significant in distance education?

This study investigated the effects of instructor monitoring on student attitudes in an interactive television course. A comparison was made between learners located at sites with and without video feedback to the instructors. Data were collected for four variables: level of interaction as perceived by the student, value of the content taught, student assessment of gains in knowledge, and overall satisfaction. Multivariate analysis of variance (MANOVA) results indicated that student perceptions were not affected by the absence of video monitoring. The effect size was estimated to be practically zero and corroborated by the qualitative evidence collected in a subsequent study using interviews in other television (TV) courses. Findings were discussed in relation to instructional characteristics and system improvement priorities. The paper concluded that given appropriate instructional characteristics, partial absence of video feedback to the instructor may be adequately rectified to justify a wider coverage by interactive TV instruction even where real-time video feedback is not feasible.     

Comparing student,instructor, classroom and institutional data to evaluate a seven-year department-wide science education initiative

We compared seven unrelated data-sets to evaluate a major education improvement initiative. Perceptions of students in 54 course sections were surveyed regarding the helpfulness of 39 specific teaching or learning strategies, and relative workloads and enthusiasm were compared to their other courses. Classes were observed using an established protocol, instructors completed a teaching practices inventory, and their experience with evidence-based pedagogies was established. A graduation exit survey provided longitudinal indications of changes prior to the study, and institutional student ratings of instruction were obtained. This study sought to determine whether improvements were consistently revealed by these data, how perceptions depended upon class size, course improvement model and instructor experience, and whether student ratings captured consistent perceptions about effectiveness. Overall, results compared favourably. Student perceptions and observed effectiveness depended mainly upon class size and improvement strategy. Students found experiences more effective in courses taught by experienced instructors and in classes observed to be more active. Relative workloads were not correlated with any measure of effectiveness while relative enthusiasm was higher in courses perceived to be more effective. Student ratings were consistent with other data-sets, although they did not provide information specific enough for informing further improvements.     

From Corrector to Collaborator: The Range of Instructor Roles in Writing‐Based Natural and Applied Science Classes

This study of writing‐intensive (WI) undergraduate natural and applied science courses examined the relationships among instructors' course goals, instructional activities, and students' assessment of their learning of content and writing. Using multiple sources of data, investigators found that instructors held common goals but varied greatly in their instructional activities. Findings suggest that science instructors can be described along a continuum anchored by instructor as corrector on one end and instructor as collaborator on the other. Instructors who were the sole audience for a single writing assignment were correctors. Collaborators varied writing tasks, encouraged collaboration, and emphasized professional contexts for writing; they generally received highest student satisfaction ratings. Peer editing assignments that simulated critical, anonymous journal reviews affected female and male students differently. The findings support the National Academy of Science's teaching standards and assumptions concerning the crucial roles of instructors in socializing students into science communities. We discuss instructional strategies that may be more inclusive to traditionally underrepresented groups such as females and minorities. © 2000 John Wiley & Sons, Inc. J Res Sci Teach 37: 3–25, 2000     

The Impact of Instructional Technology Use,Course Design,and Sex Differences on Students’ Initial Perceptions of Instructor Credibility

This study explores the impact that instructional technology use, course design, and sex differences have on students’ initial perceptions of instructors’ credibility (i.e., competence, trustworthiness, and perceived caring). Participants included 864 students from two Midwestern universities who were randomly assigned to one of 16 experimental conditions (i.e., scenarios) manipulating the use of technology across two types of courses and across two types of instructors (i.e., male and female instructors). Multivariate analyses revealed a two-way interaction effect of technology use by student sex on perceptions of instructor credibility. Follow-up procedures revealed both a curvilinear and a linear effect for technology use on students’ perceptions of instructor competence. In terms of instructor trustworthiness, however, only a curvilinear effect emerged. Finally, the results suggested that technology use has both a curvilinear and a linear effect on perceptions of perceived caring, though the linear effect size was marginal at best. Among the most important implications of this research is the fact that technology use produced the strongest effect size for the perceived caring dimension of instructor credibility.     

Improvement of instruction and teacher effectiveness: are student ratings reliable and valid?

This paper reviews the evidence on whether student ratings are reliable and valid enough to be used for the purpose of the improvement of instruction and teacher effectiveness. A consensus of evidence available in the literature tends to show a measure of consistency, stability and validity of student ratings. Several variables such as sex of rater and ratee, class size, mood of students, rank of instructors, grades students were expecting, time of the day courses are taught, to mention only a few, have been found to have a low to high positive relationship, with student ratings. It also appears that the use of student ratings leads to the improvement of instruction, provided the evaluation data are fed back to the instructor and that an expert or consultant provides assistance to the instructor.     

Effects of class size on alternative educational outcomes across disciplines

This is the first study to use self-reported ratings of student learning, instructor recommendations, and course recommendations as the outcome measure to estimate class size effects, doing so across 24 disciplines. Fixed-effects models controlling for heterogeneous courses and instructors reveal that increasing enrollment has negative and significant effects on student satisfaction in Sociology, Political Science, Computer Science and Engineering, and Mechanical and Aerospace Engineering. Educational outcomes in Linguistics, Psychology, Biological Sciences, Chemistry, Math, Physics, Cognitive Sciences, Visual Arts, History, and Philosophy are unaffected by class size. Other disciplines, including Economics, have inconclusive findings. No discipline benefits from increasing enrollment.     

Measuring the impacts of administrative policies on student performance in higher education

In this study, we investigate the joint impacts of class size, modality of content delivery and instructor workloads on 1078 student performances from 2010 through 2014 in a required College of Business Administration Information Systems (IS) course at a public Midwestern university, controlling for 11 demographic and other factors that might impact student performance. Class size, modality of delivery and instructor assignments are strategic college decisions affected by and subject to administrative policies that can impact the probabilities of student performance. Using the Akaike Information Criteria (AIC) and Chi-square significance tests, we construct a parsimonious binary logit model of individual student performance in the IS course and examine the effects of the administratively controllable factors on student performance. We find that the class size and the assignment of instructors to teach the course significantly impact student performance while modality of delivery does not. We offer and evaluate administrative policy recommendations likely to improve student performances.     

Criticality in international higher education research: a critical discourse analysis of higher education journals

The current study utilizes the theory of planned behavior (Ajzen Organizational Behavior and Human Decision Processes, 50, 179–211 Ajzen 1991) to examine an instructor confirmation-interaction model in the instructional communication context to discover a means by which instructors might cultivate positive student attitudes and increase beliefs that interactions with instructors would be beneficial in the future. Specifically, the model examines how teacher confirmation (Ellis 2000) influences students’ behavioral intention to communicate with instructors. Surveys were distributed to 343 college students (41.7% male and 58.1% female) in a basic communication course. Results were primarily consistent with the proposed model; teacher confirmation was significantly related to attitudes toward communicating with an instructor, subjective norms, and perceived behavioral control. Subjective norms and perceived behavioral control were also positively related to students’ behavioral intention to communicate with the instructor. However, results reveal attitudes toward communicating do not predict students’ behavioral intention to communicate with instructors. It is recommended that future models examine a more contemporary, hi-tech representation of attitude toward student-instructor interactions as it may produce a significant association with students’ behavioral intent to communicate with them. The study concludes with theoretical and practical implications to examine student classroom communication via the confirmation-interaction model and the theory of planned behavior.     

Similarities and Differences in the Play Activities of Children in Two Public Schools with Contrasting Environment

Based on previous work by H. G. Murray, this study examined relationships between overt classroom teaching behaviors of college instructors and student ratings of teaching effectiveness. One hundred seven students who enrolled in six sections of an introductory psychology course rated instructors on 12 measures of teaching effectiveness and reported the frequencies of 20 observable, inclass teaching behaviors. Relationships between teaching behaviors and student ratings were examined by simple, multiple, and canonical correlation procedures. A complex pattern of relationships emerged. The canonical analysis revealed a two-dimensional structure consisting of instructor clarity and student-teacher rapport dimensions. Teaching behaviors loading on these dimensions were examined.     

Students' evaluation and instructors' self-evaluation of university instruction

The purpose of the study was to investigate the relationship between students' evaluations of university instruction and self-ratings of instructors. The sample consisted of 52 instructors, from the School of Education at Tel-Aviv University, who taught 93 classes. The instructors, as well as their students, responded to a 20-item instructional-practices questionnaire. Instructor self-ratings had only a modest relationship with the ratings given by students (a median correlation of .28). Discrepancies between instructor ratings and ratings given by the students were further analyzed for (1) varying training in teaching - no difference was found; and (2) number of years of teaching experience - differences were noted; the self-ratings of less experienced instructors were closer to student ratings.     

The effect of integrated course and faculty development: Experiences of a university chemistry department in the Philippines

It is widely recognized that lectures continue to dominate college chemistry instruction, especially in developing countries, and that lectures limit student intellectual engagement. To address this concern, a General Chemistry course in a Philippine university was reconstructed to implement an instructional cycle consisting of three phases: a plenary or mini‐lecture, seatwork activity, and a summary or closure. An expert instructor coached the instructors to improve their teaching. Two instructors were involved in pilot implementation and 13 instructors in a large‐scale implementation. This article describes the instructors’ adoption of the instructional cycle using qualitative and quantitative methods that involved multiple data sources. The instructional cycle and intensive coaching enabled most instructors to change their practices, shift their focus from teaching to learning, and enhance their knowledge of student learning difficulties. Nine instructors were able to significantly change their teaching and apply meaningful student seatwork in their lessons. These nine instructors used student seatwork and activities 30–70% of the time, whereas previously 90% of the time involved lectures. Videotape records showed that more than 70% of the students were continuously on task. Four instructors had considerable difficulties in applying the new approach but also had difficulties with conventional lectures. The project constituted the start of a departmental reorientation with a focus on effectiveness of teaching and learning. Subsequently the faculty and course development model developed in this study was used to revise other courses. The theory of Rogan and Grayson proved useful in describing the change processes.