Student ratings of instruction: Validity and normative interpretations

Using 3,355 class section means, the relationship between six predictor variables and student ratings of instruction (CEQ) was investigated by computing the intercorrelation matrix among all variables and by performing several regression analyses. Results of the study indicated that all linear interactions were negligible and that more than one-fourth of the criterion variance was shared with all the predictor variables. Two predictor variables, expected grade and required-elective, provided extremely large contributions to the prediction of the criterion measure, however. Implications of the validity results with respect to normative data and thus to the administrative use of the ratings were illustrated.     

A path analysis of factors often found to be related to student ratings of teaching effectiveness

The relationships among several variables outside of the instructor's classroom control and student ratings of teaching effectiveness are investigated in a causal network. Student ratings are relatively independent of external variables. Students may be able to take into account more factors than generally assumed when they rate their instructors.     

The factor invariance of student ratings of instruction under three sets of directions

This research addressed itself to the issue of the validity of student ratings of instruction, describing and comparing the factor patterns obtained under three different sets of directions. Graduate education students (n=414) were randomly assigned to one of three conditions: administrator use, instructor use, and student use. Subjects in each condition received a different cover letter which explained the purported use to which their ratings were to be put and which asked them to rate their course and instructor on a 33-item rating scale. Data were factor analyzed using principal-axes factor method followed by an oblique transformation. The factor patterns obtained were then compared using the coefficient of congruence. While two clusters (organization/structure and rapport/interaction) emerged across all three conditions, a third cluster appeared which was unique to each condition. The coefficients of congruence obtained generally indicated that the factors could not be considered invariant across the three conditions.     

A study of the validity of four types of student ratings of college teaching assessed on a criterion of student achievement gains

The purpose of the study was to investigate relationships between student ratings of college teaching using four types of student rating instruments and pre- vs. post-student achievement gains in 36 sections of an undergraduate analytic geometry and calculus course. Student rating instruments used varied according to type of items (high vs. low inference) and focus (students rating their own perceived growth vs. rating the instructor). Data were collected on 799 students (66% freshmen; 16% sophomore; and 15% juniors) at the University of Florida, and relationships were analyzed using the Pearson product-moment correlation technique. Significant relationships were not found between student ratings and student achievement.     

Different administrative directions and student ratings of instruction: Cognitive versus affective effects

A manipulation of the instructions students received prior to completing the 7-item Endeavor Instructional Rating card differentially affected their ratings on two types of items. Specifically, when students were led to believe their ratings would have a strong impact on the instructor's career, they tended to be more lenient on items measuring rapport (i.e., the affective domain); this same effect was not observed for items measuring pedagogical skill (i.e., the cognitive domain). The different items on our instructional rating instrument appear to be measuring different things. One implication of this observation is that the inconsistent findings reported in past research on student ratings of instruction may be due to the differential mix of items from one instrument to another. When instructors are compared on ratings given them by students, unbiased interpretation requires that the multidimensional nature of teaching (and of the rating instrument) be considered.     

The quality of student ratings of teacher behaviour

In a research project into the effectiveness of mathematics teaching in the first year of secondary education, external observers and students rated teachers' behaviour. The reliability and validity of both methods were established. The results show that teacher behaviour is assessed well when student ratings are aggregated at the classroom level. The quality of aggregated student ratings is as good as the quality of data from external observers. The predictive validity of aggregated student ratings is higher than the predictive validity of external observations when subject motivation is taken as a dependent variable.     

Expectancy violation and student rating of instruction

This study examined whether violations that are incongruent with student expectations are significantly different than congruent violations of expectancy in relation to student ratings of instruction. Analysis using the Scheffe' post ANOVA test revealed that college students having high expectations/high experiences evaluated teachers more favorably than students with low expectations/high experiences, low expectations/low experiences and high expectations/low experiences. Reasons why these findings did not coincide with the expectancy violation model are offered.     

Student ratings of college teaching: a comparison of faculty and their students

Based on the review of student ratings myths by Aleamoni (1987, 1999), a survey research design was used to analyse differences between college students' (n = 968) and faculty's (n = 34) perceptions. Generally, students held stronger beliefs in these myths, in that they believed faculty with excellent publication records were better qualified to evaluate teaching and that student ratings on single general items are accurate measures of teaching effectiveness. On the other hand, faculty believed that student ratings were invalid and unreliable. Further examination of student characteristics revealed that male students held stronger beliefs in these myths. Finally, students' beliefs in these myths were correlated with their actual ratings of nine dimensions of the Student Evaluation of Educational Quality. A discussion as well as suggestions for using student ratings is provided.     

Comparison of online and classroom-based student evaluations of instruction

In recent years, colleges have been moving from traditional, classroom‐based student evaluations of instruction to online evaluations. Because of the importance of these evaluations in decisions regarding retention, promotion and tenure, instructors are justifiably concerned about how this trend might affect their ratings. We recruited faculty members who were teaching two or more sections of the same course in a single semester and assigned at least one section to receive online evaluations and the other section(s) to receive classroom evaluations. We hypothesised that the online evaluations would yield a lower response rate than the classroom administration. We also predicted that there would be no significant differences in the overall ratings, the number of written comments, and the valence (positive/neutral/negative) of students’ comments. A total of 32 instructors participated in the study over two semesters, providing evaluation data from 2057 students. As expected, online evaluations had a significantly lower response rate than classroom evaluations. Additionally, there were no differences in the mean ratings, the percentage of students who provided written comments or the proportion of comments in the three valence categories. Thus, even with the lower response rate for online evaluations, the two administration formats seemed to produce comparable data.     

Bias hypotheses under scrutiny: investigating the validity of student assessment of university teaching by means of external observer ratings

To advance the discussion on the validity of student evaluations of university teaching, student ratings of two teaching dimensions – student involvement and rapport – were compared with corresponding observer ratings. Seven potential bias variables were tested with regard to their impact on the students’ teaching assessment: three teacher characteristics (first impression, enthusiasm, humour) and four student characteristics (prior interest, expected grades, study experience, class attendance). Bias was defined as an impediment of the students’ assessment of teaching on course level. By means of bivariate correlations with course averages and two-level latent moderated structural equations, data of 1,716 students in 80 courses were analysed. Results showed that all three teacher characteristics were genuinely connected to rapport, and even explained variance of the student-rated variable when controlling for observer-rated rapport. The assessment of student involvement was not modified by the teacher characteristics except for teacher enthusiasm, which affected the student evaluation when controlling for observed involvement and, moreover, moderated the relation between the observed and the student-rated variable. For the examined student characteristics, no biasing effects were found – neither on rapport nor on student involvement.     

A two-dimensional analysis of student ratings of instruction

Student ratings of instruction were analyzed in terms of two global factors. One factor, which includes items on advanced planning, presentation clarity, and increased student knowledge, was named pedagogical skill. The other factor taps information about class discussion, grading, and the availability of help and was named rapport. Ratings on the skill factor did not covary with class size or the leniency of the instructor's grading but did correlate with a reasonable external criterion of student learning. Ratings of rapport correlated inversely with class size and directly with average class grade and showed only a weak relationship to the external criterion of student learning. The skill factor showed more interclass stability than the rapport factor. Previous research studies which have examined the reliability and validity of instructional ratings and their relationship to student grades and class size have reported inconsistent findings. These inconsistencies appear to result from an inappropriate unidimensional analysis of ratings which should be examined in terms of two of more separate attitude dimensions.     

Validity of student self-reported ratings of learning

This study examined the relationship between individual student self-reported ratings of progress on relevant learning objectives and performance on exams administered during a college course. Across three sections of the same course taught by a single instructor, 188 students rated themselves at the end of the course on two objectives identified by the instructor as either essential or important. They also rated themselves on 10 other objectives the instructor identified as having minor or no importance. Self-ratings on course-relevant objectives correlated significantly and positively with four out of five exams and the course total, whereas ratings on irrelevant objectives did not. Students who rated their progress as either exceptional or substantial generally performed better on course examinations than those who rated their progress as moderate or less. These findings support the validity of student self-reported ratings of learning.     

Student ratings and various instructional variables from a within-instructor perspective

The relationship between class size, instructional method, course level, reason for enrollment, and student ratings of instruction was assessed from a within-instructor perspective. Two hundred fifty-four pairs of courses taught by the same instructor were correspondingly identified and subjected to a stepwise multiple regression procedure. Only class size was found to be a significant predictor of ratings once individual differences between instructors were controlled, hence underlining the importance of (1) taking cognizance of the size of the course when using student ratings of instructors as a measure of teaching effectiveness, and (2) controlling for systematic variation due to instructor idiosyncracies in instructional research.     

Standards-based performance assessment for the evaluation of student teachers: a consequential validity study

The instrument Samples of Teaching Performance (STP) was developed to assess student teachers' capacity to plan, deliver and evaluate a unit of instruction. The current study reports consequential validity data collected from supervisors (n?=?20) and student teachers (n?=?62) from three elementary and five secondary teacher preparation programs in Chile that participated in the field-testing of the STP. Student teachers described how this assessment had honed their sense of professionalism and promoted learning of the skills assessed. Supervisors reported enlarging the topics discussed with student teachers and making some changes to the supervisory process. These findings are complemented by an analysis of the STP scores obtained by 24 student teachers, which showed better development of instructional skills when compared to pedagogical reasoning and reflection. These results raise questions about the structure of student teaching to support the implementation of standards-based assessments that entail tasks at different levels of cognitive complexity.     

Relationships between psychometric dimensions of item quality and student ratings of item relevancy and ambiguity

Students rated the quality of the items on a classroom test that had been taken previously. On the same test, psychometric item indices were calculated. The results showed that the student ratings were related to the item difficulty, but not to the item-test correlation. In addition, the better-achieving students tended to rate the items as less ambiguous. Finally, the ambiguity ratings were more highly related to the item-test correlations for the better achieving students. These findings support opinions held by many instructors of students' judgments of item quality.     

Grading leniency, grade discrepancy, and student ratings of instruction

The purpose of this study was to examine how grading leniency and grade discrepancy (the difference between expected grades and deserved grades) were associated with various dimensions of student ratings of instruction. A sample of 754 undergraduate college students completed a student ratings of instruction instrument and provided responses to a number of other questions on topics such as course difficulty and workload. A series of multilevel regression analyses were conducted and results showed that an instructor's grading leniency, as perceived by students, was positively associated with student ratings on 11 of 12 dimensions of instruction examined. This finding suggests that more lenient instructors tend to receive higher student ratings. The second finding shows that grade discrepancy was negatively associated with most dimensions of instruction. This supports the self-serving bias hypothesis under attribution theory (Gigliotti & Buchtel, 1990) in that students tended to punish instructors with lower ratings when expected grades were lower than students believed they deserved, yet little evidence of a pattern of rewards existed in student ratings when students expected grades higher than they deserved.     

Perceived autonomy support,intrinsic motivation,and student ratings of instruction

Motivation theory suggests that autonomy supportiveness in instruction often leads to many positive outcomes in the classroom, such as higher levels of intrinsic motivation and engagement. The purpose of this study was to determine whether perceived autonomy support and course-related intrinsic motivation in college classrooms positively predict student ratings of instruction. Data were collected from 47 undergraduate education courses and 914 students. Consistent with expectations, the results indicated that both intrinsic motivation and autonomy support were positively associated with multiple dimensions of student ratings of instruction. Results also showed that intrinsic motivation moderated the association between autonomy support and instructional ratings—the higher intrinsic motivation, the less predictive autonomy support, and the lower intrinsic motivation, the more predictive autonomy support. These results suggest that incorporating classroom activities that engender autonomy support may lead to improved student perceptions of classroom instruction and may also enhance both student motivation and learning.     

An empirical test of the validity of student evaluations of teaching made on RateMyProfessors.com

The present article examined the validity of public web‐based teaching evaluations by comparing the ratings on RateMyProfessors.com for 126 professors at Lander University to the institutionally administered student evaluations of teaching and actual average assigned GPAs for these same professors. Easiness website ratings were significantly positively correlated with actual assigned grades. Further, clarity and helpfulness website ratings were significantly positively correlated with student ratings of overall instructor excellence and overall course excellence on the institutionally administered IDEA forms. The results of this study offer preliminary support for the validity of the evaluations on RateMyProfessors.com.     

The Relative Merits of Different Types of Overall Evaluations of Teaching Effectiveness

An unresolved issue in student evaluations of teaching effectiveness (SETE) literature is what type of overall evaluation of teaching effectiveness should be used in personnel decisions. The objective of this study is to compare the merits of: (a) an overall evaluation made by students, (b) a weighted average overall evaluation with the weights determined by students, (c) a weighted average overall evaluation with the weights being determined by the individual instructors teaching their respective classes, (d) an unweighted average overall evaluation, and (e) a second-order factor that proxies for an overall evaluation. Our results indicate that: (a) all of these overall evaluations were very highly intercorrelated, (b) the unweighted and weighted average overall evaluations measured virtually the same thing, and (c) the second-order factor that served as an overall evaluation was most highly correlated with the other overall evaluations and had the advantage of being most understandable to the faculty.