Investigating substantive and consequential validity of student ratings of instruction

This study aimed at exploring faculty and student perspectives on student evaluations, as well as identifying their perceptions of the usefulness and appropriateness of the ratings for evaluating teaching effectiveness. More specifically, the study aimed at identifying the consequences, both intended and unintended, of using the evaluations, in addition to better understanding the process students used in responding to evaluations and what use faculty members made of them. Two surveys were developed and placed on the website of the Office of Institutional Research and Assessment. Emails were sent to all students and faculty participating in evaluations soliciting their cooperation and requesting their input. Faculty and student perceptions were compared qualitatively and statistically. Results revealed that students and faculty believe in the effectiveness and usefulness of the system with the need to overcome some negative consequences and biases inherent in its application at the University. Recommendations for system improvement are provided.     

Student Perspectives on Teaching and its Evaluation

The research on student ratings of instruction, while voluminous, has had minimal focus on the perceptions of the students who do the ratings. The current study explored student perspectives on course and teacher ratings as well as some issues related to teaching effectiveness and faculty roles. It was found that students are generally willing to do evaluations and to provide feedback, and have no particular fear of repercussions. However, they have little confidence that faculty or administrators pay attention to the results, and do not even consult the ratings themselves. The students view teaching and advising as the most important roles that should be played by faculty, yet project that faculty, while also viewing teaching as the most important, would rank research above the more student-interactive advising. Canonical correlations among various scales reveal a strong emphasis on such issues of the importance of faculty respect for student views.     

Determinants of teaching quality: What's important to students?

A method for using student evaluations to help faculty improve their teaching performance is presented. A survey of current methods of student evaluations of teaching identified a need to improve the statistical information obtained from these evaluations. An ordinary least squares framework is used to identify the factors that students feel are important in teacher and course ratings. This framework is used to estimate weights that students assign to various teacher and course attributes and to test whether students apply these weights consistently across teachers and courses. About 81 percent of the explained variation in teacher ratings was associated with attributes that contribute to student enjoyment of the learning process. Over 90 percent of the explained variation in course ratings was associated with attributes that measure how much a student learned in the course. Students were found to apply these attributes or weights consistently across teachers and courses. Implications for developing effective teaching strategies, faculty recruitment, and curriculum reform are discussed.     

Bias and the intended use of student evaluations of university faculty

This study examines whether the intended use of data obtained from student evaluations of university faculty biases the ratings. The paper reviews and critiques previous research done in the area. Previous studies produced contradictory results, but the results were confounded by several methodological problems. The research reported here remedied some of the problems in previous studies. The findings indicated that, while students are aware of the intended use of ratings as stipulated in written directions, different uses do not result in statistically significant differences in the ratings. Implications for the use of student ratings in the evaluation of faculty are discussed.     

Assessing academic program and department effectiveness using student evaluation data

The use of aggregated student evaluations of their courses and course elements (e.g., subject functionality, affect, difficulty, graded assignments) is suggested as an efficient and useful means of obtaining program and department assessments. Given that the instruments used to collect student evaluations are valid (if they are not, they should not be used for any purpose), then averaging class data is likely to provide a valid and reliable index of program and department effectiveness as evaluated by students.Program and department assessment data are presented and discussed for a large northeastern professional school. Large and significant differences in the ratings of program elements were found. Although many of the elements designed into the program by the administration and faculty were perceived as operational by the students, some discrepancies between the design and student perceptions existed. Substantial departmental differences were also found which indicated areas of strength and weakness both within and across departments. The potential usefulness of the assessment for internal change and development is discussed.     

Faculty Perspectives on Course and Teacher Evaluations

Student ratings of instruction have been the subject of numerous studies with much of the research focusing on the validity and reliability of the ratings themselves. Comparatively little empirical investigation has been devoted to the perceptions of the individuals who are the subjects of the ratings, that is, the faculty. The current study explored faculty perspectives on the usefulness of student ratings for formative and summative purposes, and the actual use of student ratings for summative purposes. Contrary to what might have been deduced from the anecdotal literature, the results of this study do not portray a great deal of resistance to student ratings in general or to their use for formative and summative evaluation. It was also found that student ratings are actually being used for the latter purpose. The usefulness of the student feedback was viewed differentially by the faculty, with feedback on their interaction with students seen as most useful, followed by feedback on their grading practices, global ratings of instructor and course, and finally structural issues of the course.     

The challenge of Improving Teaching in a Globalizing World

The USA has served as a beacon of hope for thousands of foreign students and academics with its diverse and superior opportunities created by a system of meritocracy unparalleled in the world. In keeping with other industries, academia is increasingly becoming a global village and foreign‐born professors constitute a large proportion of university faculties. As higher education is increasingly accessible to students with varying levels of academic preparedness, faculties have become more aware of the importance of the opinions of students – the system’s consumers – on teaching. Bearing in mind the ‘similarity‐attraction paradigm’, this study sought to examine whether teaching evaluations are affected by cultural similarity or difference between students and instructors. Our data are based on teaching ratings from the largest Israeli public college. The analysis relates to 42,874 teaching ratings of 768 instructors, of whom 602 are Israeli‐born and 166 are immigrants from the Former Soviet Union (FSU). The findings indicate that FSU immigrant students awarded higher evaluations to FSU immigrant faculty members than their native‐born peers. Similarly, Israeli‐born students awarded higher evaluations to Israeli‐born faculty than the FSU immigrant students. We conclude by discussing the educational and managerial implications of these findings for higher education institutions with ethnically diverse faculty and students.     

On the use of student ratings of faculty in determination of pay,promotion and tenure

Student ratings of faculty have traditionally been obtained in a manner designed to be useful to the individual faculty member or to other students. It is now sometimes proposed that the resulting data be used to determine faculty pay, promotion, and tenure. Recent articles and the reviews of past literature on ratings are analyzed to determine whether student ratings are usually associated with teaching effectiveness, whether they are sometimes biased by irrelevant factors, whether faculty can effectively revise their behaviors to improve their ratings, and whether improved ratings result in improved teaching effectiveness. The results tend to indicate that student ratings of faculty as they are currently collected cannot be trusted for considerations of pay, promotion, and tenure.     

Student ratings of college teaching: a comparison of faculty and their students

Based on the review of student ratings myths by Aleamoni (1987, 1999), a survey research design was used to analyse differences between college students' (n = 968) and faculty's (n = 34) perceptions. Generally, students held stronger beliefs in these myths, in that they believed faculty with excellent publication records were better qualified to evaluate teaching and that student ratings on single general items are accurate measures of teaching effectiveness. On the other hand, faculty believed that student ratings were invalid and unreliable. Further examination of student characteristics revealed that male students held stronger beliefs in these myths. Finally, students' beliefs in these myths were correlated with their actual ratings of nine dimensions of the Student Evaluation of Educational Quality. A discussion as well as suggestions for using student ratings is provided.     

A game theoretic analysis of faculty competition and academic standards

Grade inflation, particularly but not exclusively in higher education, is a serious concern of educators, educational policy-makers and researchers. It has been suggested that student evaluations of faculty are among its principal causes because students tend to give favorable evaluations to professors who give high grades, and that these evaluations are used by university administrators as part of the criteria for promotions, salary increases and similar faculty benefits. This explanation suggests that faculty members compete for favorable student evaluations. It can be generalized to state that faculty members cooperate and compete not only for favorable evaluations, but also for the enrollment of students in the courses they teach. The relevance of faculty cooperation and competition suggests that the Theory of Games could be a useful instrument to analyze the interactions among university professors. The object of this paper is to present a model based on these assumptions and to analyze the consequences that can be derived from it that are relevant for university policy decisions.     

STUDENT EVALUATION OF INSTRUCTION: PERCEPTIONS OF COMMUNITY COLLEGE STUDENTS,FACULTY, AND ADMINISTRATORS

The purpose of this study was to compare the opinions of students, teachers, and administrators relative to student evaluation of instruction in selected community colleges. While important educational decisions in community colleges are made on the basis of students’ evaluations (as in retention, promotion, tenure, and pay), little has been accomplished in testing the assumptions behind student evaluation of instruction. The student evaluation process assumes that students are honest, serious, and evaluate instruction, not some incidental activity.

A 25‐item Student Evaluation Process Scale was completed by 607 students, 130 faculty, and 45 administrators in five Illinois community colleges. Findings revealed little significant differences in the opinions of students regarding evaluation of instruction based on variables of sex, age, school location, student type (transfer or occupational), and class standing. There were little significant differences in faculty opinion and within the administrative groups based on selected variables. There were significant differences when the opinions of students, faculty, and administrators were compared. Students and faculty tended to agree with those items that questioned the objectivity of student evaluation of instruction. Administrators and students tended to agree with items reflecting the seriousness with which students evaluate instruction. Faculty and administrators indicated that student evaluation of instruction impacted faculty members’ instructional performances. Neither students, faculty, nor administrators supported the concept of merit pay tied to student evaluation of instruction.

The role of student evaluation of instruction in a faculty evaluation system must be investigated. A variety of groups should participate in this investigation.     

Evaluating the quality of peer and self evaluations as measures of student contributions to group projects

ABSTRACT

Educators in higher education commonly use peer and self evaluations to help assess student performance on group projects. Although these evaluations provide multiple benefits, many educators are wary of using them due to concerns about their quality. This study addresses three questions debated in the literature regarding the quality of these assessments. How much do students differentiate among peer contributions through their ratings? How reliable are peer ratings? How much agreement exists between peer and self ratings? Although these questions have been addressed to varying degrees in past work, their answers have been far from settled. While many studies focus on just one of the questions, this study’s data make it possible to address all three questions for the same group of students as well as examine each question by student performance level. The evaluations assessed in this study were completed by a large number of students under conditions associated with obtaining more valid and reliable ratings. Overall, the results provide support for using peer and self evaluations to help assess student contributions to group projects. Peer ratings were largely reliable as group members generally agreed on the scores given to their peers. In addition, most students differentiated among group member contributions through their ratings. Students also tended to rate themselves higher than their peers rated them. This study has implications for how peer and self evaluations can be most effectively used by educators to measure student performance in group work.     

Student Evaluation of Teaching Effectiveness: An assessment of student perception and motivation

Over the past century, student ratings have steadily continued to take precedence in faculty evaluation systems in North America and Australia, are increasingly reported in Asia and Europe and are attracting considerable attention in the Far East. Since student ratings are the most, if not the only, influential measure of teaching effectiveness, active participation by and meaningful input from students can be critical in the success of such teaching evaluation systems. Nevertheless, very few studies have looked into students' perception of the teaching evaluation system and their motivation to participate. This study employs expectancy theory to evaluate some key factors that motivate students to participate in the teaching evaluation process. The results show that students generally consider an improvement in teaching to be the most attractive outcome of a teaching evaluation system. The second most attractive outcome was using teaching evaluations to improve course content and format. Using teaching evaluations for a professor's tenure, promotion and salary rise decisions and making the results of evaluations available for students' decisions on course and instructor selection were less important from the students' standpoint. Students' motivation to participate in teaching evaluations is also impacted significantly by their expectation that they will be able to provide meaningful feedback. Since quality student input is an essential antecedent of meaningful student evaluations of teaching effectiveness, the results of this study should be considered thoughtfully as the evaluation system is designed, implemented and operated.     

Exploring graduate and undergraduate course evaluations administered on paper and online: a case study

The purpose of this study was to analyse the students’ evaluations of the course and instructor for all statistics courses offered during fall semester 2009 at a large university in the southern United States. Data were collected and analysed for course evaluations administered both online and on paper to students in both undergraduate and graduate courses. Unlike most previous studies on this subject, class section rather than student was treated as the unit of analysis. It was of specific interest to verify prior research findings that evaluation surveys administered online would not result in lower course and instructor ratings and lower response rates. The results showed that there is not sufficient evidence within the collected data to conclude that either course and instructor ratings or response rates are lower for evaluations administered online (online evaluations) than they are for evaluations administered on paper (paper evaluations). Of secondary interest was whether class ratings would be associated with students’ attendance and a comparison of variability among answers for undergraduate vs. graduate students. It was observed that class and teacher ratings were not related to students’ attendance and individual students did not tend to give the same answer for every question on their survey.     

SELF-RATINGS OF COLLEGE TEACHERS: A COMPARISON WITH STUDENT RATINGS

College teachers' self-ratings were investigated in this study by comparing them to ratings given by students. The sample consisted of 343 teaching faculty from five colleges; these teachers, as well as the students in one of their classes, responded to a 21-item instructional report questionnaire. Teacher self-ratings had only a modest relationship with the ratings given by students (a median correlation of .21 for the items). In addition to the general lack of agreement between self and student evaluations, there was also a tendency for teachers as a group to give themselves better ratings than their students did.
Discrepancies between individual teacher ratings and ratings given by the class were further analyzed for: (a) sex of the teacher (no difference found); (b) number of years of teaching experience (no difference); and (c) subject area of the course (differences noted for natural science courses vs. those in education and applied areas).     

A review of the research on student evaluation and a report on the effect of different sets of instructions on student course and instructor evaluation

An extensive review of the research concerning the effect of different variables on student ratings is presented. A study is then reported comparing the effects of different sets of instructions on student evaluations of the course and instructor. The results indicated that the students who were informed that the results of their ratings would be used for administrative decisions rated the course and instructor more favorably on all aspects than students who were informed that the results of their ratings would only be used by the instructor.The authors are indebted to Professor Robert A. Waller for cooperating in obtaining the data on his two history courses.     

The association between student ratings of specific instructional dimensions and student achievement: Refining and extending the synthesis of data from multisection validity studies

This synthesis extends part of Peter Cohen's well-known examination of the association between student ratings of instruction and student achievement. The results of increasing the number of specific instructional dimensions considered and of preserving more of the information in the evaluations by multiple coding into these dimensions are compared with Cohen's findings. Extending the analysis further by changing the unit of analysis and the method of averaging correlations generally decreases the size of the associations. The instructor's preparation and organization, clarity and understandableness, stimulation of interest, motivation of students to reach high standards, encouragement of discussion and openness to others' opinions, and elocutionary skills consistently explain 10% or more of the variance in student achievement regardless of the method of averaging or the unit of analysis used. The size of the association between an instructional characteristic and student achievement, as one indicator of the characteristic's importance, is compared with three other indicators (extent to which students say the characteristic is important, extent to which faculty say it is important, and size of its correlation with the overall evaluation of the instructor).     

Assessing faculty performance using student evaluations of teaching in an uncontrolled setting

This paper provides new evidence on the disparity between student evaluation of teaching (SET) ratings when evaluations are conducted online versus in‐class. Using a multiple regression analysis, we show that after controlling for many of the class and student characteristics not under the direct control of the instructor, average SET ratings from evaluations conducted online are significantly lower than average SET ratings conducted in‐class. Further, we demonstrate the importance of controlling for the factors not under the instructor’s control when using SET ratings to evaluate faculty performance in the classroom. We do not suggest that moving to online evaluation is overly problematic, only that it is difficult to compare evaluations done online with evaluations done in‐class. While we do not suppose that one method is ‘more accurate’ than another, we do believe that institutions would benefit from either moving all evaluations online or by continuing to do all evaluations in‐class.     

The Relative Merits of Different Types of Overall Evaluations of Teaching Effectiveness

An unresolved issue in student evaluations of teaching effectiveness (SETE) literature is what type of overall evaluation of teaching effectiveness should be used in personnel decisions. The objective of this study is to compare the merits of: (a) an overall evaluation made by students, (b) a weighted average overall evaluation with the weights determined by students, (c) a weighted average overall evaluation with the weights being determined by the individual instructors teaching their respective classes, (d) an unweighted average overall evaluation, and (e) a second-order factor that proxies for an overall evaluation. Our results indicate that: (a) all of these overall evaluations were very highly intercorrelated, (b) the unweighted and weighted average overall evaluations measured virtually the same thing, and (c) the second-order factor that served as an overall evaluation was most highly correlated with the other overall evaluations and had the advantage of being most understandable to the faculty.