The quality of student ratings of teacher behaviour

In a research project into the effectiveness of mathematics teaching in the first year of secondary education, external observers and students rated teachers' behaviour. The reliability and validity of both methods were established. The results show that teacher behaviour is assessed well when student ratings are aggregated at the classroom level. The quality of aggregated student ratings is as good as the quality of data from external observers. The predictive validity of aggregated student ratings is higher than the predictive validity of external observations when subject motivation is taken as a dependent variable.     

Predicting student ratings: the relationship between actual student ratings and instructors’ predictions

This study examined the relationship between student ratings and instructors’ predictions of these ratings, taking into account other instructor, student, and course characteristics. Participants in the study were 198 instructors in the School of Education at a major teacher training college in Israel. Data corresponding to one randomly selected course per instructor were collected using student and instructor questionnaires and college records. Results indicate a systematic positive relationship between instructors’ predictions and actual student ratings with respect to overall ratings and the ratings of three dimensions of teaching. Results also demonstrate a systematic trend whereby low‐rated instructors tend to overestimate their student ratings, high‐rated instructors underestimated ratings, and moderately rated instructors gave accurate predictions. Results have implications for using predictions to motivate teaching improvement.     

The detection and correction of bias in student ratings of instruction

With the use of surveys of instructional effectiveness that use Likert rating scales, bias is a potential threat to the validity of interpretations. Simple summation of ratings or the use of larger samples are not methods for removing bias. In this study, a new model for scaling ratings is examined. The method both identifies and corrects for bias. Working with a database of student ratings of college instruction, the model was tested in terms of a variety of criteria. Results indicated that bias was detected and that it was large enough to warrant our concern. The statistical corrections were significant both in terms of order and magnitude of class means. Implications for future studies include the specification of more potential sources of bias, the interaction of some of these factors, and the development of more systematic evidence supporting the need to be attentive to bias. The many-faceted Rasch model used in this study needs more evaluation before we are convinced of its utility to study and correct for bias, but preliminary evidence is encouraging. Recommendations were offered for a theoretical rationale for studying bias in student ratings of instructional effectiveness and a program of research leading to the use of this model for reporting results for use in improving instruction and for promotion, tenure, and merit decisions. An earlier version of this paper was presented at the annual meeting of the American Educational Research Association, Atlanta April 1993.     

The impact of individual consultation on student ratings of teaching

Abstract

The present study addressed the impact of individual consultation on teaching improvement as measured by changes in student ratings. Subjects included 91 professors who presented naturally for individual consultation services over a seven‐year period at the teaching centre of a Canadian university. Interventions by the consultant fell into three categories: 1) Feedback‐Consultation, 2) Feedback‐Consultation‐Class Observation, and 3) Feedback‐Consultation‐Class Observation and Student Consultation. End of term student ratings for the course that was the subject of the consultation were compared with student ratings for the same course taught between one and three years prior to the consultation service, and for the same course taught between one to three years following consultation. The results showed that, overall, consultation was effective in improving the quality of the consultees’ teaching, as evidenced by an increase in mean student ratings of instruction. This effect persisted post consultation. Not all intervention groups, however, showed the same pattern of results. Change was evident immediately after the intervention except in the case of brief consultation, although follow‐up data showed improved teaching for the latter group. Control data provided evidence that the change in student ratings post consultation could reasonably be attributed to consultation effects.     

Validity of student self-reported ratings of learning

This study examined the relationship between individual student self-reported ratings of progress on relevant learning objectives and performance on exams administered during a college course. Across three sections of the same course taught by a single instructor, 188 students rated themselves at the end of the course on two objectives identified by the instructor as either essential or important. They also rated themselves on 10 other objectives the instructor identified as having minor or no importance. Self-ratings on course-relevant objectives correlated significantly and positively with four out of five exams and the course total, whereas ratings on irrelevant objectives did not. Students who rated their progress as either exceptional or substantial generally performed better on course examinations than those who rated their progress as moderate or less. These findings support the validity of student self-reported ratings of learning.     

The effects of organizational factors on student ratings and perceived instruction

The objective of this study is to explore a) which dimension of student ratings and which aspects of perceived instruction are affected by the two organizational factors, enrollment size and academic affiliation, and b) the nature of their effect. Two thousand five hundred students participating in 125 courses evaluated their instructors on Q-1 Evaluation of Instruction by Students. The 125 evaluated instructors responded to Q-2 Perception of own Instruction indicating the extent to which they employ various teaching planning and strategy attributes (TPS). Major findings suggest that academic affiliation has no effect on student ratings yet affects abstract aspects of perceived instruction. Instructors of the social sciences, unlike those of the humanities, manifest a vocational outlook in their instruction planning. Data indicate that enrollment size has an effect on the dimensions of student ratings and the perceived instruction, referring to concrete aspects of teaching. Students participating in small classes are more critical of instruction than those in larger classes. The nature of its effect on perceived instruction is mainly in terms of practical solutions of teaching methods and strategies and is hardly manifested in the planning phase of instruction. The relevance of these findings to university administrators is also discussed.     

A two-dimensional analysis of student ratings of instruction

Student ratings of instruction were analyzed in terms of two global factors. One factor, which includes items on advanced planning, presentation clarity, and increased student knowledge, was named pedagogical skill. The other factor taps information about class discussion, grading, and the availability of help and was named rapport. Ratings on the skill factor did not covary with class size or the leniency of the instructor's grading but did correlate with a reasonable external criterion of student learning. Ratings of rapport correlated inversely with class size and directly with average class grade and showed only a weak relationship to the external criterion of student learning. The skill factor showed more interclass stability than the rapport factor. Previous research studies which have examined the reliability and validity of instructional ratings and their relationship to student grades and class size have reported inconsistent findings. These inconsistencies appear to result from an inappropriate unidimensional analysis of ratings which should be examined in terms of two of more separate attitude dimensions.     

Effect of survey size on student ratings of teaching

Questionnaires filled in by students have become increasingly important due to the part they play in decisions about the tenure, promotion and merit pay of faculty. Since it is rare to get a complete return, it is important to determine what reliance to place on various completion rates. Using data from the faculty of engineering in one university the authors conclude that if class size is 30 or more, a 50% response rate gives an acceptable indication of rating. In a small class, less than 30, something like an 80% return is needed if the same degree of confidence is to be achieved.     

The factor invariance of student ratings of instruction under three sets of directions

This research addressed itself to the issue of the validity of student ratings of instruction, describing and comparing the factor patterns obtained under three different sets of directions. Graduate education students (n=414) were randomly assigned to one of three conditions: administrator use, instructor use, and student use. Subjects in each condition received a different cover letter which explained the purported use to which their ratings were to be put and which asked them to rate their course and instructor on a 33-item rating scale. Data were factor analyzed using principal-axes factor method followed by an oblique transformation. The factor patterns obtained were then compared using the coefficient of congruence. While two clusters (organization/structure and rapport/interaction) emerged across all three conditions, a third cluster appeared which was unique to each condition. The coefficients of congruence obtained generally indicated that the factors could not be considered invariant across the three conditions.     

The generalizability of student ratings of instructors: Item specificity and section effects

A number of recent studies have used generalizability theory to examine the dependability of student ratings of instruction. This study extends this line of research by examining the consistency of ratings between different sections of a course taught in a given semester by the same instructor, and by comparing the performance of global- and attribute-type instructor rating items. Five samples of physics instructors, varying in size from 5 to 12 instructors, were rated by their students on a form containing two global and eight attribute items. Each instructor taught two sections of a course. The study found that the section effect was small (ratings of instructors were consistent across different sections of the same course), and that the generalizability of ratings was substantially influenced by item specificity. For summary purposes, one global item seemed sufficient.     

论学生社团在校园文化中的作用

学生社团在管理方式和规模限制上都显得比较宽泛,吸纳的学生人数也比较多;在形式上,学生们根据共同爱好,自发组织、自我教育、自我服务,体现了更多的自主性。因此,学生社团以其影响的广泛性、参与者的自愿性以及活动方式的多样性和活动效果的有效性,在校园文化中的影响力也日益彰显,发挥着独有的作用。     

The relationship between psychological identification with instructors and student ratings of college courses

Drawing upon Freud's seminal discussion of psychological identification vis-a-vis group psychodynamics, this study investigated the relationship between degree of student identification with an instructor and students' ratings of instructional effectiveness. Specifically, it was hypothesized that students who highly identify with their instructor will rate the course more favorably, perceive greater progress on their part toward course related objectives, and report higher motivation to learn than students who do not identify with their instructor. Using a 50 item personality trait Q-sample, 96 undergraduates from eight different courses taught by three different instructors were asked to describe the instructor, their actual self, and their ideal self; Q-sorts for the instructor and ideal self were correlated separately for each student as the measure of identification. Instructional effectiveness was assessed using the Kansas State IDEA form. Correlational and path analyses of the data provided support for the hypothesis. Results are discussed in light of the unique contribution made by the traditional notion of identification to instructional outcomes and evaluation.     

Kinds of Keys of student ratings of faculty teaching effectiveness

Three substudies of effects of different formats on student ratings of faculty teaching effectiveness were conducted. One substudy, Kinds of Keys, investigated three main kinds of keys: Agreement, Evaluation, and Needs Improvement. The second, NO TUP, investigated numbers of positive rating categories. The third, Item Wording Direction, investigated the same items worded positively, negatively, and neutrally, respectively. Practically important differences in level of ratings obtained in Kinds of Keys and practically and statistically significant differences obtained in NO TUP and Item Wording Direction. It was concluded that additional research is necessary to determine if apparent differences in teaching effectiveness are actually differences in teaching effectiveness or differences due to the methods of measurement.     

Following student ratings over time with a catalog-based system

First, the longitudinal nature of student ratings of instructors has not received deserved emphasis from researchers. Second, the use of item banks for designing student rating questionnaires, especially for instructor feedback, has needed attention. These two factors are investigated in this study, which tracks 304 instructors over a four semester period. It was found that the type of questionnaire generated from the item bank led to statistically significant differences among designated groups. The longitudinal analysis, however, indicated only minor improvement over time, regardless of whether or not an instructor chose to use items yielding specific feedback on the instructional components of a course. Additionally, although main effect differences were noted between teaching assistants and regular faculty, other results were very similar.     

Investigating substantive and consequential validity of student ratings of instruction

This study aimed at exploring faculty and student perspectives on student evaluations, as well as identifying their perceptions of the usefulness and appropriateness of the ratings for evaluating teaching effectiveness. More specifically, the study aimed at identifying the consequences, both intended and unintended, of using the evaluations, in addition to better understanding the process students used in responding to evaluations and what use faculty members made of them. Two surveys were developed and placed on the website of the Office of Institutional Research and Assessment. Emails were sent to all students and faculty participating in evaluations soliciting their cooperation and requesting their input. Faculty and student perceptions were compared qualitatively and statistically. Results revealed that students and faculty believe in the effectiveness and usefulness of the system with the need to overcome some negative consequences and biases inherent in its application at the University. Recommendations for system improvement are provided.     

The tutor/facilitator role in student retention

Open and distance learning provide particular (and sometimes overlapping) contexts for discussions of student retention. This paper argues that the environment for learning created by distance learning tutors is a key factor in student retention, whether retention is taken to refer to student completion of a single course or student persistence through a programme of study. Discussion of tutors' roles in retaining students is stimulated by the recent symposium on student retention and draws on a new analysis of three linked empirical inquiries located within the UK Open University. A fruitful integration of questionnaire surveys, project evaluation and practitioner action inquiries suggest that, where questions of retention focus on the student and their learning experience, tutors' facilitative roles are complex and valuable. Institutional policies and course design practices need to support tutors in their retention roles by using and sharing appropriate pedagogies, systems, student information and integrated professional development opportunities.