Using student ratings to measure quality of teaching in six European countries

This paper argues for the value of using student ratings to measure quality of teaching. An international study to test the validity of the dynamic model of educational effectiveness was conducted. At classroom level, the model consists of eight factors relating to teacher behaviour: orientation, structuring, questioning, teaching modelling, application, management of time, teacher role in making classroom a learning environment and assessment. In each participating country (i.e. Belgium/Flanders, Cyprus, Germany, Greece, Ireland and Slovenia), a sample of at least 50 primary schools was used and all grade 4 students (n?=?9967) were asked to complete a questionnaire concerning the eight factors of the dynamic model. Structural equation modelling techniques were used to test the construct validity of the questionnaire. Both across- and within-country analyses revealed that student ratings are reliable and valid for measuring the functioning of the teacher factors of the dynamic model. Implications for teacher education are drawn.     

Meta-analysis of faculty's teaching effectiveness: Student evaluation of teaching ratings and student learning are not related

Student evaluation of teaching (SET) ratings are used to evaluate faculty's teaching effectiveness based on a widespread belief that students learn more from highly rated professors. The key evidence cited in support of this belief are meta-analyses of multisection studies showing small-to-moderate correlations between SET ratings and student achievement (e.g., Cohen, 1980, Cohen, 1981; Feldman, 1989). We re-analyzed previously published meta-analyses of the multisection studies and found that their findings were an artifact of small sample sized studies and publication bias. Whereas the small sample sized studies showed large and moderate correlation, the large sample sized studies showed no or only minimal correlation between SET ratings and learning. Our up-to-date meta-analysis of all multisection studies revealed no significant correlations between the SET ratings and learning. These findings suggest that institutions focused on student learning and career success may want to abandon SET ratings as a measure of faculty's teaching effectiveness.     

College student perceptions of teaching and learning quality

Numerous instructional design models have been proposed over the past several decades. Instead of focusing on the design process (means), this study investigated how learners perceived the quality of instruction they experienced (ends). An electronic survey instrument containing nine a priori scales was developed. Students responded from 89 different undergraduate and graduate courses at multiple institutions (n = 140). Data analysis indicated strong correlations between student self-reports on academic learning time, how much they learned, First Principles of Instruction, their satisfaction with the course, perceptions of their mastery of course objectives, and global course ratings. Most importantly, these scales measure principles with which instructional developers and teachers can evaluate their products and courses, regardless of design processes used: provide authentic tasks for students to do; activate prior learning; demonstrate what is to be learned; provide repeated opportunities for students to successfully complete authentic tasks with coaching and feedback; and help students integrate what they have learned into their personal lives.

Validity of student self-reported ratings of learning

This study examined the relationship between individual student self-reported ratings of progress on relevant learning objectives and performance on exams administered during a college course. Across three sections of the same course taught by a single instructor, 188 students rated themselves at the end of the course on two objectives identified by the instructor as either essential or important. They also rated themselves on 10 other objectives the instructor identified as having minor or no importance. Self-ratings on course-relevant objectives correlated significantly and positively with four out of five exams and the course total, whereas ratings on irrelevant objectives did not. Students who rated their progress as either exceptional or substantial generally performed better on course examinations than those who rated their progress as moderate or less. These findings support the validity of student self-reported ratings of learning.     

Student ratings of teaching quality in primary school: Dimensions and prediction of student outcomes

The contribution examines theoretical foundations, factorial structure, and predictive power of student ratings of teaching quality. Three basic dimensions of teaching quality have previously been described: classroom management, cognitive activation, and supportive climate. However, student ratings, especially those provided by primary school students, have been criticised for being biased by factors such as teacher popularity. The present study examines ratings of teaching quality and science learning among third graders. Results of multilevel confirmatory factor analyses (N = 1556 students, 89 classes) indicate that the three-dimensional model of teaching quality can be replicated in ratings of third graders. In a longitudinal study (N = 1070 students, 54 classes), we found ratings of classroom management to predict student achievement, and ratings of cognitive activation and supportive climate to predict students' development of subject-related interest after teacher popularity is controlled for. The analyses show that student ratings can be useful measures of teaching quality in primary school.     

Early impact and outcomes of an institutionally aligned,student focused learning perspective on teaching quality assurance1

This paper describes an evidence‐based quality assurance system for teaching and learning, which takes as its starting point a well‐researched theoretical perspective on student learning in higher education. We argue this explicit use of a relevant theoretical base promotes coherence between quality assurance and improvement processes. We outline the principal features of our university's quality assurance strategy, systems and processes, and describe how the university's teaching quality assurance policy and systems have been implemented. We then consider the extent to which the project has achieved its goals of fostering an evidence‐based approach to teaching consistent with the student learning perspective on which the policy and systems are based. We also present data on student learning experiences showing reliable changes in the quality of the student learning experience. We discuss the applied significance of these changes, with a particular focus on changes in the experiences of commencing first year university students.     

Effect of survey size on student ratings of teaching

Questionnaires filled in by students have become increasingly important due to the part they play in decisions about the tenure, promotion and merit pay of faculty. Since it is rare to get a complete return, it is important to determine what reliance to place on various completion rates. Using data from the faculty of engineering in one university the authors conclude that if class size is 30 or more, a 50% response rate gives an acceptable indication of rating. In a small class, less than 30, something like an 80% return is needed if the same degree of confidence is to be achieved.     

Reconceptualizing student evaluation of teaching: an ethical framework for changing times

Conversations about student evaluation of teaching are longstanding. Ethical principles in university teaching have been suggested. However, conversations that bring together the topics of ethics, teaching and student evaluation of that teaching are rare. New expectations in relation to the evaluation of teaching, for example, expectations about the role of evaluation of teaching in promotion and probation and about the public availability of student evaluation results on institution web sites, require a reconceptualization of evaluation of teaching policies and practice at many Australian universities. It is timely then, to present the issues raised (privacy, consent, interpretation, authorship and ownership, and accessibility) and the guidelines for practice that emerged from a series of conversations begun with a focus on the ethics of online student evaluation of online teaching. The outcome of conversations, an ethical framework for student evaluation of teaching, suggests a set of values, principles and practices to guide both individuals and institutions in relation to student evaluation of teaching in these changing times.     

Effects of graduate teaching assistant attire on student learning,misbehaviors, and ratings of instruction

This study examined the influence of teaching assistant attire in the university classroom. Significant relationships were found between instructor attire and student cognitive learning, affective learning, and ratings of instruction. Analysis of variance indicated significant differences in these variables by levels of teaching assistant dress. Significant negative relationship was found between instructor attire and student likelihood of misbehavior. Analysis of variance revealed that student misbehaviors were less likely for teaching assistants with high professional dress.     

Improving the quality of student learning: the influence of learning context and student approaches to learning on learning outcomes

Previous studies of the relationship between perceptions and/or evaluations of the learning environment and approaches to study have either not included measures of students' learning outcomes, or have included quantitative differences and not qualitative differences in learning outcomes. The studies reported in this paper focus on the relationship between qualitative differences in learning outcomes, perceptions/evaluations of the learning environment and approaches to study. The results support previous research in identifying relationships between perceptions/evaluations of the learning environment and approach to study and between approach to study and the quality of the learning outcomes. The second of the two studies reported also identifies a relationship between perceptions, approaches and the quality of the outcomes. The results suggest that perceived environments which encourage deep approaches are more likely to facilitate higher quality learning than environments designed to discourage surface approaches.     

How should we use student ratings to evaluate teaching?

The proper use of student ratings to evaluate teaching is the subject matter of this paper. In particular, two divergent views are contrasted: the multidimensional view of evaluation and the unidimensional. Marsh (Int. J. Ed. Res., 1987, v.11, pp. 253–388) is the most outspoken advocate of the multidimensional view of evaluation based on factor scores. Because we disagree over such an important use of ratings, I explain the position I have adopted, summarize Marsh's most recent position, and respond to his concerns.An earlier version of this paper was presented at the annual meeting of the American Educational Research Association, New Orleans, LA, April, 1988.     

The validity of student ratings of teaching quality: Factorial structure,comparability, and the relation to achievement

Student ratings are widely used to assess teaching quality in primary and secondary education. This necessitates a thorough examination their validity and reliability in such contexts. To this end, we present validity evidence for using student ratings of teaching quality based on their factorial structure, comparability across grades and subjects, and relation to achievement. Based on data from the 2019 Trends in Mathematics and Science Study (TIMSS) and its national extensions in Norway (N = 3951 5th graders and N = 4575 9th graders), we could confirm a theoretically assumed three-dimensional factor structure (i.e., classroom management, cognitive activation, and supportive climate). Moreover, classroom management and supportive climate were more comparable across grades and subjects then cognitive activation. Finally, classroom management was associated with achievement across all groups, whereas other dimensions showed mixed results, especially across grades. Overall, the findings indicate that students, especially in Grade 9, can provide a valid assessment of teaching quality.     

The impact of individual consultation on student ratings of teaching

Abstract

The present study addressed the impact of individual consultation on teaching improvement as measured by changes in student ratings. Subjects included 91 professors who presented naturally for individual consultation services over a seven‐year period at the teaching centre of a Canadian university. Interventions by the consultant fell into three categories: 1) Feedback‐Consultation, 2) Feedback‐Consultation‐Class Observation, and 3) Feedback‐Consultation‐Class Observation and Student Consultation. End of term student ratings for the course that was the subject of the consultation were compared with student ratings for the same course taught between one and three years prior to the consultation service, and for the same course taught between one to three years following consultation. The results showed that, overall, consultation was effective in improving the quality of the consultees’ teaching, as evidenced by an increase in mean student ratings of instruction. This effect persisted post consultation. Not all intervention groups, however, showed the same pattern of results. Change was evident immediately after the intervention except in the case of brief consultation, although follow‐up data showed improved teaching for the latter group. Control data provided evidence that the change in student ratings post consultation could reasonably be attributed to consultation effects.     

Classroom discourse: teaching,learning, and learning disabilities

This study examined teacher–student discourse from a sociocultural perspective in order to ascertain the nature of effective instruction for school-age students with learning disabilities. This discussion examines one teacher's resource room instruction that was part of a larger study of teacher–student discourse. The results suggested two considerations: (1) the importance of teacher self-reflection and the implications for effective teaching and learning, and (2) differential discourse patterns across academic domains. Wood's notion of contingent teaching (1991) provided a guideline for coding teacher–student discourse. The results suggest that feedback combined with cognitive structuring contribute to effective teaching and learning.     

The paradigm shifts: Examining quality of teaching through assessment of student learning

Student learning is the strongest criterion for evaluating effective teaching. This article looks at three practices which partner students, teachers, and faculty peers in understanding the quality of teaching and learning through student course portfolios and teacher course portfolios, majors portfolios and teaching portfolios, and collaborative classroom assessment. Two outcomes of using a learning paradigm are building community and representing successes in higher education through documentation of effective teaching and learning.Barbara L. Cambridge, Professor of English and Associate Dean of the Faculties at Indiana University-Purdue University Indianapolis, serves on the IUPUI Program Review and Assessment Committee and recently chaired the Task Force on Accountability and Assessment for Indiana University's Strategic Directions Longrange Planning. She has made presentations and written on portfolio assessment, assessment and evaluation of faculty work, and issues in undergraduate education. Editor of theJournal of Teaching Writing, she is past President of the National Council of Writing Program Administrators.     

Self paced learning: A student perspective

Training reform in Australian Vocational Education and Training (VET) has been well under way for over a decade. Reforms include marketisation, the widespread adoption of competency-based training, and most recently an adoption of various forms of flexible delivery. One of these alternative methods of delivery is self paced learning (SPL). SPL is presented by VET management as being advantageous to students and their employers, and in addition is generally less resource-intensive than face-to-face teaching. This paper examines SPL in a TAPE college from a student viewpoint. The research, using qualitative methods, examined the experiences of SPL, in Office Administration subjects, of students who have a Non English Speaking Background (NESB) or no regular access to computers. The paper describes how they adapted, reacted and responded to self paced learning. The difficulties experienced by these students when studying by self paced learning are discussed in detail. The findings indicate that, whilst students are appreciative of the flexibility which self paced learning affords, there are some concerns about its efficacy in relation to educational outcomes.     

Self paced learning: A student perspective

Training reform in Australian Vocational Education and Training (VET) has been well under way for over a decade. Reforms include marketisation, the widespread adoption of competency-based training, and most recently an adoption of various forms of flexible delivery. One of these alternative methods of delivery is self paced learning (SPL). SPL is presented by VET management as being advantageous to students and their employers, and in addition is generally less resource-intensive than face-to-face teaching. This paper examines SPL in a TAFE college from a student viewpoint. The research, using qualitative methods, examined the experiences of SPL, in Office Administration subjects, of students who have a Non English Speaking Background (NESB) or no regular access to computers. The paper describes how they adapted, reacted and responded to self paced learning. The difficulties experienced by these students when studying by self paced learning are discussed in detail The findings indicate that, whilst students are appreciative of the flexibility which self paced learning affords, there are some concerns about its efficacy in relation to educational outcomes.     

The structure and goals of a student teaching program and the character and quality of supervisory discourse

This study compares the form and substance of supervisory discourse between university supervisors and elementary student teachers during postobservation conferences in two teacher education programs with similar organizational structures but with very different ideological orientations. One program represents a traditional-craft orientation to teacher education while the other represents an inquiry-oriented approach. The findings indicate a great deal of similarity with regard to both the form and substance of supervisory discourse in the two programs and suggest the need for changes in the organizational context of student teaching if innovations in the curriculum of student teaching are to be realized.     

The quality of student ratings of teacher behaviour

In a research project into the effectiveness of mathematics teaching in the first year of secondary education, external observers and students rated teachers' behaviour. The reliability and validity of both methods were established. The results show that teacher behaviour is assessed well when student ratings are aggregated at the classroom level. The quality of aggregated student ratings is as good as the quality of data from external observers. The predictive validity of aggregated student ratings is higher than the predictive validity of external observations when subject motivation is taken as a dependent variable.