A psychometric approach to the validation of a student evaluation of teaching instrument

The purpose of this study was to conduct a validation analysis of an SET and provide a validation framework of SETs that can be included when designing complete evaluations of teaching within higher education institutions. A series of Rasch analyses was conducted on the results of the SET, examining the responses of students within a college and three departments. Results show the majority of items were moderately difficult to endorse in the college and departments, there were issues with DIF, and two items did not consistently fit the model. The study provides an analysis framework that may aid policymakers and institutional administrators in developing higher quality SETs, and demonstrates the need for validating SETs being implemented in higher education settings.     

Rate my attitude: research agendas and RateMyProfessor scores

The literature on student evaluations of teaching (SETs) generally presents two opposing camps: those who believe in the validity and usefulness of SETs, and those who do not. Some researchers have suggested that ‘SET deniers’ resist SETs because of their own poor SET results. To test this hypothesis, I analysed essays by 230 SET researchers (170 lead authors) and classified the researchers as having negative, neutral or positive attitudes towards SETs. I retrieved their RateMyProfessors.com (RMP) scores and, using logistic regression, found that lead authors with negative attitudes towards SETs were 14 times more likely to score below an estimated RMP average than lead authors with positive attitudes towards SETs. Co-authors and researchers with neutral attitudes, on the other hand, did not significantly differ from the RMP average. These results suggest that personal attitudes towards SETs may drive research findings.     

The relative importance of teaching evaluation criteria from the points of view of students and faculty

Abstract

The student evaluation of teaching (SET) tool is widely used to measure student satisfaction in institutions of higher education. A SET typically includes several criteria, which are assigned equal weights. The motivation for this research is to examine student and lecturer perceptions and the behaviour of the students (i.e. ratings given by them to lecturers) of various criteria on a SET. To this end, an analytic hierarchy process methodology was used to capture the importance (weights) of SET criteria from the points of view of students and lecturers; the students' actual ratings on the SET were then analysed. Results revealed statistically significant differences in the weights of the SET criteria; those weights differ for students and lecturers. However, analysis of 1436 SET forms of the same population revealed that, although students typically rate instructors very similarly on all criteria, they rate instructors higher on the criteria that are more important to them. The practical implications of this research is the reduction of the number of criteria on the SETs used for personnel decisions, while identifying for instructors and administrators those criteria that are perceived by students to be more important.     

Student evaluations of teaching: the impact of faculty procedures on response rates

In the last 10–15 years, many institutions of higher education have switched from paper-and-pencil methods to online methods of administering student evaluations of teaching (SETs). One consequence has been a significant reduction in the response rates to such instruments. The current study was conducted to identify whether offering in-class time to students to complete online SETs would increase response rates. A quasi-experiment (nonequivalent group design) was conducted in which one group of tenured faculty instructed students to bring electronic devices with internet capabilities on a specified day and offered in-class time to students to complete online SETs. A communication protocol for faculty members’ use was developed and implemented. A comparison group of tenured faculty who did not offer in-class time for SET completion was identified and the difference-in-differences method was used to compare the previous year’s response rates for the same instructor teaching the same course across the two groups. Response rates were substantially higher when faculty provided in-class time to students to complete SETs. These results indicate that high response rates can be obtained for online SETs submitted by students in face-to-face classes if faculty communicate the importance of SETs in both their words and actions.     

Improving SET Response Rates: Synchronous Online Administration as a Tool to Improve Evaluation Quality

Institutions of higher education continue to migrate student evaluations of teaching (SET) from traditional, in-class paper forms to online SETs. Online SETs would favorably compare to paper-and-pencil evaluations were it not for widely reported response rate decreases that cause SET validity concerns stemming from possible nonresponse bias. To combat low response rates, one institution introduced a SET application for mobile devices and piloted formal synchronous classroom time for SET completion. This paper uses the Leverage Salience Theory to estimate the impact of these SET process changes on overall response rates, open-ended question response rates, and open end response word counts. Synchronous class time best improves SET responses when faculty encourage completion on keyboarded devices and provide students SET completion time in the first 15 min of a class meeting. Full support from administrators requires sufficient wireless signal strength, IT infrastructure, and assuring student access to devices for responses clustering around meeting times.     

The effect of incentives and other instructor-driven strategies to increase online student evaluation response rates

Course evaluations (often termed student evaluations of teaching or SETs) are pervasive in higher education. As SETs increasingly shift from pencil-and-paper to online, concerns grow over the lower response rates that typically accompany online SETs. This study of online SET response rates examined data from 678 faculty respondents and student response rates from an entire semester. The analysis focused on those tactics that faculty employ to raise response rates for their courses, and explored instructor and course characteristics as contributing factors. A comprehensive regression model was evaluated to determine the most effective tactics and characteristics. Using incentives had the most impact on response rates. Other effective tactics that increase response rates include reminding students to take the evaluation, explaining how the evaluations would be used to improve instruction, sending personal emails and posting reminders on Blackboard^®. Incentives are not widely used; however, findings suggest that non-point incentives work as well as point-based ones, as do simple-to-administer minimum class-wide response rate expectations (compared to individual completion).     

Variability in Demand for Special Education Teachers: Indicators,Explanations, and Impacts

After decades of growth, the number of special education teachers (SETs) has begun to decline. In 2009, U.S. schools employed 13% fewer SETs than in 2006. The number of annual new hires of SETs also dropped dramatically in some states. The onset of these declines predated the economic downturn of 2008 and resulted in part from a steady decline since 2005 in the number of students with disabilities (SWD) served. We consider factors that may be contributing to declining demand for SETs, among them the number of SWD, service delivery, the economic downturn, and present supporting evidence. We also consider the potential impact of reduced demand on SET supply, teacher education, equitable distribution of teachers, and, most importantly, outcomes for SWD. We call for vigilance and monitoring of SET employment data to assure that all students receive the appropriate education to which they are entitled.     

Honesty on student evaluations of teaching: effectiveness,purpose, and timing matter!

Student evaluations of teaching (SETs) are an important point of assessment for faculty in curriculum development, tenure and promotion decisions, and merit raises. Faculty members utilise SETs to gain feedback on their classes and, hopefully, improve them. The question of the validity of student responses on SETs is a continuing debate in higher education. The current study uses data from two universities (n = 596) to determine whether and under what conditions students are honest on in-class and online SETs, while also assessing their knowledge and attitudes about SETs. Findings reveal that, while students report a high level of honesty on SETs, they are more likely to be honest when they believe that evaluations effectively measure the quality of the course, the results improve teaching and benefit students rather than the administration, and when they are given at the end of the term. Honesty on evaluations is not associated with socio-demographic characteristics.     

Management by results: Student evaluation of faculty teaching and the mis-measurement of performance

Using data on 4 years of courses at American University, regression results show that actual grades have a significant, positive effect on student evaluations of teaching (SETs), controlling for expected grade and fixed effects for both faculty and courses, and for possible endogeneity. Implications are that the SET is a faulty measure of teaching quality and grades a faulty signal of future job performance. Students, faculty, and provost appear to be engaged in an individually rational but socially destructive game of grade inflation centered on the link between SETs and grades. When performance is hard to measure, pay-for-performance, embodied by the link between SETs and faculty pay, may have unintended adverse consequences.     

Internal quality assurance systems in Portugal: what their strengths and weaknesses reveal

In Portugal, the agency for assessment and accreditation of higher education has recently included in its remit, beyond programme accreditation, the certification of internal quality assurance systems. This implies lighter touch accreditation and aims to direct institutions towards improvement, in addition to accountability. Twelve institutions have already undertaken the certification, and both self-assessment and external assessment reports are available. Based on the qualitative analysis of the nature of institutional strengths and weaknesses highlighted in these evaluation reports, the paper aims to understand whether the identified strengths and weaknesses are related to procedural and organisational matters or to cultural change (values and beliefs), in turn offering an insight into the quality culture(s) which characterise higher education institutions in Portugal. Findings suggest that the quality culture of the analysed institutions is somewhere between responsive and reactive. Overall, all reports dwell more on the prioritisation of formal and structural procedures, both regarding strengths and weaknesses. External reports point towards more weaknesses related to stakeholders’ participation. Both aspects are more frequent in polytechnics than in universities. These findings suggest that polytechnics are more reactive, whereas universities are more responsive. Therefore, accountability apparently continues to be, for the time being, a more pressing concern than improvement.     

Students’ writing ‘in transition’ from A-levels to university: how assessment drives students’ understandings,practices and discourses

The lament that ‘students can’t write’ remains loud and defiant, even after years of research pointing to the myriad factors that make students’ writing challenging, particularly when they move into university. This paper reports on a longitudinal, ethnographic study which explored students’ writing ‘in transition’, from A-levels to university in the UK, through the critical lens offered by the academic literacies conceptual framing. This paper offers critical analysis of the ways that students, teachers and institutions position writing at A-level and university, exploring the assumptions and beliefs that underpin their understandings and practices using Ivani?’s framework of discourses of writing. The analysis proposes that the centrality of assessment in the treatment of language at both levels creates an ‘assessment discourse of writing’, which originates in school, and becomes a defining and restrictive frame for students’ writing as they move into higher education. The analysis further suggests that assessment is the principal cause for the students’ challenges with adapting to the writing requirements of university. Moreover, assessment is used as a metalanguage for discussing writing at A-levels, and can become an unhelpful ‘anchor of continuance’ for students as they move into university.     

How to cope with stress in special needs education? Stress-inducing dysfunctional cognitions of teacher students: the perspective of professionalisation

The goal of the present study is to identify prospective special education teachers (SETs) who may have difficulties in coping with occupational stresses and burdens. International comparative studies show that SETs have a higher level of stress than their colleagues who work in the general school system. Compared with teachers in regular schools, SETs leave their profession at a higher rate. For this reason, the study focuses on so-called dysfunctional cognitions, which might increase stress in strenuous occupational situations and which have long-term harmful effects on health. The construct of dysfunctional cognitions is supposed to be a basis for developing supportive measures for teacher students with low stress tolerance. In the framework of the present study located in the German educational system, teacher students (N = 333) from the special needs areas, including learning disability, mental disability, emotional/social disorders and sensory disabilities (hearing impairment, or speech or language disorders), are taken into account. A hierarchical cluster analysis and a discriminant analysis were used to identify four different clusters: Cluster 1 shows the lowest attitude level of equally all dysfunctional cognitions. Clusters 2 and 3 are characterised by higher degrees, especially of ‘dependency’, ‘internalisation of failure’ and ‘personal standards (perfectionism)’. The fourth profile indicates the highest degree of the dysfunctional cognitions. Lower levels of dysfunctional cognitions (with the exception of perfectionism) occur together with a higher expectation of self-efficacy. No cluster applies for all teacher students of one specific special needs area. Prospective teachers in the area of mental disability are overrepresented in cluster 3, with a high level of internalisation of failure and dependency. Teacher students in the area of emotional/social disorders belong, to a considerable degree, to the ‘risk’ cluster. The profiles are used to develop measures for those teacher students who may experience difficulties in stressful occupational situations. Therefore, these student teachers need to be closely supported through additional intensive training elements and balanced career counselling.     

Exploring the impact of faculty reflection on weekly student evaluations of teaching

This exploratory study considered Larrivee’s assessment of teachers’ reflective practice levels by using a formative, weekly, online student evaluation of teaching (SET) tool through a virtual learning environment (VLE) as a means to encourage reflective practice. In‐depth interviews were conducted with six faculty members in three departments at a university college in the UK. The study found that: (a) faculty who experienced surface‐level reflection were more likely to have a reactive reflection style; and (b) faculty who experienced higher levels of reflection were more likely to have a proactive reflection style. Overall, the tool was found to be an efficient means of encouraging reflection by all participants and demonstrated that reflective practice could come about as a result of these weekly formative SETs. The study concludes with suggestions for academic development and future research on reflection that could be conducted using SETs via a VLE.     

Authentically assessing graduate teaching: outside and beyond neo-liberal constructs

In this paper, we challenge the current focus on ‘best practice’, graduate teacher tests, and student test scores as the panacea for ensuring teaching quality and argue for ways of thinking about evidence of quality beginning teaching outside and beyond the current neoliberal accountability discourses circulating in Australia and other countries. We suggest that teacher educators need to reinsert themselves as key players in the debates around quality beginning teaching, rather than being viewed as a source of the problem. To enable teacher educators to assume accountability for quality beginning teachers, we propose the framework of a capstone teacher performance assessment—a structured portfolio called the Authentic Teacher Assessment (ATA)—and examine examples of these assessments through the lens of critical discourse analysis. As a measure of ‘readiness to teach’, the ATA is compared with supervising teachers’ assessments of preservice teachers. We argue that structured portfolios that include artefacts derived from preservice teachers’ practice in classrooms along with graduate teacher self assessments provide a stronger accountability measure of effective beginning teaching and demonstrably address the current anxiety regarding ‘evidence’. We suggest that such an approach should be reliable enough to be ‘read’ by external assessors (and moderated across other teacher education institutions). Rigorous research on a national basis is called for in order to develop and implement a structured portfolio as rich evidence of graduates’ quality and readiness to teach.     

Longitudinal analysis of the environmental attitudes of university students

This article addresses the important questions that higher education institutions ask concerning their impact on their students’ sustainability-related attributes ‘How do our students’ worldviews change as they experience higher education with us?’ The process of monitoring such a dynamic entity is fraught with statistical complexity but may not be impossible for an institution willing to ask whether or not its educational efforts in ‘education for sustainability’, ‘education for sustainable development’ or ‘environmental education’, and campus sustainability developments, are paralleled by changes in the attitudes of its students. We describe here a longitudinal survey process based on the revised New Ecological Paradigm scale, with two cohorts of students, in three programmes of study, operating over four years, with multiple survey inputs by each student. We implemented the longitudinal analysis using a linear mixed-effects model and describe here the development and testing of this model. We conclude that higher education institutions can benchmark the sustainability attributes of their students and monitor changes, if they are minded to. We invite higher education practitioners worldwide to join us in further developing suitable research instruments, processes and statistical models, and in further analysing the assumptions that link higher education to sustainability and to global citizenship.     

Multiple dimensions of university teacher self-concept

Despite the recognised importance of positive self-concepts inmany settings, surprisingly little attention has been paid to teacherself-concepts – teachers' self-perceptions of their own teachingeffectiveness. We integrate research literatures on self-concept and onstudents' evaluations of teaching effectiveness (SETs), develop amultidimensional university teacher self-concept instrument, andevaluate its psychometric properties (factor structure, reliability,validity). A multitrait-multimethod analysis of relations betweenmultiple dimensions of teacher self-concept and corresponding SET ratingdimensions provides good support for the construct validity of teacherself-concept responses. In support of a priori hypotheses based onself-concept theory, agreement between teacher self-concepts and SETswas moderate (median r = 0.20) for teachers who had notpreviously received SET feedback, but substantially higher (median r= 0.40) for teachers who had previously received SET feedback.Implications for further research on teacher self-reflection and forimproving teaching effectiveness in higher education arediscussed.     

Real GPA and Real SET: two antidotes to greed,sloth and cowardice in the college classroom

Recently published evidence of limited learning among American college students confirms the damage done when students, faculty and institutions pursue interests that conflict with the educational process. The ‘disengagement compact’ in which faculty tacitly trade lenient workloads and grading for higher student evaluation of teaching (SET) scores and fewer complaints from students does damage wherever it operates. The work of Johnson confirms the link between SET and grade inflation. We propose a modification of an earlier grade index, the Real Grade Point Average (GPA), and propose as well an index for SET scores, the Real SET, to make inflated grades and inflated SET scores more visible. Used by institutions, parts of institutions or individual faculty, Real GPA and Real SET would encourage and protect faculty and students who offer or seek out educational experiences that have not been deflected by greed, sloth or cowardice.     

Whose feedback? A multilevel analysis of student completion of end-of-term teaching evaluations

Student evaluation of teaching (SET) is now common practice across higher education, with the results used for both course improvement and quality assurance purposes. While much research has examined the validity of SETs for measuring teaching quality, few studies have investigated the factors that influence student participation in the SET process. This study aimed to address this deficit through the analysis of an SET respondent pool at a large Canadian research-intensive university. The findings were largely consistent with available research (showing influence of student gender, age, specialisation area and final grade on SET completion). However, the study also identified additional influential course-specific factors such as term of study, course year level and course type as statistically significant. Collectively, such findings point to substantively significant patterns of bias in the characteristics of the respondent pool. Further research is needed to specify and quantify the impact (if any) on SET scores. We conclude, however, by recommending that such bias does not invalidate SET implementation, but instead should be embraced and reported within standard institutional practice, allowing better understanding of feedback received, and driving future efforts at recruiting student respondents.     

Competition unleashed: horizontal differentiation in German higher education

Unlike in the US and other western countries, the higher education market in Germany has been sheltered from competition. This changed recently, when a governmental ‘excellence initiative’ began to allocate substantial supplementary funds dependent on higher education institutions’ performance, in 2005. This study is aimed at assessing differentiation tendencies arising from the acute need to become more discernible in an increasingly competitive environment. Based on a content analysis of the full sample of German higher education institutions’ mission statements, we measure the degree of horizontal differentiation among institutions, applying correspondence and cluster analysis techniques. We conclude that horizontal differentiation is not incisive to date, and identify idle potential for creating a more perceptible brand personality.     

Testing Times: Questions concerning assessment for school improvement

Contemporary education now appears to be dominated by the continual drive for improvement measured against the assessment of what students have learned. It is our contention that a foundational relation with assessment organises contemporary education.

Here we draw on a ‘way of thinking’ that is deconstructive in its intent. Such thinking makes clear the vicious circularity of the argument for improvement, wherein assessment valorised in discourses of improvement provides not only a rationalisation for improvement via assessment, but also the very means of achieving such possibilities via targets grounded in limited specifications of assessment.

On reading Heidegger's ‘question concerning technology’ we sought to reconsider the vicious circle of improvement in relation to Being. We claim that the means‐ends driven technology of assessment, rather than being at our disposal and under our control, only serves to reveals the Real to us in accordance with the restricting principle of reason.

The principle of reason, we argue, grounds ‘Enframing’ that ranks and orders the very beings of education as objects to produce an objective ‘world as picture’, rather than opening the possibility of their identity as belongings with a movement of difference.

So, ‘improvement’ becomes normative and binding for institutions and practices on grounds of the principle of assessment, and renders agents of education as functionaries of ‘Enframing’.