Psychometric quality of a student evaluation of teaching survey in higher education

Student evaluations of teaching (SET) are used globally by higher education institutions for performance assessment of academic staff and evaluation of course quality. Higher education institutions commonly develop their own SETs to measure variables deemed relevant to them. However, ‘home-grown’ SETs are rarely assessed psychometrically. One potential consequence of this limitation is that an invalidated instrument may not provide accurate information for the intended purposes. Moreover, in the absence of psychometric assessment, the students’ voices collected by the SETs often fail to provide insight relative to their intended purpose. The present study evaluates a ‘home-grown’ SET using a Rasch model and confirmatory factor analysis. Our results identified weaknesses in two areas: the rating categories and the number of items used to measure the intended constructs. Suggestions are provided to address these weaknesses. This work provides an additional tool set for critical analysis of SET that is generally applicable for a variety of institutions, including those in Asia.     

Expectancy theory outcomes and student evaluations of teaching

As student evaluation of teaching (SET) instruments are increasingly administered online, research has found that the response rates have dropped significantly. Validity concerns have necessitated research that explores student motivation for completing SETs. This study uses Vroom's [(1964). Work and motivation (3rd ed.). New York, NY: John Wiley & Sons] expectancy theory to frame student focus group responses regarding their motivations for completing and not completing paper and online SETs. Results show that students consider the following outcomes when deciding whether to complete SETs: (a) course improvement, (b) appropriate instructor tenure and promotion, (c) accurate instructor ratings are available to students, (d) spending reasonable amount of time on SETs, (e) retaining anonymity, (f) avoiding social scrutiny, (g) earning points and releasing grades, and (h) being a good university citizen. Results show that the lower online response rate is largely due to students’ differing feelings of obligation in the 2 formats. Students also noted that in certain situations, students often answer SETs insincerely.     

A psychometric approach to the validation of a student evaluation of teaching instrument

The purpose of this study was to conduct a validation analysis of an SET and provide a validation framework of SETs that can be included when designing complete evaluations of teaching within higher education institutions. A series of Rasch analyses was conducted on the results of the SET, examining the responses of students within a college and three departments. Results show the majority of items were moderately difficult to endorse in the college and departments, there were issues with DIF, and two items did not consistently fit the model. The study provides an analysis framework that may aid policymakers and institutional administrators in developing higher quality SETs, and demonstrates the need for validating SETs being implemented in higher education settings.     

The relative importance of teaching evaluation criteria from the points of view of students and faculty

Abstract

The student evaluation of teaching (SET) tool is widely used to measure student satisfaction in institutions of higher education. A SET typically includes several criteria, which are assigned equal weights. The motivation for this research is to examine student and lecturer perceptions and the behaviour of the students (i.e. ratings given by them to lecturers) of various criteria on a SET. To this end, an analytic hierarchy process methodology was used to capture the importance (weights) of SET criteria from the points of view of students and lecturers; the students' actual ratings on the SET were then analysed. Results revealed statistically significant differences in the weights of the SET criteria; those weights differ for students and lecturers. However, analysis of 1436 SET forms of the same population revealed that, although students typically rate instructors very similarly on all criteria, they rate instructors higher on the criteria that are more important to them. The practical implications of this research is the reduction of the number of criteria on the SETs used for personnel decisions, while identifying for instructors and administrators those criteria that are perceived by students to be more important.     

Improving SET Response Rates: Synchronous Online Administration as a Tool to Improve Evaluation Quality

Institutions of higher education continue to migrate student evaluations of teaching (SET) from traditional, in-class paper forms to online SETs. Online SETs would favorably compare to paper-and-pencil evaluations were it not for widely reported response rate decreases that cause SET validity concerns stemming from possible nonresponse bias. To combat low response rates, one institution introduced a SET application for mobile devices and piloted formal synchronous classroom time for SET completion. This paper uses the Leverage Salience Theory to estimate the impact of these SET process changes on overall response rates, open-ended question response rates, and open end response word counts. Synchronous class time best improves SET responses when faculty encourage completion on keyboarded devices and provide students SET completion time in the first 15 min of a class meeting. Full support from administrators requires sufficient wireless signal strength, IT infrastructure, and assuring student access to devices for responses clustering around meeting times.     

Multiple dimensions of university teacher self-concept

Despite the recognised importance of positive self-concepts inmany settings, surprisingly little attention has been paid to teacherself-concepts – teachers' self-perceptions of their own teachingeffectiveness. We integrate research literatures on self-concept and onstudents' evaluations of teaching effectiveness (SETs), develop amultidimensional university teacher self-concept instrument, andevaluate its psychometric properties (factor structure, reliability,validity). A multitrait-multimethod analysis of relations betweenmultiple dimensions of teacher self-concept and corresponding SET ratingdimensions provides good support for the construct validity of teacherself-concept responses. In support of a priori hypotheses based onself-concept theory, agreement between teacher self-concepts and SETswas moderate (median r = 0.20) for teachers who had notpreviously received SET feedback, but substantially higher (median r= 0.40) for teachers who had previously received SET feedback.Implications for further research on teacher self-reflection and forimproving teaching effectiveness in higher education arediscussed.     

Student evaluations of teaching: the impact of faculty procedures on response rates

In the last 10–15 years, many institutions of higher education have switched from paper-and-pencil methods to online methods of administering student evaluations of teaching (SETs). One consequence has been a significant reduction in the response rates to such instruments. The current study was conducted to identify whether offering in-class time to students to complete online SETs would increase response rates. A quasi-experiment (nonequivalent group design) was conducted in which one group of tenured faculty instructed students to bring electronic devices with internet capabilities on a specified day and offered in-class time to students to complete online SETs. A communication protocol for faculty members’ use was developed and implemented. A comparison group of tenured faculty who did not offer in-class time for SET completion was identified and the difference-in-differences method was used to compare the previous year’s response rates for the same instructor teaching the same course across the two groups. Response rates were substantially higher when faculty provided in-class time to students to complete SETs. These results indicate that high response rates can be obtained for online SETs submitted by students in face-to-face classes if faculty communicate the importance of SETs in both their words and actions.     

Special Education Teachers’ Self-Efficacy in Implementing Social–Emotional Learning to Support Students with Learning Disabilities

This study was designed to determine how special education teachers (SETs) in Saudi Arabia perceive their self-efficacy in implementing social–emotional learning (SEL) practices to support students with learning disabilities (LD). The data were collected by surveying 109 SETs of students with LD who teach at elementary and middle schools in Riyadh. The findings demonstrate that SETs had low to moderate self-efficacy in putting SEL into practice, and the participants highlighted the importance of receiving support that can enhance their self-efficacy in SEL. Moreover, the study's findings show some statistically significant variations in the responses of the teachers, as those who attended more professional development programs reported higher levels of self-efficacy. Additionally, middle school teachers and teachers with fewer years of experience showed a higher need for receiving support to enhance their implementation of SEL. Recommendations for future studies and practices are provided.     

Does co-teaching improve academic achievement for students with disabilities? A meta-analysis

Co-teaching consists of general and special education teachers who team to instruct students with and without disabilities in the same classroom. Additionally, some students with disabilities (SWD) receive instruction in special education settings. The purpose of this meta-analysis was to compare academic achievement of SWD in co-taught classes to achievement of SWD in special education settings. A comprehensive search of published and gray literature yielded 26 eligible studies with a total of 3,714 SWD. Results indicate a moderate effect size (g = 0.47) for students in co-taught settings compared to students in special education settings. Although the findings support SWD's academic achievement in co-taught classes, more information is needed to identify what distinguishes characteristics of the SWD in co-taught versus special education settings, as well as instruction in each setting. Factors related to results are described, including the lack of information about instructional practices and equivalency of students assigned to co-taught or special education settings. Implications for future researchers are provided, along with cautions about widespread placement of SWD in either setting without monitoring that effective instructional practices are in place, and that students receive specially designed instruction as stipulated on their Individualized Education Programs.     

Advancing text-analysis to tap into the student voice: a proof-of-concept study

Abstract

Student evaluations of teaching and courses (SETs) are part of the fabric of tertiary education and quantitative ratings derived from SETs are highly valued by tertiary institutions. However, many staff do not engage meaningfully with SETs, especially if the process of analysing student feedback is cumbersome or time-consuming. To address this issue, we describe a proof-of-concept study to automate aspects of analysing student free text responses to questions. Using Quantext text analysis software, we summarise and categorise student free text responses to two questions posed as part of a larger research project which explored student perceptions of SETs. We compare human analysis of student responses with automated methods and identify some key reasons why students do not complete SETs. We conclude that the text analytic tools in Quantext have an important role in assisting teaching staff with the rigorous analysis and interpretation of SETs and that keeping teachers and students at the centre of the evaluation process is key.     

Is it possible to redress noninstructional biases in student evaluation of teaching surveys? Quantitative analysis in accounting and finance courses

Several studies have reported that student evaluation of teaching (SET) presents important problems. First, depending on the area, there are significant differences in the evaluations. Second, numerous noninstructional biases exist, such as when those teachers who award better grades obtain better SETs. Correcting the rankings by considering these biases (e.g., adjusting SETs according to the class grade) has been proposed. In this paper, we analyse a third problem: it is impossible to correct the biases because they are specific to each area, level, and even class. On a sample of 15,439 SETs, we compared the biases present in two very close areas (accounting and finance) and at two levels (undergraduate and postgraduate). Then, we used a procedure based on the analysis of residuals in OLS models to eliminate area- and level-specific biases. However, there are still latent biases apparently linked to each specific group of students.     

Honesty on student evaluations of teaching: effectiveness,purpose, and timing matter!

Student evaluations of teaching (SETs) are an important point of assessment for faculty in curriculum development, tenure and promotion decisions, and merit raises. Faculty members utilise SETs to gain feedback on their classes and, hopefully, improve them. The question of the validity of student responses on SETs is a continuing debate in higher education. The current study uses data from two universities (n = 596) to determine whether and under what conditions students are honest on in-class and online SETs, while also assessing their knowledge and attitudes about SETs. Findings reveal that, while students report a high level of honesty on SETs, they are more likely to be honest when they believe that evaluations effectively measure the quality of the course, the results improve teaching and benefit students rather than the administration, and when they are given at the end of the term. Honesty on evaluations is not associated with socio-demographic characteristics.     

Correlation between grade point averages and student evaluation of teaching scores: taking a closer look

One of the most contentious potential sources of bias is whether instructors who give higher grades receive higher ratings from students. We examined the grade point averages (GPAs) and student ratings across 2073 general education religion courses at a large private university. A moderate correlation was found between GPAs and student evaluations of teaching (SETs); however, this global correlation did not hold true for individual teachers and courses. In fact, there was a large variance in the correlations between GPAs and SETs, including some teachers with a negative correlation and a large variance between courses.     

Rate my attitude: research agendas and RateMyProfessor scores

The literature on student evaluations of teaching (SETs) generally presents two opposing camps: those who believe in the validity and usefulness of SETs, and those who do not. Some researchers have suggested that ‘SET deniers’ resist SETs because of their own poor SET results. To test this hypothesis, I analysed essays by 230 SET researchers (170 lead authors) and classified the researchers as having negative, neutral or positive attitudes towards SETs. I retrieved their RateMyProfessors.com (RMP) scores and, using logistic regression, found that lead authors with negative attitudes towards SETs were 14 times more likely to score below an estimated RMP average than lead authors with positive attitudes towards SETs. Co-authors and researchers with neutral attitudes, on the other hand, did not significantly differ from the RMP average. These results suggest that personal attitudes towards SETs may drive research findings.     

The skills cline: higher education and the supply-demand complex in South Africa

This paper investigates the relationship between Grade 12 learner preferences for study in higher education, student enrolment in higher education programmes, and student graduations in different programme areas, considering the match between these supply-side indicators and a forecast of skills demand in South Africa as a first step towards ascertaining the extent to which the higher education system is meeting the demand for skills in the labour market. While learner preferences are predominantly for study in the field of Science, Engineering and Technology (SET), student graduations are predominantly in the Humanities. And since a large proportion of scarce-skills areas are SET-related, this creates a mismatch that has implications for the economic development of the country. Simultaneously, however, the demand for educators (school teachers and academics) is even greater: the strongest discrete demand is for educators, followed by that for managers. The paper concludes by urging more concerted development of teachers and managers.     

"十一五"期间中小学教师需求预测及建议——以重庆市为例

据重庆市各年份统计年鉴公布数据显示,重庆市年出生人口总体呈现下降趋势,这一变化必然带来在校生规模逐渐缩小和教师的需求量减少。这对预测重庆市未来5年间中小学在校生人数和教师需求量的变化,做好中小学发展规划,加快基础教育结构调整和改革,以及对高等师范院校的学科专业结构、办学规模和教育教学改革都很有参考价值。     

The effect of incentives and other instructor-driven strategies to increase online student evaluation response rates

Course evaluations (often termed student evaluations of teaching or SETs) are pervasive in higher education. As SETs increasingly shift from pencil-and-paper to online, concerns grow over the lower response rates that typically accompany online SETs. This study of online SET response rates examined data from 678 faculty respondents and student response rates from an entire semester. The analysis focused on those tactics that faculty employ to raise response rates for their courses, and explored instructor and course characteristics as contributing factors. A comprehensive regression model was evaluated to determine the most effective tactics and characteristics. Using incentives had the most impact on response rates. Other effective tactics that increase response rates include reminding students to take the evaluation, explaining how the evaluations would be used to improve instruction, sending personal emails and posting reminders on Blackboard^®. Incentives are not widely used; however, findings suggest that non-point incentives work as well as point-based ones, as do simple-to-administer minimum class-wide response rate expectations (compared to individual completion).     

Whose feedback? A multilevel analysis of student completion of end-of-term teaching evaluations

Student evaluation of teaching (SET) is now common practice across higher education, with the results used for both course improvement and quality assurance purposes. While much research has examined the validity of SETs for measuring teaching quality, few studies have investigated the factors that influence student participation in the SET process. This study aimed to address this deficit through the analysis of an SET respondent pool at a large Canadian research-intensive university. The findings were largely consistent with available research (showing influence of student gender, age, specialisation area and final grade on SET completion). However, the study also identified additional influential course-specific factors such as term of study, course year level and course type as statistically significant. Collectively, such findings point to substantively significant patterns of bias in the characteristics of the respondent pool. Further research is needed to specify and quantify the impact (if any) on SET scores. We conclude, however, by recommending that such bias does not invalidate SET implementation, but instead should be embraced and reported within standard institutional practice, allowing better understanding of feedback received, and driving future efforts at recruiting student respondents.     

Towards another kind of borderlessness: online students with disabilities

Online learning is crucial to success for higher education institutions. Whilst the existing literature predominantly focused on its economic advantages, we focused on its inclusivity. At an online learning unit of a UK university, the number of students with disabilities (SWD) is three times higher than the national average. Having a degree makes significant financial and psychological differences in the lives of SWD. Though recent literature focused on inclusivity of online learning, an appraisal of first-hand experience of SWD studying online is a missing perspective. Accordingly, we aimed to explore their experience, using thematic analysis of semi-structured interviews involving ten SWD. Three themes emerged: (1) having control over studies as an advantage of online learning, (2) personal touch helps SWD’s online learning, and (3) challenges SWD experience with the social element of online learning. Our findings will help to develop the inclusivity of online learning to a new level.     

Validity,Reliability, and Potential Bias of Short Forms of Students' Evaluation of Teaching: The Case of UAE University

Students' opinions continue to be a significant factor in the evaluation of teaching in higher education institutions. The purpose of this study was to psychometrically assess short students evaluation of teaching (SET) forms using the UAE University form as a model. The study evaluated the form validity, reliability, the overall question, and potential bias with respect to gender, college, grade point average, expected grade, and class size. A total of 3,661 students participated in this study in different random samples. Results indicated that the short SET form lacked content validity and could not identify key dimensions of evaluating teaching effectiveness. The form showed stability over time and acceptable internal reliability. Results indicated also that there was a potential bias due to college, expected grade, and class size, but there was no relationship between grade point average and students' ratings. It was concluded that short SET forms do not cover all domain content and unable to provide teachers with enough information for the improvement of teaching.