Online student evaluation improves Course Experience Questionnaire results in a physiotherapy program

This paper reports the use of an online student evaluation system, Course Experience on the Web (CEW), in a physiotherapy program to improve their Course Experience Questionnaire (CEQ) results. CEW comprises a course survey instrument modelled on the CEQ and a tailored unit survey instrument. Closure of the feedback loop is integral in the CEW system. Analysis of the data shows that the students’ evaluation in their final year of the program is closely correlated with their CEQ results. Increases in the CEQ scores from 2001–04 included an increase in the Good Teaching Scale (27.5), Generic Skills Scale (10.3) and Overall Satisfaction Index (29.3). By using CEW, academics at the School of Physiotherapy were able to determine students’ perceptions during the course, make changes to teaching and learning, where appropriate, in a timely manner and, as a result, the CEQ scores were improved markedly.     

Validating a teaching survey which drives increased response rates in a unit survey

At Curtin University, student perceptions of teaching and learning are gathered and reported online through eVALUate, a system that includes separate unit and teaching surveys. This article reports the development and validation of one of those surveys, the eVALUate teaching survey, which was developed based on the research literature on excellent teaching and evaluation. Since its development in 2006, repeated statistical testing using progressively larger samples has shown that the survey is valid and reliable. Moreover, the way in which the teaching survey is deployed within eVALUate, appended to the unit survey, which provides crucial institutional data, has significantly increased university response rates. This validated instrument is used for self-reflection, professional development, and rewarding staff.     

The relationship between student understanding,satisfaction and performance in an Australian engineering programme

It has often been suggested that actual or anticipated final grades may influence the ratings given by students in student experience surveys but few studies have been able to test this using actual grades. A study was carried out involving six courses over all four year levels of an undergraduate engineering programme, where students were asked to identify themselves in an experience survey by providing their student ID on the survey form. The aim of the study was to investigate a number of questions related to the readiness of students to identify themselves, and to examine any correlation between final examination grades, ratings of student satisfaction and the students’ perception of their level of understanding of material in their courses. Students were discovered to have a poor idea of how well they understand the concepts presented in their courses. This lack of an accurate idea of their own understanding is particularly important because ‘student understanding’ correlated to the ratings they gave to the course. Ratings were largely unaffected by final marks but students who gave their ID outperformed those who did not in end‐of‐year examinations. Higher year level students were more inclined to identify themselves and ratings tended to increase with year level.     

Enhancing student engagement in student experience surveys: a mixed methods study

Background

Measuring the student experience is becoming increasingly important in higher education in the UK. Student experience surveys are used as indicators of quality and form the basis of rankings of higher education institutions. They are also used by them as tools to assist their quality enhancement initiatives. However, these surveys frequently suffer from low response rates, which can reduce the reliability and usefulness of their data. The UK Postgraduate Taught Experience Survey (PTES) is a relatively new survey and suffers from a low response rate. As this survey is new, little is known about why students do not respond to it.

Purpose

This study aimed to explore the reasons why postgraduate students do not respond to the PTES.

Sample

Three hundred and fifty-five postgraduate taught students from four health faculties in one UK higher education institution completed an online survey. Of these, seven participated in one of two focus groups.

Design and methods

The online survey was completed both by students who completed the PTES in 2011 and those who did not. This provided us with cross-sectional data to compare both groups’ knowledge of PTES and their reasons for completing or not completing it. We used multivariate regression analysis to explore which variables were associated with response to PTES. We led two focus groups to explore the themes that emerged from the survey in more depth. This data was analysed by two researchers using thematic analysis.

Results

The cross-sectional data found that students who were not clear about the purpose of PTES were less likely to respond, independent of other potential predictor variables. Focus group data indicated that if postgraduate students felt a stronger connection to the university community they may be more likely to respond to PTES.

Conclusions

This study suggests that higher education institutions may wish to review their strategies for advertising student experience surveys to focus more on their purpose rather than their impact.     

Validation of a research-based student survey of instruction in a college of education

As a preparer of students for the profession of teaching, a college, school, or department of education in higher education has an expectation that instructors teach competently. In higher education, the student survey of instruction is one of the most important among other indicators of teaching effectiveness. This article presents a validation of a new student survey of instruction in a college of education. In this validation, we found the new survey to be as troublesome as the previous survey. Four major findings emerged from this study that offer hope for a new and improved survey of instructional effectiveness. These findings involve estimating reliability, subscore validity, course relevance and rigor of the instructor, and construct-irrelevant variables that threaten validity. Given the failed validation, many cogent observations provide guidance for the design and development of improved forms that may overcome these threats to validity and guide others in validating teaching effectiveness surveys.     

A psychometric approach to the validation of a student evaluation of teaching instrument

The purpose of this study was to conduct a validation analysis of an SET and provide a validation framework of SETs that can be included when designing complete evaluations of teaching within higher education institutions. A series of Rasch analyses was conducted on the results of the SET, examining the responses of students within a college and three departments. Results show the majority of items were moderately difficult to endorse in the college and departments, there were issues with DIF, and two items did not consistently fit the model. The study provides an analysis framework that may aid policymakers and institutional administrators in developing higher quality SETs, and demonstrates the need for validating SETs being implemented in higher education settings.     

电工电子技术实验学生评价体系设计

概括介绍了"教学框架"的基本理论,分析了电工电子技术实验教学中学生评价体系现状和存在的问题,构建了电工电子技术实验教学框架,并重点阐述了学生实验评价体系的构建原则、评价要点、评价标准、权重和评价等级。相关研究内容对于深化学生评价体系创新和实验教学改革具有重要意义。     

The effect of incentives and other instructor-driven strategies to increase online student evaluation response rates

Course evaluations (often termed student evaluations of teaching or SETs) are pervasive in higher education. As SETs increasingly shift from pencil-and-paper to online, concerns grow over the lower response rates that typically accompany online SETs. This study of online SET response rates examined data from 678 faculty respondents and student response rates from an entire semester. The analysis focused on those tactics that faculty employ to raise response rates for their courses, and explored instructor and course characteristics as contributing factors. A comprehensive regression model was evaluated to determine the most effective tactics and characteristics. Using incentives had the most impact on response rates. Other effective tactics that increase response rates include reminding students to take the evaluation, explaining how the evaluations would be used to improve instruction, sending personal emails and posting reminders on Blackboard^®. Incentives are not widely used; however, findings suggest that non-point incentives work as well as point-based ones, as do simple-to-administer minimum class-wide response rate expectations (compared to individual completion).     

大学生参与教师教学评价的调查研究

大学生参与教师教学评价是提升高校教育教学质量的重要手段,也是大学生发挥主体作用的具体体现。从问卷和访谈调查中发现,高校虽然普遍开展了学生评教活动,但在评价的时机、形式、内容、结果利用以及师生的认识和态度等方面,仍然存在着一些不容忽视的问题。高等学校应拓宽教学评价的内容,开辟学生参与评教的多种途径,开发适合各类课程的评价工具,使学生的评教真正有助于教师的教学改革。     

Psychometric quality of a student evaluation of teaching survey in higher education

Student evaluations of teaching (SET) are used globally by higher education institutions for performance assessment of academic staff and evaluation of course quality. Higher education institutions commonly develop their own SETs to measure variables deemed relevant to them. However, ‘home-grown’ SETs are rarely assessed psychometrically. One potential consequence of this limitation is that an invalidated instrument may not provide accurate information for the intended purposes. Moreover, in the absence of psychometric assessment, the students’ voices collected by the SETs often fail to provide insight relative to their intended purpose. The present study evaluates a ‘home-grown’ SET using a Rasch model and confirmatory factor analysis. Our results identified weaknesses in two areas: the rating categories and the number of items used to measure the intended constructs. Suggestions are provided to address these weaknesses. This work provides an additional tool set for critical analysis of SET that is generally applicable for a variety of institutions, including those in Asia.     

Student faculty evaluation (SFE) at Jordanian universities: a student perspective

This study is the first to address student evaluation of faculty members (SFE) from a student perspective at a major Jordanian public university using a comprehensive (71-item) questionnaire administered to 620 undergraduates. Addressed are students’ perceptions of the SFE process in terms of: (a) their paper-based vs. online-format preferences; (b) their beliefs about the process; (c) the standards they adopt in the evaluation; (d) the fruitfulness of SFE outcomes; and (e) their opinions about the evaluation instruments currently used at their institutions. It also explores whether students’ beliefs vary according to their gender, GPA and college (Educational Sciences vs. Sciences). The results reveal that most students prefer online evaluation, adopt academic rather than interpersonal criteria in their evaluation, have a low level of satisfaction with the impact of previous evaluations on faculty teaching behaviours and the evaluation instruments used at their institution. Additionally, the findings suggest that gender has a less influential role in shaping students’ beliefs about the SFE process compared to their college or GPA. The findings of this study and their implications are very useful pointers for faculty and higher education administrators, faculty members and students towards further improvement at higher education institutions.     

The performance of a student evaluation of teaching system

Student evaluation of teaching (SET) is now commonplace in many universities internationally. While much effort has been devoted to examining the statistical validity of SET instruments, there has been limited examination of the methodological and consequential validity (together referred to as ‘utility’) of the ways in which SET data are used. This paper examines the SET system at Deakin University from the perspective of utility. It draws on publicly available SET results for an entire annual cycle of unit offerings. Consideration is given to the representativeness of the data produced, and to the utility of the data reported, by the system. While this investigation focuses on the SET system currently employed at Deakin University, it offers both an analysis methodology and conclusions that can be applied more generally.     

Faculty attitudes toward student evaluation of teaching

This study dealt with university faculty attitudes toward the validity of and varying uses for student evaluations of teaching. It was hypothesized that faculty attitudes toward the potential uses of student evaluations would be systematically related to a professor's a priori frame of reference regarding the validity of these instruments. On the basis of responses to selected questionnaire items, faculty were divided into those whose frame of reference (1) opposed the idea that student evaluations could be made valid, (2) supported the idea that existing evaluation forms were valid, and (3) were uncommitted regarding this validity question. A fourth group, administrators at the university, were also surveyed. Large and consistent differences among these groups were found on policy matters such as the use of student evaluations in tenure and promotion decisions, in providing information to students to help them choose courses, and on a variety of other issues. In each case the results supported the hypothesis.Authors' names are ordered randomly.     

Correlation between grade point averages and student evaluation of teaching scores: taking a closer look

One of the most contentious potential sources of bias is whether instructors who give higher grades receive higher ratings from students. We examined the grade point averages (GPAs) and student ratings across 2073 general education religion courses at a large private university. A moderate correlation was found between GPAs and student evaluations of teaching (SETs); however, this global correlation did not hold true for individual teachers and courses. In fact, there was a large variance in the correlations between GPAs and SETs, including some teachers with a negative correlation and a large variance between courses.     

A snapshot of the student experience: exploring student satisfaction through the use of photographic elicitation

Student satisfaction is both an important and yet controversial issue within the higher education sector, which is typically measured through policy-driven metrics such as the National Student Survey. However, less is understood about the qualities of a satisfying student experience ‘as lived’ from the perspective of the student, thus questioning the adequacy of such measures. In response to this, the current study used student-driven photographic elicitation as a means of more adequately capturing the holistic student experience. This entailed nine final-year undergraduate psychology students who each gathered a series of photos, which formed the basis for discussion in an interview. Thematic analysis of the narratives of the interview discussions revealed several main themes surrounding their experiences. These were: ‘Learning Environment’, ‘Work–Life Balance’ and ‘Wider University Community’. Findings are discussed with reference to the implications of student satisfaction, and national metrics used for measuring it, for institutional policies of recruitment and retention.     

基于AHP下的江苏省高校体育课程学生评价体系研究

对江苏省普通高校体育课程进行了调查研究。用层次分析法(AHP),采用多元综合评价的方式,探讨了高校学生体育成绩评价方法,并提出评价过程中应重视学生的学习效果,重视社会有关方面的评价意见,以此尽量能客观的描述每一个个体,全面、客观地给出一个完整评价。     

Relationship of changes in student motivation to student evaluations of instruction

Research has demonstrated that student evaluations of instruction are influenced by variables extraneous to the instructional procedures being evaluated. One of the most important of these is the student's motivation to take the course. The Instructional Development and Effectiveness Assessment (IDEA) system controls this variable by comparing a course evaluation to a norm group of courses having students with similar motivation. The present study examined the possibility that the IDEA procedure of having students rate their precourse motivation at theend of a course might be unacceptable, because the rating would be influenced by experiences within the course itself. The data indicated that postcourse ratings of precourse motivation do deviate somewhat from actual precourse ratings, but the deviation is not of an order of magnitude which would seriously distort the interpretation of the ratings.     

Updating RIGs: including the systematic influence of online study on student evaluation of teaching

Based on student evaluation of teaching (SET) ratings from 1,432 units of study over a period of a year, representing 74,490 individual sets of ratings, and including a significant number of units offered in wholly online mode, we confirm the significant influence of class size, year level, and discipline area on at least some SET ratings. We also find online mode of offer to significantly influence at least some SET ratings. We reveal both the statistical significance and effect sizes of these influences, and find that the magnitudes of the effect sizes of all factors are small, but potentially cumulative. We also show that the influence of online mode of offer is of the same magnitude as the other 3 factors. These results support and extend the rating interpretation guides (RIGs) model proposed by Neumann and colleagues, and we present a general method for the development of a RIGs system.     

The potential of student narratives to enhance quality in higher education

University policies are increasingly developed with reference to students’ learning experiences, with a focus on the concept of the ‘student voice’. Yet the ‘student voice’ is difficult to define and emphasis is often placed on numerical performance indicators. A diverse student population has wide-ranging educational experiences, which may not be easily captured within the broad categories provided by traditional survey tools, which can drown out the rich, varied and gradual processes of individual development. There is no single tool that can be used to measure students’ experiences. This paper draws on findings from four narrative inquiry studies, carried out in the United Kingdom, Australia, South Africa and New Zealand, to illustrate how a narrative approach could be used to complement performance indicators. This provides a richer context for educators’ understanding of students’ experiences and for supporting and setting institutional agendas.     

Multilevel analysis of the effects of student and course characteristics on satisfaction in undergraduate liberal arts courses

The purpose of this study is to analyze the effects of student and course characteristics on student satisfaction in courses by using a two-level hierarchical linear model (HLM). Based on literature reviews, 13 research variables (1 student gender, 2 student academic year, 3 student major, 4 student reason for taking the course, 5 student level of course participation, 6 student expected grade, 7 student achieved grade, 8 faculty gender, 9 faculty age, 10 faculty status, 11 academic field of the course, 12 class size, and 13 course type) were selected and specified as fixed effects in the analysis model. Data were 57,216 ratings of 1,481 undergraduate liberal arts courses at Seoul National University in 2006. The result of unconditional model analysis revealed that student characteristics’ effects on student satisfaction in courses (within-course effects, 88.1%) were much larger than course characteristics’ effects (between-courses effects, 11.9%). The result of conditional model analysis specifying student and course level predictors revealed that those 12 research variables, with the exception of student gender, had statistically significant effects on student satisfaction in courses. The explained variance was 22.0% in student level, 65.8% in course level, and 27.2% of the total variance. An earlier version of this paper was presented at the 8^th International Conference on Education Research, Seoul National University, Seoul, Korea, October 23–25, 2007.