Improving SET Response Rates: Synchronous Online Administration as a Tool to Improve Evaluation Quality

Institutions of higher education continue to migrate student evaluations of teaching (SET) from traditional, in-class paper forms to online SETs. Online SETs would favorably compare to paper-and-pencil evaluations were it not for widely reported response rate decreases that cause SET validity concerns stemming from possible nonresponse bias. To combat low response rates, one institution introduced a SET application for mobile devices and piloted formal synchronous classroom time for SET completion. This paper uses the Leverage Salience Theory to estimate the impact of these SET process changes on overall response rates, open-ended question response rates, and open end response word counts. Synchronous class time best improves SET responses when faculty encourage completion on keyboarded devices and provide students SET completion time in the first 15 min of a class meeting. Full support from administrators requires sufficient wireless signal strength, IT infrastructure, and assuring student access to devices for responses clustering around meeting times.     

The influence of student and course characteristics on monotonic response patterns in student evaluation of teaching in South Korea

Using multilevel models, this study examined the effects of student- and course-level variables on monotonic response patterns in student evaluation of teaching (SET). In total, 11,203 ratings taken from 343 general education courses in a Korean four-year private university in 2011 were analyzed. The results indicated that 96 % of variance of monotonic response patterns could be explained by student characteristics, such as gender, academic year, major, grade point average, SET score, and perceptions about course difficulty, while controlling for course-level variables. Furthermore, 4 % of variance of monotonic response patterns was derived from course characteristics, including faculty age and class size, while controlling for student-level variables. The findings suggest that Korean higher education institutions need to take proper measures to encourage students to participate more actively and sincerely in SET for the best and proper use of the evaluation’s outcomes.     

The relations of a school’s capacity for institutional diversity to student achievement in socio-economically,ethnically, and linguistically diverse schools

This study examines how a school’s capacity for institutional diversity relates to student achievement in socio-economically, ethnically, and linguistically diverse schools. It also investigates whether various student groups benefit differently from a school’s level of student diversity and its institutional capacity for diversity. Using data from the High School Longitudinal Study of 2009 (HSLS:09), the study uses two-level hierarchical linear models to explore the relationships between multiple types of school diversity and student achievement and to examine the role played by an institution’s capacity for diversity. Our findings confirm that different types of school diversity have different associations with the academic achievement of students from varied family and language backgrounds. We also find that a school’s capacity for diversity is a significant predictor of the academic success of students, particularly for students from lower income families and those who use a home language other than English.     

Acquiescence,instructor’s gender bias and validity of student evaluation of teaching

Abstract

The validity of student evaluation of teaching (SET) scores depends on minimum effect of extraneous response processes or biases. A bias may increase or decrease scores and change the relationship with other variables. In contrast, SET literature defines bias as an irrelevant variable correlated with SET scores, and among many, a relevant biasing factor in literature is the instructor’s gender. The study examines the extent to which acquiescence, the tendency to endorse the highest response option across items and bias in the first sense affects students’ responses to a SET rating scale. The study also explores how acquiescence affects the difference in teaching quality (TQ) by instructor’s gender, a bias in the latter sense. SET data collected at a faculty of education in Ontario, Canada were analysed using the Rasch rating scale model. Findings provide empirical support for acquiescence affecting students’ responses. Latent regression analyses show how acquiescence reduces the difference in TQ by instructor’s gender. Findings encourage greater attention to the response process quality as a way to better defend the utility of SET and prevent potentially misleading conclusions from the analysis of SET data.     

Challenging Performative Fabrication: Seeking authenticity in academic development practice

While extensive discussion has been devoted to internationalising the curriculum in a response to the presence of international students, little is known about how academic staff members perceive the presence of this student cohort. Using data collected through semi‐structured interviews with 80 academic staff from four different faculties in one higher education institution, this study investigates staff understanding regarding international students, with particular attention given to how disciplinary differences shape their responses. Findings indicate that staff from ‘soft’ and ‘hard’ disciplines hold different instructional beliefs, which then affect their teaching and learning practice. Discussions are provided on how these differing conceptions of teaching fit into the context of academic development with respect to international education.     

Why bother? Student motivation to participate in student evaluations of teaching

When response rates on student evaluations of teaching (SETs) are low, inability to properly interpret and use responses from the students who do participate is a big problem. Where does the motivation to participate break down, and where and how does it make sense to invest efforts in rectifying that? In this study, we examined 641 university students’ reported behaviours and motivation related to their SET participation. In terms of behaviour, students who seldom or never participate in online SET tools reported a willingness to invest, at most, five minutes in the process, though the majority never even open the online evaluation links when they receive them. In terms of motivation, they significantly differed from students who always participate with distinctly lower levels of: (1) willingness to participate at all, (2) perception of autonomy and competence, (3) meaningfulness, (4) personal value, (5) engagement in others’ participation, and (6) understanding of the value of their own participation for others’ benefit. Based on these findings, we propose a strategy for increasing future response rates, particularly among recalcitrant students, in order to be able to gather sufficient and reliable results for the work of improving teaching.     

Lernen an zwei Orten: das Problem des ‘Dualen System’ in der lehrerausbildung

On the basis of a study of the literature and of empirical research, the article puts forward suggestions for the improvement of teacher training programmes within a dual system.

A consecutive approach to training implies that a student spends a part of the training period in an educational institution and a part in a practical situation (i.e. on teaching practice). However, the consecutive system presents considerable problems in respect both of organisation and of the psychology of learning. Organisationally, there is the question of sequence to be faced, if the elements of theory and practice are to be brought together.

On the basis of a study of the literature of other forms of professional training, the following suggestions are made:

1. students’ activities in the school must re‐inforce their learning and be integrated with their work in the training institution;

2. their educational studies must support the acquisition of the competences needed by the teacher at the outset of his/her professional career;

3. attention must be paid to the learning needs and the potential of each student.

The article describes how a teacher training programme may follow these guidelines, leading to a ‘concurrent’ set of cycles bringing together elements of theory, practice and reflection upon practice, and in which a student may alternate between the academic institution and school practice.

After giving an overview of possible models of training the author focuses upon a specific example and provides a model of the institution‐practice link. What is important is that the learning style of the student be taken into account. Each student selects from four alternative programmes, namely: instructional variants, reflection variants, selfstudy variants, and practice variants. The contents of these four programmes are explained, and the learning outcomes of students from these differentiated programmes compared with those of students from undifferentiated, ‘normal’ programs. In summary, it is claimed that in the research project described there are positive indications for a theory‐practice link which gives students a choice of approaches and a better integration of their learning into the practical classroom situation.     

Localizing College Retention Efforts: The Distance between Theoretical Orientation and Institution-Specific Needs

The study used a theoretically guided questionnaire to examine student experience in college and to gain a better understanding about how college environment affects student persistence. Data were collected from a single four-year institution; the findings suggest that institutional control over academic quality is the most critical factor in reducing students’ dropout intention along with their ability to pay for college education. The results highlight the inconsistency between the specific needs of students in their particular academic settings and the dominant theoretical frameworks that focus on academic and social engagement, and these results offer encouragement for localized retention interventions based on sufficient understanding of students’ experiences.     

高等学校学生评教行为偏差研究——基于南京仙林大学城若干大学的分析

学生评教是高校教学质量保障体系的重要组成部分.目前,学生在评教过程中存在评教行为偏差,主要表现为引致评教、恶意评教、随意评教和放弃评教.其直接原因在于高校在学生评教管理中的若干强制规定和院系对学生评教的不当引导;根本原因则是由学生评教定位偏差以及学生评教运作无关学生利益造成的.因此,高校需要科学定位学生评教,极力完善学生评教的制度设计,构建学生评教和学生利益紧密关联的基点,以便促成教师根据学生评教改进教学和服务学生、满足学生学习需要这一根本目的的实现.     

Using new vocabulary in writing: Exploring how word and learner characteristics relate to the likelihood that writers use newly taught vocabulary

Understanding academic vocabulary is essential to student success in school. Use of academic vocabulary words in writing is considered one of the strongest measures of how well a reader understands a given word. In theory, willingness to use academic vocabulary in writing indicates the complexity of acquiring representations of the word’s orthography, phonology, and semantics based on the word’s characteristics. In addition, a learner’s overall literacy skills should relate to whether they attempt to use words. In the present study, sixth graders (n = 66), seventh graders (n = 60), and eighth graders (n = 41) learned 25 academic vocabulary words in a supplementary academic vocabulary intervention and were then asked to use those words in short persuasive essay drafts. We measured whether these students attempted to use a word (a binary uses outcome) and the number of times a student used a word (a continuous attempts outcome) and used cross-classified random effects models to examine how (a) the orthographic, phonological, and semantic characteristics of words and (b) the students’ literacy-related characteristics related to their uses and attempts. For word characteristics, students were more likely to use and attempt high frequency than low frequency words. For student characteristics, students proficient on the state exam were more likely to use and attempt the words, and students learning English were less likely to attempt the words. Implications for vocabulary intervention and writing instruction are discussed.     

Student evaluations of teaching: the impact of faculty procedures on response rates

In the last 10–15 years, many institutions of higher education have switched from paper-and-pencil methods to online methods of administering student evaluations of teaching (SETs). One consequence has been a significant reduction in the response rates to such instruments. The current study was conducted to identify whether offering in-class time to students to complete online SETs would increase response rates. A quasi-experiment (nonequivalent group design) was conducted in which one group of tenured faculty instructed students to bring electronic devices with internet capabilities on a specified day and offered in-class time to students to complete online SETs. A communication protocol for faculty members’ use was developed and implemented. A comparison group of tenured faculty who did not offer in-class time for SET completion was identified and the difference-in-differences method was used to compare the previous year’s response rates for the same instructor teaching the same course across the two groups. Response rates were substantially higher when faculty provided in-class time to students to complete SETs. These results indicate that high response rates can be obtained for online SETs submitted by students in face-to-face classes if faculty communicate the importance of SETs in both their words and actions.     

What is a Doctorate? Changing Meanings and Practices in Communication Sciences in Switzerland

In this article, we analyse the organisation of the doctorate in communication sciences in the context of the overall discussion on the changing organisation of doctoral studies in Switzerland. We focus on three tensions which appear central for the field, namely the employment status of doctoral students, the importance of academic vs. professional training and, finally, the organisation of doctoral studies and the possibilities and difficulties in the introduction of a graduate school model. Our results show that in this field the doctorate has to be considered more as an orientation period, where professional and academic training coexist and where there is an extremely high diversity of objectives, activities and organisation forms, both between universities and individual students. This model is surprisingly well adapted to the situation of a field characterised by high internal diversity, rather low research intensity and strong orientation to application. Reforms like the introduction of graduate schools or the reinforcement of academic training have thus to be implemented with some care.     

You can’t ignore us: what role does family play in student engagement and alienation in a Ghanaian university?

Much of the existing literature on student engagement focuses on what happens within the higher education environment or what the institution has direct control over, restricting understanding of how issues outside of the institution affect engagement. This paper argues that efforts to improve student engagement should be broadened to incorporate issues relating to the family. It employed a case study design in order to develop a broad and more in-depth understanding of how family influences the amount of time and effort students invest in their learning in a large public university in Ghana. It was established that engagement is influenced by family expectation, financial and social support, as well as monitoring of students’ academic performance. The unique contribution of the study is that it broadens current understandings of engagement because it adds an aspect that has not been highlighted in current work on engagement, making a case for institutions to work more with families to get a broader understanding of challenges students face in order to provide comprehensive support.     

‘What drives students’ affective commitment towards their university?’

How can universities build ongoing, committed relationships with students, able to withstand the financial and emotional challenges of studying in higher education? The research proposes that students’ ongoing attachment to their university, based on positive feelings towards the university, is an important aspect of the student experience. This ongoing attachment is conceptualised here as students’ affective commitment towards their institution. Using an online survey-method and a research sample comprising undergraduate students studying in the UK, this research identifies three factors which drive students’ affective commitment towards their institution. These factors include students’ affective commitment towards academics and students’ calculative commitment towards the institution; factors which draw from the relational literature. A third factor, commitment balance, was developed within this research. Commitment balance occurs when a student’s commitment to their university is perceived to be reciprocated by the university’s commitment to the student. The study found that commitment balance was the most important driver of students’ ongoing attachment to their institution. The paper proposes that commitment balance is a key idea to consider within relational studies generally, but has a particular relevance in the higher education context for understanding the student experience. Commitment balance reflects the pulse of reciprocity which energises relational exchanges between students and institution. The findings of this research reinforce how critically important it is for universities and academics to build relationships with students. The desired outcome is to enhance the student experience, create positive attachment between students and university and ultimately improve student retention.     

Class acts? Working class student officers in students’ unions

Abstract

This article explores the recent emergence of ‘working-class student officer’ roles in students’ unions associated with elite UK universities. These student representative roles are designed to represent the interests of working-class students within their universities and sit alongside student representatives for liberation groups and/or student communities. Based on interviews with postholders and using Bourdieu’s concepts of habitus and field and Reay’s applications of a ‘reflexive habitus’, I explore how these students have come to assert a public and political ‘working-class student’ identity within their universities. Their commentaries reveal the ‘makings of class’ in a context where students are very aware of claims for recognition and the ‘hidden injuries of class’ and offer an insight into how working-class students are finding new ways to navigate their classed identities in HE.     

Professional staff contributions to student retention and success in higher education

ABSTRACT

Student attrition remains a persistent problem within the Australian higher education sector. Contributing factors include financial, reputational and quality issues, which can pose significant risks for a university’s sustainability. Institutional culture is fundamental to decisions student make about withdrawing or remaining in higher education. Therefore, student retention requires a sustained, deeply embedded commitment from all parts of the institution, placing student experience at the forefront of all activities in the student lifecycle. Applying a lifecycle approach to the way in which institutions think about student retention benefits students by providing comprehensive and consistent support. Given that less is known about how professional staff contribute to student academic learning journeys as part of whole of institution responses to student retention, this paper focusses on the contributions that professional staff make within the student lifecycle and how they can most readily map their contributions to student retention and success.     

中学生班级集体效能感、学业自我效能感与学业成绩的关系研究

通过对1773名中学生施测班级集体效能感量表(CCEI)和学业自我效能感量表,探讨中学生班级集体效能感、学业自我效能感与学业成绩之间的关系。经统计结果表明中学生在班级集体效能感水平上中等偏上,在学校类型、年级、是否班干部、是否独生子女、生源以及用功程度这几个变量上存在显著差异;回归分析表明,部分人口学变量、学业自我效能感变量对于班级集体效能感具有显著的预测作用;进一步结构模型分析发现,班级集体效能感对学生的学业成绩具有正向预测作用,学生自我效能感对学生的学业成绩也具有正向预测作用,且对学生班级集体效能感也具有正向预测作用。     

What response rates are needed to make reliable inferences from student evaluations of teaching?

This paper addresses the determination of statistically desirable response rates in students’ surveys, with emphasis on assessing the effect of underlying variability in the student evaluation of teaching (SET). We discuss factors affecting the determination of adequate response rates and highlight challenges caused by non-response and lack of randomization. Estimates of underlying variability were obtained for a period of 4?years, from online evaluations at the University of British Columbia (UBC). Simulations were used to examine the effect of underlying variability on desirable response rates. The UBC response rates were compared to those reported in the literature. Results indicate that small differences in underlying variability may not impact desired rates. We present acceptable response rates for a range of variability scenarios, class sizes, confidence level, and margin of error. The stability of estimates observed at UBC, over a 4-year period, indicates that valid model-based inferences of SET could be made.     

ACADEMIC AND SOCIAL INTEGRATION OF COMMUNITY COLLEGE STUDENTS: A CASE STUDY

The literature suggests that most studies on student retention have focused on student satisfaction at 4-year institutions where students tend to be more traditional in nature. The study discussed in this article investigated how Tinto's (Tinto, Russo, & Kadel, 1994) model of retention could be applied to 2-year institutions. It explored academic and social integration and their effects on student withdrawal rates as well as the effect of background skills on withdrawal rates. Study participants were 462 second-semester degree-seeking community college students who completed a survey regarding their satisfaction with the academic and social climate of the community college. Performance on Computer Placement Tests (CPTs) were correlated with withdrawal rates to determine the association between background skill levels and withdrawal patterns. No correlation was found between academic and social integration and withdrawal rates. However, findings did show that the poorer the CPT performance, the more likely students were to withdraw from courses.     

Exploring Resilience in Latina/o Academic Outcomes: A Latent Class Approach

The purpose of this study is to explore the relationship between various protective factors with academic outcomes of Latina/o high school students. We use two groups of protective influences, individual and family, and their relationship to 12th grade mathematics achievement, dropout rates, and enrollment in post-secondary education. Latent class analysis was used to identify academic protective profiles, or latent groups/classes, among high school Latina/o students (N?=?1610) and assess group differences with respect to gender, SES, immigrant status, student’s native language, preschool attendance, and 10th grade mathematics. Results indicated the presence of four academic protective groups, which differed with respect to academic discussions with parents, and attitudes about mathematics. The four classes are compared with respect to academic outcomes and differences are discussed as well as implications for practice.