An application of importance-performance analysis to students’ evaluation of teaching

Educational Assessment, Evaluation and Accountability - Students’ feedback is usually gathered in institutions of higher education to evaluate the teaching quality from the students’...     

Evaluating research-oriented teaching: a new instrument to assess university students’ research competences

Several concepts have been developed to implement research-oriented teaching in higher education in the last 15 years. The definition of research competences, however, has received minor attention so far. Some approaches to modeling research competences describe these competences along the research process but either focus on a specific academic discipline and/or specific facets or offer no empirical verification for cross-disciplinary approaches. We have developed a new approach to model research competences across various academic disciplines in form of a competence model—the RMRC-K-model. This model comprises five dimensions: skills in reviewing the state of research, methodological skills, skills in reflecting on research findings, communication skills, and content knowledge. In this study, we present an instrument (R-Comp) developed on the basis of this cross-disciplinary competence model. The factorial structure of the R-Comp was examined with data from 391 university students in three groups, either enrolled in a Bachelor’s degree, Master’s degree, or a PhD program. The sample represented various academic disciplines. Results from confirmatory factor analyses supported the hypothesized structure of the R-Comp for the five dimensions in accordance with the RMRC-K-model. Additionally, results provided evidence for a more detailed differentiation of all dimensions with sub-dimensions reflecting theorized facets of the model. Both Cronbach’s α and construct reliability H indicated overall good reliability. Despite limitations to this study, there is some first evidence of a valid and reliable assessment of student research competences.     

Correction to: An application of importance-performance analysis to students’ evaluation of teaching

Educational Assessment, Evaluation and Accountability -     

A dynamic multilevel modeling approach to university students’ anxiety and hope regulation

Preparing for an exam entails various exam-related emotions. To ensure psychologically healthy behavior and an effective learning process, emotion regulation is highly important. In the present study, we used a dynamic multilevel modeling approach to analyse the interplay of exam-related anxiety and hope, as well as four regulation strategies over time. This study provides valuable insights into students’ emotion regulation in academic settings. Specifically, we measured students’ emotions and regulation strategies (cognitive approach, cognitive avoidance, behavioral approach, behavioral avoidance) across two weeks of measurement. In total, 217 university students answered several questionnaires via an electronic device during two different measurement periods lasting 7 days each. These measurement periods were 5 weeks (day 35–29) and one week (day 7 to 1) prior to an important exam. During their everyday life, they answered these questionnaires six times a day. We applied dynamic multilevel modeling to analyse the data. Results revealed cross-lagged relationships of exam-related emotions on regulation strategies, with the strongest cross-lagged relationship being found for hope and the cognitive approach regulation strategy. Moreover, the temporal distance to the exam affected the strength and direction of these cross-lagged relationships. This study suggests the importance of considering the mechanisms of emotion regulation on the situation level, and that strategies should be tailored to the individual situation rather than employing universal strategies. Further, the extent to which we can interpret the lagged relationships as suggestive of causal mechanisms are discussed.     

A face in a sea of faces: exploring university students’ reasons for non-attendance to teaching sessions

In the past 20 years, the landscape of Higher Education in the United Kingdom has changed considerably. With the prospect of tuition fees spiralling above £9,000 per year, per student, it is important to explore the impact this has upon the student experience. In spite of the vast financial investment current students make towards their education, student attendance and engagement remain low. The present study therefore adopted a qualitative approach using focus groups to explore reasons for non-attendance at university teaching sessions. Four key themes emerged; these surrounded issues about a sense of belonging to university, views of the teaching (material taught and the personality and method of the tutor), perceptions of being a consumer and external pressures. This research suggests that universities should pay particular attention to the reasons why students do not attend and possibly change practice, providing more support where appropriate.     

How reliable are students’ evaluations of teaching quality? A variance components approach

The inter-rater reliability of university students’ evaluations of teaching quality was examined with cross-classified multilevel models. Students (N = 480) evaluated lectures and seminars over three years with a standardised evaluation questionnaire, yielding 4224 data points. The total variance of these student evaluations was separated into the variance components of courses, teachers, students and the student/teacher interaction. The substantial variance components of teachers and courses suggest reliability. However, a similar proportion of variance was due to students, and the interaction of students and teachers was the strongest source of variance. Students’ individual perceptions of teaching and the fit of these perceptions with the particular teacher greatly influence their evaluations. This casts some doubt on the validity of student evaluations as indicators of teaching quality and suggests that aggregated evaluation scores should be used with caution.     

Determinants of university students’ attendance

ABSTRACT

Background: Attendance of university students at their timetabled teaching sessions is usually associated with higher levels of educational attainment. Attendance is usually considered to reflect students’ level of engagement with their course and to be critical to student success; despite the potential for technological alternatives, lectures and other face-to-face sessions still tend to be the primary method of teaching at university.

Purpose: Here we review studies which have investigated these determinants of attendance in order to gain a better understanding of whether – and how – Higher Education Institutions are able to improve attendance rates.

Sources of evidence: Electronic databases (e.g. ERIC, Web of Science) were used to identify articles exploring attendance in Higher Education settings.

Main argument: Some of the most debated determinants of attendance are reviewed: teaching issues (e.g. quality, style and format); effects of university expectations and policy (e.g. mandating attendance, awarding grades for attendance); scheduling issues; provision of materials online; and the effects of individual factors arguably outside of the Higher Education Institution’s control (e.g. finance, student employment, student demographics and psychological factors).

Conclusions: It is suggested that, although some individual factors influence student attendance and are arguably out of the control of HEIs, it is possible for them to facilitate attendance through adjustments to aspects of degree delivery such as attendance policies and monitoring, timetabling and style of teaching. Implications for policies on the recording of lectures, curriculum design and student term-time working are also discussed. Future research on student attendance should include longer and larger studies which simultaneously consider a range of influences; examining both inter- and intra-individual variability and different types of teaching sessions.     

Graduate students’ conceptions of university teaching and learning: formation for change

The paper introduces a framework concerning conceptions of teaching and learning of advanced graduate students and the trajectory of change in their conceptions following a formal course on course design and teaching. Students were enrolled in six sections of this graduate course, two offered in a 6-week format and 4 in a 13-week format. The courses were taught by three different instructors. Data were obtained from 88 doctoral students before and after the course using four questions pertaining to the meaning of effective teaching, effective learning, role of the teacher, and role of the learner. Using an open coding procedure, four conceptions were identified. These were transmitting knowledge, preparing context/managing instruction, promoting course learning, and promoting life-long learning. The generalized linear model procedure was used to conduct within and between group pre and post course comparisons. Overall, there was a significant change in response frequency ratios in all four categories across all courses, indicating a decrease in the responses in the first two categories and an increase in the frequency of responses in the last two categories. No significant differences were attributed to course type or instructor.     

Does evaluation of teaching lead to improvement of teaching?

Abstract

Given the widespread use of student evaluation of teaching in North American colleges and universities, it is reasonable to ask whether student evaluation has accomplished one of its major intended outcomes, namely improvement of instructional quality. A review of research evidence from three independent sources (faculty surveys, field experiments and longitudinal comparisons) suggests that student evaluation does in fact contribute significantly to improvement of certain aspects of university teaching, particularly if evaluation is supplemented by expert consultation. Furthermore, there is no clear evidence that student evaluation has led to undesirable instructional side‐effects, such as grade inflation and entrenchment of traditional teaching methods.     

A boundary-crossing approach to support students’ integration of statistical and work-related knowledge

Vocational students and beginning professionals typically find it hard to integrate the mathematics and statistics that they have learned at school with work-related knowledge. To explore how such an integration process could be supported, we conducted an intervention in secondary vocational laboratory education. Our boundary-crossing approach was informed by the literature on boundary crossing and accompanying learning mechanisms (e.g., reflection in the form of perspective making and taking, and transformation in the form of hybridization). We hypothesized that reflection, as making and taking perspectives on school-taught and work-related knowledge, could lead to transformation, i.e., help students integrate these types of knowledge into a hybridized whole. Data collection included video and audio recordings of five 1-h meetings with three students, the data from their research projects, and interviews with the teacher and two workplace supervisors. The analysis of the students’ reasoning during the meetings revealed that their level of integrating school-taught statistics and work-related knowledge increased significantly and with a medium effect size. This suggests that a boundary-crossing approach can support students in integrating school-taught and work-related knowledge.     

Ready for university? A cross-national study of students’ perceived preparedness for university

Students’ preparedness for higher education is seen as one of the main factors affecting first-year attrition or study success. In this paper we report on a cross-national study in which students’ preparedness for university was measured before students commenced their study at a university in New Zealand or in the Netherlands. This cross-national project provided a unique opportunity to compare students’ perceptions of readiness for university where students are prepared for higher education in quite different secondary school systems. Departing from a transition framework, and comparing the results in both countries using logistic regression techniques to investigate which aspects of readiness could predict perceived preparedness, we discovered similarities in as well as differences between students’ perceived readiness for university study. It could be argued that differences are caused by the different educational systems at secondary level. However, overall we can conclude that, in spite of differences between the educational systems in the two countries, many differences were not remarkable or very significant. This has clear implications for how we view the relative importance of secondary school preparation and tertiary induction. We can expect greater benefit from implementing first-year pedagogical practices in universities that would assist students to develop their academic skills, than from demanding that high schools prepare students better.     

An exploratory study of changes to university EFL teachers’ teaching practices

The brief report describes an exploratory investigation into changes to three university EFL teachers’ teaching practices in a research-intensive university framed by managerialist technologies to improve its visibility and global ranking. The findings revealed that substantial changes have been brought about to their teaching practices in terms of falling into the routine of ‘technification of teaching’, producing a spectacle ‘what is there simply to be seen and judged’ or striving to seek out new spaces for ‘authentic’ English teaching. The study concluded with implications for facilitating the managerial reforms that are congruent with the cognitive and contextual realities of different generations of Chinese university EFL teachers.     

IEDC method： A new approach to promote students＇ communicative competence

IEDC is an acronym from interaction of English dormitory and class teaching. It is a new, student-centered teaching approach intended for non-English majors in China. This articles states that communicative competence can be developed through various means of cooperative learning.     

Instruments to assess students’ critical thinking—A qualitative approach

Critical thinking (CT) skills are essential to academic and professional success. Instruments to assess CT often rely on multiple-choice formats with inherent problems. This research presents two instruments for assessing CT, an essay and open-ended group-discussion format, which were implemented in an undergraduate business course at a large public university. The topics were intentionally chosen to be digitally-related (internet access and social media) since they raise complex issues, are applicable in many disciplines, and students are avid users, thereby making these instruments relevant, multidisciplinary, engaging, and multifaceted. These instruments may be adapted to enable educators to effectively assess undergraduate students’ ability to think critically. Qualitative analysis of the verbatim data with NVivo helps demonstrate the instruments’ efficacy in assessing CT, with (1) high-quality responses receiving high grades, and (2) overarching themes identified by content analysis, that coalesce into the three dimensions of social justice theory (distributive, procedural, and interactional), thus illustrating students’ consideration of complex fairness norms and societal needs. Excerpts of student responses are provided as illustrations of the indicators/evidence of critical thinking. Educators can use these instruments to first assess students’ CT skills, and then devise targeted interventions to improve the quality of students’ thought processes.     

The ‘Historical-investigative’ approach to teaching science

The paper describes the author's experience in using the history of science in teaching physics to science teachers. It was found that history becomes more useful to teachers when explicitly combined with investigative experimentation, which, in turn, can benefit from various uses of the history of science.     

The number of feedbacks needed for reliable evaluation. A multilevel analysis of the reliability,stability and generalisability of students’ evaluation of teaching

A multilevel analysis approach was used to analyse students’ evaluation of teaching (SET). The low value of inter-rater reliability stresses that any solid conclusions on teaching cannot be made on the basis of single feedbacks. To assess a teacher’s general teaching effectiveness, one needs to evaluate four randomly chosen course implementations. Two implementations are needed when one course is evaluated, and if one implementation is evaluated, up to 15 feedbacks are needed. The stability of students’ ratings is very high, which reflects students’ stable rating criteria. There is an obvious rating paradox: from the student’s point of view, each rating is very precise, stable and justifiable, but from the teacher’s point of view a single feedback reflects the quality of teaching to just a moderate extent. Cross-hierarchical analysis reveals that there are large discrepancies between the uses of rating scales; some students are systematically more lenient in their rating whereas others are systematically more severe. The study also reveals that some courses are generally rated more favourably and that some courses are more suitable for certain teachers. Managers can thus improve the quality of teaching by finding the most suitable courses for each teacher.     

Kenyan students’ perceptions of new technologies to improve access to education

The aim of this paper is to present the findings of a qualitative study carried out on perceptions of high school students on the use of ICT in learning in a secondary school in Mombasa, Kenya. The study involved 18 students in the 11th grade (form three). The students took part in three focus group discussions and six one-on-one interviews over a period of 2 weeks. Several themes on the learners’ understanding of and reaction to ICT were identified. It was revealed that learners had a sound understanding of the various types of ICT and could even define them while explaining the uses of these ICT tools. Findings showed that the learners were generally motivated about using ICT and that ICT encouraged student-centered learning leading to better understanding and performance in their studies. All in all, the findings from this study indicate that learners are aware of the importance of ICT in their learning and feel that at the moment the use of ICT in school is no longer a luxury but a necessity that has taken too long to materialize.     

Exploring foreign undergraduate students’ experiences of university

Although international students are an important source of income to universities in the UK, the emotional impact of their experiences may be ignored and unacknowledged. This study explored the personal experiences of international students studying for an undergraduate degree in the UK. Semi-structured interviews with five participants were analysed using interpretative phenomenological analysis, which aimed to capture the personal perceptions of these participants. Two analytical themes are presented: “adaptation and identity”, which reflects the difficulties in acculturation, adaptation and development of identity in a foreign country; and “us and them”, which pays close attention to the perception of disconnectedness, unfairness and exclusion encountered in trying to integrate into a different way of life. Findings are discussed in relation to the impact that participants’ experiences have had on their self-worth, confidence and ability to engage with domestic individuals.     

Exploring students’ expectations of learning analytics: A person-centered approach

Education and Information Technologies - In order to successfully implement learning analytics (LA), we need a better understanding of student expectations of such services. Yet, there is still a...