Moving beyond face value: re-envisioning higher education evaluation as a generator of professional knowledge

Conventional student-led evaluation is now an orthodox feature of the North American, UK and Australian higher education landscape. Increasingly, it is guiding major institutional decisions around educational quality, academic promotion and more recently institutional funding by government. Yet significant research around student-led evaluation has demonstrated that this form of student opinion-based evaluation remains highly fragile and susceptible to multiple forms of influence. This paper argues that student-led evaluation of itself is not sufficiently robust to appropriately inform notions of educational quality or pedagogical capability of academics. An alternative approach to evaluation in higher education is proposed, broadly drawing on the conception of fourth generation evaluation. The model foregrounds academic collaboration in shaping a professional evaluative framework at an activity level (i.e. programme or sub-discipline level) based on an ongoing dialogue between peers and with students, as well as qualitative evaluation of student learning. The model attempts to make tensions and conflicts in the teaching and learning context more explicit and the active focus of evaluative efforts to devise appropriate and sustainable responses to improve the quality of learning and professional practice. This paper presents some preliminary research on the effectiveness of this model in its initial piloting in postgraduate programmes in an Australian university.     

The significance of a process evaluation in interpreting the validity of an RCT evaluation of a complex teaching intervention: the case of Integrated Group Reading (IGR) as a targeted intervention for delayed Year 2 and 3 pupils

The significance of a process evaluation for understanding randomised controlled trials (RCTs) of complex teaching interventions in a classroom setting is discussed in relation to the evaluation of the Integrated Group Reading (IGR) programme trial. This process evaluation is based on realist principles and draws on different data sources. The findings indicate that the efficacy of a classroom-based complex intervention like IGR is hard to evaluate given the nature of the intervention and its context. These findings illustrate that a process evaluation should not be seen as a subsidiary study to the experimental trial, but as an independent study in its own right as it can shed light onto the setting up and running of the trial. The paper argues that RCTs by themselves do not necessarily provide the ‘best evidence’ and need to be supplemented by other evaluation perspectives.     

Improving indigenous and ethnic minority student success in foundation health study

This study investigates what teaching practices in the ‘non-lecture context of a foundation programme' help or hinder Māori and Pasifika students' success in a New Zealand university. This two-year qualitative project used Kaupapa Māori and Pasifika Research (KM/PR) methodologies conducted in three phases: (1) needs analysis, (2) intervention and (3) evaluation. Twenty-eight Māori or Pasifika students were interviewed using the Critical Incident Technique identifying 798 incidents grouped into four themes for institutional development: (I) use effective practices for teaching and learning, (II) grow independent learners, (III) support the empowerment of the learner and (IV) harness the positive cohort effect. Initially, students reported that intensive support provided by the foundation programme was not preparing students well for success in degree-level study. Following interventions of institutionally-led changes, students reported better preparation for ongoing study. The overall learning environment and provision of Māori and Pasifika academic and pastoral support were important factors for success.     

Little changes,big results: the impact of simple changes to early years learning environments

The Strengthening Foundations of Learning (SFL) project is an early years CPD intervention and part of the wider Preparing for Life (PFL) Initiative in an area of urban disadvantage in Dublin, Ireland. Working within the context of the Aistear curriculum framework, and building on research evidence on the importance of practitioners to quality Early Childhood Education (ECE), the primary aim of the project is to improve the quality of early years practice. The two-year project provides a complex training and mentoring programme to educators in order to deliver ‘the best outcomes for children’. This paper presents findings from the evaluation of the ‘learning environment’ strand. The findings indicate that the embedded nature of CPD used in the SFL project has resulted in positive changes in early years practice and pedagogical language. Mid-term results suggest that participants are providing improved early learning environments and responding to learning opportunities in a more focused and informed way.     

Evaluation of a Study Skills Intervention Programme: A Mixed Methods Study

This article reports on a mixed methods study that evaluated the efficacy of a study counselling intervention programme at a South African university. Quantitative data, collected by means of the Learning and Study Strategy Inventory, indicated that the study skills intervention programme brought about statistically significant improvements in participants' study skills profiles. The qualitative findings that were collected from 10 participants served as a contextual lens to develop a more holistic understanding of the quantitative results. It is argued that multidisciplinary collaboration is required to address the heteregenous needs of students within the higher education context.     

Postgraduate study and managers’ subsequent work experience: an exploratory evaluation

Drawing on an exploratory qualitative study, this article considers the link between business school teaching at graduate level and subsequent work behaviour and experiences of former students. It evaluates the student experience some time after graduation. The findings of the retrospective evaluation point to the value of classroom peer discussion, the testing of ideas against prior work experience and the opportunity to make sense of organisational issues by setting them into broader context. The importance of andragogical approaches to teaching is discussed as well as the implications of the study findings for teaching quality enhancement.     

Evaluating postgraduate preparation in the South African context

Little work is being undertaken in South Africa to systematically and intentionally prepare undergraduate students to pursue postgraduate studies. This is concerning given the shortage of postgraduate students and the small scale of postgraduate studies. The few programmes and endeavours that exist to prepare students for postgraduate studies are not necessarily evaluated to assess their achievements and shortcomings. This paper provides an evaluation of an academic year-long postgraduate preparation programme, and is specifically concerned with examining subsequent postgraduate enrolment and improvement of participants’ marks. The study draws on both quantitative and qualitative data. The findings indicate that the majority of programme participants proceeded to subsequently enrol in postgraduate studies immediately after completing the programme and that the programme played a role in the pursuit of postgraduate studies. However, the findings also indicate that overall participants’ marks did not improve after participating in the programme. The study brings to light that, while some achievements are possible, the limits of the programme must also be acknowledged.     

Action learning – a process which supports organisational change initiatives

This paper reflects on how action learning sets (ALSs) were used to support organisational change initiatives. It sets the scene with contextualising the inclusion of change projects in a masters programme. Action learning is understood to be a dynamic process where a team meets regularly to help individual members address issues through a highly structured, facilitated team process of reflection and action. The key findings from the evaluation of the students and facilitators’ experiences are reflected on, together with plans for improving the experience for all stakeholders for the next academic year. In sharing this experience, the purpose of the paper is to highlight the most significant learning from the evaluation. Good preparation for action learning is vital to ensure a positive experience for all involved. From the student perspective, an appropriate learning set mix is needed to ensure a balance of support and challenge for the ALS. In addition to a preparatory workshop for action learning facilitators, regular meetings with the action learning facilitators in the form of ALSs could be scheduled as a support especially for those new to the process. Finally, it is hoped that this account will encourage readers to use action learning for supporting and engaging students in organisational change initiatives.     

Teacher stance in creative learning: A study of progression

This paper reports focused findings from a qualitative scoping study undertaken in four sites in England focusing on progression in creative learning (PICL) (February 2005–January 2006), funded by Creative Partnerships, a national development programme. The study sought to explore how progression in creative learning can be described in two curriculum areas. The paper introduces the analytic framework which emerged from the study, and summarises some of the key findings, with a particular focus on teacher stance, taking this slice of analysis further than is done in the original research report and discussing in particular teachers’ attitudes toward creative learning and toward teaching for creative learning.     

An evaluation of a group CBT programme for children with a diagnosis of autism spectrum disorder in a school age disability service

Research has shown that rates of clinical anxiety levels are higher among young people with an ASD than their typically developing peers. This study conducted an exploratory investigation into the impact of a specifically tailored CBT group programme on anxiety levels in young people with ASD. The 12 week CBT programme was delivered to 12 children with a diagnosis of ASD in a children’s disability team in Ireland. An embedded mixed methods design was utilised and inferences were drawn by collecting and analysing quantitative and qualitative data separately. Findings indicate CBT can be delivered successfully to a group of children with ASD. As is the nature of an embedded design, quantitative findings are reported as supportive evidence for the qualitative findings of this study. The positive preliminary findings suggest a rigorous, large scale evaluation is needed to determine the overall effectiveness of the programme with this population.     

Evaluating initial teacher education programmes: Perspectives from the Republic of Ireland

Research studies in teacher education have focussed on the outcomes of preparatory programmes. Less attention has been paid to the processes through which professional learning is acquired. This article argues that the study of attitudes and persuasion is very important in teacher education. The elaboration likelihood model (ELM) of persuasion provides an integrative framework to analyze empirical evidence from a five-year study, conducted in the Republic of Ireland, which presents findings from an in-depth survey of 2348 respondents. This model highlights many salient issues that teacher educators should consider when engaged in programme development, delivery and evaluation.     

The general and specific effects of educational programmes aimed at promoting awareness of and respect for diversity among young children

This article reports the findings of a mixed‐method evaluation of a pilot educational programme undertaken with six to seven year olds in a sample of primary schools in England with the aim of increasing their awareness of and respect for diversity through theatre, workshops and related teacher‐led classroom activities. The qualitative feedback from the teachers involved was extremely positive and encouraging and an analysis of the actual impact of the pilot programme on the children’s attitudes and awareness, using an experimental design, demonstrated some positive effects. In particular, the programme was found to increase the children’s general awareness of diversity and their ability to recognize instances of exclusion. While not a planned objective of the pilot programme, the evaluation also examined whether it had any effects on the children’s attitudes to specific differences, in this particular case racial differences. Interestingly, however, no evidence was found of any change in the children’s racial attitudes. With this in mind the article suggests that there is a need to distinguish between the general and specific effects of such educational programmes. The article considers the implications of this for future work in the area and also stresses the need to begin undertaking more thorough and rigorous evaluations of such initiatives.     

A proposed methodology for contextualised evaluation in higher education

This paper aims to inspire stakeholders working with quality of higher education (such as members of study boards, study programme directors, curriculum developers and teachers) to critically consider their evaluation methods in relation to a focus on student learning. We argue that many of the existing methods of evaluation in higher education are underpinned by a conception of learning that is de‐contextualised. As a consequence, many data collection methods do not address aspects that affect students’ learning. This is problematic because the core aim of higher education is to facilitate student learning. We propose a contextualised evaluation methodology, guided by 10 key questions, which can help evaluators address concepts and questions of student learning in their evaluations.     

Course evaluation matters: improving students’ learning experiences with a peer-assisted teaching programme

In the rapidly changing global higher education sector, greater attention is being paid to the quality of university teaching. However, academics have traditionally not received formal teacher training. The peer-assisted teaching programme reported on in this paper provides a structured yet flexible approach for peers to assist each other in reinvigorating and refining their teaching practice. Academics participated in this national, multi-institutional trial for varied reasons: the majority voluntarily, others to increase low student evaluation of course scores and some as part of a graduate certificate teaching qualification. Here we report on how academics used the scheme, and the teaching areas they focused on. Student evaluation of course scores increased in the majority of courses, suggesting the changes made had a positive effect on students’ learning experiences. The experiences of the multi-institutional trial reported here may benefit others considering such a scheme to reinvigorate and refine teaching practice and improve course evaluation scores.     

怎样的评价可以促进教师潜心育人:来自中小学教师专业表现的实证证据

教育评价事关教育发展方向,影响教师潜心育人的专业表现及学生全面发展。为考察问责式和发展性两种教师评价对教师专业表现的不同影响,本研究采用问卷法,调查了我国中西部2354名县镇及乡村中小学教师,构建潜变量结构方程模型,探究教师所感知的教师评价方式、对评价的态度、教师的专业学习及专业表现之间的关系。结果发现:问责式教师评价对教师专业表现影响的直接效应不显著,但通过教师对评价反馈的态度和专业学习,显著负向影响教师专业表现;发展性教师评价对教师专业表现的总效应、直接效应和间接效应皆为显著正向影响。这提示不同教师评价对教师潜心育人专业表现的影响机制是不同的,在制定和改革教师评价方案时需要综合考虑。     

Effectiveness of the Clinical Supervision Strategy as a Tool for Improving Teaching Quality: Perceptions of South African School-Based Managers and Educators

Research was undertaken to find ways of resolving implementation challenges of supervision and evaluation systems by examining the perceptions of South African school-based managers and educators regarding the effectiveness of the clinical supervision strategy as a tool for improving the quality of teaching in compliance with the total quality management (TQM) principle of empowerment as related to the integrated quality management system (IQMS). The article reports on part of the findings of a formal study designed to investigate the effectiveness of the developmental supervision model as a tool for improving the quality of teaching in Gauteng, South Africa The findings from using a mixed method approach, comprising a quantitative phase in the form of questionnaires and a qualitative phase in the form of three focus group interviews, indicated that the clinical supervision strategy complied with the TQM principle of empowerment, which was perceived to have a positive influence on the quality of teaching. In contrast, the IQMS evaluation process was perceived to lack compliance with empowerment due to characteristics related to the traditional inspection approaches to professional development.     

Theorising teaching and learning: pre-service teachers’ theoretical awareness of learning

We examine pre-service teachers’ theoretical learning during one five-week training module, and their educators’ learning about better lecture design to foster student learning. The study is iterative: interventions (one per group) were implemented sequentially in student groups A–C, the results of the previous intervention serving as the baseline for the design developed for the next. These learning study participants, 79 students from year three of a teacher training programme, studied the variation theory (VT) of learning. Three lesson cycles were completed, each comprising four steps: (1) a pre-test, (2) a 15-min intervention discussing VT, (3) a post-test and (4) a delayed post-test conducted eight weeks later. The results indicated learning differences between groups; qualitative analysis identified three categories of student answers, i.e. emergent, premature and unaware, regarding their theoretical understanding. Group C had more students with emergent knowledge (36%) than did groups A (20%) or B (17%) at post-testing.     

Primary special school teachers' knowledge and beliefs about supporting learning in numeracy

This paper presents findings from a qualitative study of a group of 12 teachers in primary special schools in Scotland for children with moderate learning difficulties. It sets out an analysis of classroom observations and interviews that explored teachers' knowledge and beliefs about teaching and learning in mathematics with children with moderate learning difficulties. The teachers were interviewed pre‐ and post‐intervention; this was a research‐based professional development programme in children's mathematical thinking (Cognitively Guided Instruction) which teachers then developed in their classrooms. The findings showed that prior to the professional development, the teachers had a limited knowledge of children's mathematical development with teaching frequently informed by intuitive beliefs and dated and sometimes discredited practices. Most teachers had low expectations of children with learning difficulties. Post‐intervention, the teachers reviewed this stance and affirmed that a deeper understanding of children's mathematical thinking provided a more secure knowledge base for instruction. They also recognised the extent to which learners were constrained by existing classroom practices. The paper argues for the commonality of this knowledge base and considers the problematic nature of viewing such knowledge as sector specific.