The effectiveness of learning analytics for identifying at-risk students in higher education

Abstract

In this article we investigate the effectiveness of learning analytics for identifying at-risk students in higher education institutions using data output from an in-situ learning analytics platform. Amongst other things, the platform generates ‘no-engagement’ alerts if students have not engaged with any of the data sources measured for 14 consecutive days. We tested the relationship between these alerts and student outcomes for two cohorts of first-year undergraduate students. We also compared the efficiency of using these alerts to identify students at risk of poorer outcomes with the efficiency of using demographic data, using widening participation status as a case study example. The no-engagement alerts were found to be more efficient at spotting students not progressing and not attaining than demographic data. In order to investigate the efficacy of learning analytics for addressing differential student outcomes for disadvantaged groups, the team also analysed the likelihood of students with widening participation status generating alerts compared with their non-widening participation counterparts. The odds of students with widening participation status generating an alert were on average 43% higher, demonstrating the potential of such a system to preferentially target support at disadvantaged groups without needing to target directly based on immutable factors such as their socio-economic background.     

A systematic review of the role of learning analytics in enhancing feedback practices in higher education

Learning analytics (LA) offers new opportunities to enrich feedback practices in higher education, but little is understood about the ways different LA can enhance feedback practices for educators and students. This systematic literature review maps the current state of implementation of LA to improve feedback practices in technology-mediated learning environments in higher education. We used strict inclusion criteria to select relevant studies that have investigated the role of LA on feedback practices. To identify common features of LA for feedback studies, we coded relevant publications using an analytical framework that identifies four key dimensions of LA systems: what (types of data), how (analytic methods), why (objectives), and how educators and students are served by LA (stakeholders). Based on findings, we propose a conceptual framework that can guide the implementation of LA for feedback systems and also suggest future empirical research in this area.     

Towards mapping competencies through learning analytics: real-time competency assessment for career direction through interactive simulation

Abstract

The selection of career paths and making of academic choices is a difficult and often confusing task for young people. The impact on their lives, however, is enormous as it can determine entire future career possibilities. In India, a general remedy to this stress is that instead of choosing a field of study tailored to individual preferences and strengths, topics are chosen that align with the choices of the students’ families or their friends. This can have the effect of entrenching patterns of intergenerational inequity. The aim of this research is to give students greater access to the knowledge capital which will help them make better choices. This is achieved by engaging students in the career planning process, in order to convey information in a likeable and credible way. The COMPCAT (Competency and Career Assessment Tool) game engine combines the use of learning analytics and real time, interactive computer simulations designed to gain insights into the students’ engagement in the making of these complex decisions. This paper presents the conceptual architecture of the game and demonstrates its role in enhancing the learning effectiveness of the students.     

A systematic review on the deployment and effectiveness of data analytics in higher education to improve student outcomes

Abstract

This is a systematic review conducted of primary research literature published between 2007 and 2018 on the deployment and effectiveness of data analytics in higher education to improve student outcomes. We took a methodological approach to searching databases; appraising and synthesising results against predefined criteria. We reviewed research on the effectiveness of three differentiated forms of data analytics: learning, academic and learner analytics. Student outcomes are defined as retention, academic performance and engagement. Our results find that three quarters of studies report the use of educational data analytics to be effective in improving student outcomes but their relationship with student outcomes requires further and more robust investigation and assessment. We argue that research must interpret and communicate effectiveness qualitatively, as well as quantitatively, by including the student voice in assessments of impact.     

Supporting the less-adaptive student: the role of learning analytics,formative assessment and blended learning

Abstract

How can we best facilitate students most in need of learning support, entering a challenging quantitative methods module at the start of their bachelor programme? In this empirical study into blended learning and the role of assessment for and as learning, we investigate learning processes of students with different learning profiles. Specifically, we contrast learning episodes of two cluster analysis-based profiles, one profile more directed to deep learning and self-regulation, the other profile more directed toward stepwise learning and external regulation. In a programme based on problem-based learning, where students are supposedly being primarily self-directed, this first profile is regarded as being of an adaptive type, with the second profile less adaptive. Making use of a broad spectrum of learning and learner data, collected in the framework of a dispositional learning analytics application, we compare these profiles on learning dispositions, such as learning emotions, motivation and engagement, learning performance and trace variables collected from the digital learning environments. Outcomes suggest that the blended design of the module with the digital environments offering many opportunities for assessment of learning, for learning and as learning together with actionable learning feedback, is used more intensively by students of the less adaptive profile.     

Thinking critically about learning analytics,student outcomes,and equity of attainment

Abstract

This is the introduction to the special edition on learning analytics.     

Empowering learners with personalised learning approaches? Agency,equity and transparency in the context of learning analytics

Abstract

The emergence of personalised data technologies such as learning analytics is framed as a solution to manage the needs of higher education student populations that are growing ever more diverse and larger in size. However, the current approach to learning analytics presents tensions between increasing student agency in making learning-related decisions and ‘datafying’ students in the process of collecting, analysing and interpreting data. This article presents a study that explores staff and student experience of agency, equity and transparency in existing data practices and expectations towards learning analytics in a UK university. The results show a number of intertwined factors that have contributed to the tensions between enhancing a learner’s control of their studies and, at the same time, diminishing their autonomy as an active agent in the process of learning analytics. This article argues that learner empowerment should not be automatically assumed to have taken place as part of the adoption of learning analytics. Instead, the interwoven power relationships in a complex educational system and the interactions between humans and machines need to be taken into consideration when presenting learning analytics as an equitable process to enhance student agency and educational equity.     

Changing spaces of education: new perspectives on the nature of learning

This case study relates to distance learning students on open access courses. It demonstrates the use of predictive analytics to generate a model of the probabilities of success and retention at different points, or milestones, in a student journey. A core set of explanatory variables has been established and their varying relative importance at different milestones identified. The explanatory variables, milestones and reference points when the model is run will be different at other institutions but the approach may be generalised to distance learning institutions and, more broadly, to any HE institution. Institutions, and especially distance education institutions which do not have the advantages of frequently seeing students, need to make full use of any recorded information they hold to try and identify students who are, or become, at potential risk of leaving. The importance of different factors, at different milestones, may help tailor student support to individual students and therefore improve low retention in open access distance education.     

Students' personal networks in virtual and personal learning environments: a case study in higher education using learning analytics approach

The main objective of this paper is to analyse the effect of the affordances of a virtual learning environment and a personal learning environment (PLE) in the configuration of the students' personal networks in a higher education context. The results are discussed in light of the adaptation of the students to the learning network made up by two undergraduate, inter-university and online courses. Besides, we also examine the influence of this effect in the learning process. The findings reflect the effectiveness of a PLE for facilitating student participation and for assisting students in the creation of larger and more balanced personal networks with richer social capital. However, the findings do not provide evidences about a difference in the learning performance between the two environments. From a methodological point of view, this paper serves as an illustration of the analysis of personal networks on digital data collected from technology-enhanced learning environments.     

Supporting adaptive learning pathways through the use of learning analytics: developments,challenges and future opportunities

Learning Analytics (LA) and adaptive learning are inextricably linked since they both foster technology-supported learner-centred education. This study identifies developments focusing on their interplay and emphasises insufficiently investigated directions which display a higher innovation potential. Twenty-one peer-reviewed studies are identified via a systematic review and classified based on the dimensions of a proposed framework. The findings of the review suggest that interesting work has been carried out during the last years on the topic. We conclude with a need for more research on the topic in specific learning domains and settings. Recommendations include: a clear strategy for adaptation augmented by LA, the combination of on-task with pre-task measures, and the combination of system-controlled adaptation with user-controlled adaptation. Future trends include the emergence of constructivist-collaborative environments that provide insightful models of complex student behaviour.     

A Challenge for the Developer: Issues of interactivity and linguistic-cognitive appropriateness in English language learning

It is argued that language learning ought to be interactive. The traditional language classroom provides a favourable interactive situation for language learners. By contrast, the distance education mode is limited in some ways regarding language learning. Necessarily, distance education involves, primarily, self-learning. Face-to-face learning opportunities are scant compared with traditional classroom-based learning. It is, therefore, a challenge for language professionals to explore new ways of teaching language, the speaking skills in particular, within the constraints of the distance education mode. Being able to master English--an increasingly important international language--provides a key to information and learning societies in the 21 st century. In this paper, I will suggest some ways of maximizing the amount of interactive activities in English language distance learning, based on my work-based experience as a course developer of a new English enhancement course offered by the Open University of Hong Kong. I will focus on the development of listening and speaking skills, which are in fact the most problematic areas--the speaking skill, in particular, involves two-way communication. I will consider the issue from the perspectives of learners' motivation, peer support and technological support. Finally, I will present findings of a survey on the learners and tutors' feedback on the teaching materials of this new English language enhancement course.     

Learning design in diverse institutional and cultural contexts: suggestions from a participatory workshop with higher education professionals in Africa

Learning design approaches, such as those adopted by the Open University, provide a set of tools and resources for purposefully-designing modules with a focus on student experiences. However, many of the current learning design strategies have been situated within specific institutions in Europe and North America. This means that there are several issues worth considering around if and how established learning design approaches make sense in diverse institutional and cultural contexts. To critically assess the relevance and appropriateness of learning design strategies in new contexts, this article describes an in-depth participatory workshop with 34 education professionals from five African countries. Altogether, 10 suggestions for learning design practices were derived from the consensus of workshop participants, which provide a foundation for the development of learning design practices moving forward.     

Facilitating the learning journey from vocational education and training to higher education

An increase in students who enter higher education in Australia following their studies in the vocational education and training (VET) sector has led to heightened national interest in the transition experiences of these students. This paper reports on the experiences of students who, as a result of their VET studies, entered a relatively new, large, metropolitan university in 2009 and 2010 in the fields of Business and Law, Early Childhood and Nursing. The investigation elicited responses from students about a range of personal, administrative and academic experiences. A total of 529 students responded to a survey, 74 students were interviewed by telephone and 33 students attended focus groups. A major finding of the research was that many students saw the new learning context as a positive factor and responded to challenges with enthusiasm and resilience. Despite this, it seems that the transition period, particularly the first semester of study, is an unnecessarily stressful time for many students and that this could be mitigated by the provision of timely information about differences between institutional and learner expectations, workload, administrative processes and learning practices.     

Lesson study: A collaborative approach to scholarship for teaching and learning in higher education

The Scholarship of Teaching and Learning has become an important field of inquiry, focusing on the development of new and critical pedagogic approaches in higher education. It is a broad field leading to the emergence of a number of contrasting perspectives concerning the development of insights into teaching and learning. In this article, we explore the potential for Lesson Study to act as a framework for reflecting on and developing pedagogic practice in the university sector. Originating in Japan over a century ago, Lesson Study is a collaborative tool for analysing and developing understanding of student learning. This makes it an ideal tool for capturing and interrogating new and critical insights into teaching and learning. An outline framework is suggested for developing the use of Lesson Study in higher education and we discuss how it can form a positive methodology for extending the work of the Scholarship of Teaching and Learning.     

Facilitating flexible learning by replacing classroom time with an online learning environment: A systematic review of blended learning in higher education

Higher education institutions are trying to provide more flexibility and individualization, which is mainly realized through the use of new technologies and implemented in online or blended learning designs. This systematic review aims to investigate the impact of replacing classroom time with an online learning environment. The meta-analysis (k = 21 effect sizes) applied strict inclusion criteria concerning research design, measurement of learning outcomes and implementation of blended learning. The estimated effect size (Hedge's g) was positive, although not significantly different from zero and the confidence interval [-0.13, 0.25], suggesting that overall differences between blended and conventional classroom learning are small, and, at best, very small negative or moderate positive effects are plausible. This means that despite a reduction in classroom time between 30 and 79 per cent, equivalent learning outcomes were found. Consequently, blended learning with reduced classroom time is not systematically more or less effective than conventional classroom learning.     

Exploring learning and fit in the transition from higher education to the labour market: A systematic review

The transition from higher education to the labour market is an important period for youngsters, characterised by extensive changes which act as triggers for learning. Furthermore, students’ educational background and the (in)congruence with their work context is important. Accordingly, the aim of this systematic review is to explore the role of learning and fit in the transition process. Results indicate that most emphasis is put on theoretical knowledge, communication, problem-solving, and learning skills. Although the perception on what has to be learned differs for employers, educators, and graduates, each group valued generic competences most. Results show that transfer can be experienced in three ways and the need for learning at work is stressed. Concerning fit, four types of fit are distinguished: vertical, horizontal, competence, and person-environment fit. Several personal background characteristics are shown to influence fit and findings indicate that fit has an influence on career progress and personal resources.     

A social realist perspective on student learning in higher education: the morphogenesis of agency

Contemporary critiques of student learning research call for new theoretical and methodological approaches. This article proposes a social realist approach to this research, using the morphogenetic theory of sociologist Margaret Archer. The applicability of this approach is demonstrated by reference to an empirical study of engineering students at a South African university, using narrative analysis. In the article itself, two narratives are given in some detail, illustrating the key outlines of the analysis. Students’ emerging personal identities are shown to be highly dependent on their social backgrounds, yet when in the university the possibilities for the morphogenesis of student agency are very constrained. A critical interrogation of these findings proposes that a true higher education should facilitate the development of an enlarged sense of agency for students.     

Making sense of learner and learning Big Data: reviewing five years of Data Wrangling at the Open University UK

Most distance learning institutions collect vast amounts of learning data. Making sense of this ‘Big Data’ can be a challenge, in particular when data are stored at different data warehouses and require advanced statistical skills to interpret complex patterns of data. As a leading institute on learning analytics, the Open University UK instigated in 2012 a Data Wrangling initiative. This provided every Faculty with a dedicated academic with expertise data analysis and whose task is to provide strategic, pedagogical and sense-making advice to staff and senior management. Given substantial changes within the OU (e.g. new Faculty structure, real-time dashboards, two large-scale adoptions of predictive analytics approaches, increased reliance on analytics), this embedded case study provides an in-depth review of lessons learned of five years of data wrangling. We will elaborate on the design of the new structure, its strengths and potential weaknesses, and affordances to be adopted by other institutions.