A systematic review of the role of learning analytics in enhancing feedback practices in higher education

Learning analytics (LA) offers new opportunities to enrich feedback practices in higher education, but little is understood about the ways different LA can enhance feedback practices for educators and students. This systematic literature review maps the current state of implementation of LA to improve feedback practices in technology-mediated learning environments in higher education. We used strict inclusion criteria to select relevant studies that have investigated the role of LA on feedback practices. To identify common features of LA for feedback studies, we coded relevant publications using an analytical framework that identifies four key dimensions of LA systems: what (types of data), how (analytic methods), why (objectives), and how educators and students are served by LA (stakeholders). Based on findings, we propose a conceptual framework that can guide the implementation of LA for feedback systems and also suggest future empirical research in this area.     

Thinking critically about learning analytics,student outcomes,and equity of attainment

Abstract

This is the introduction to the special edition on learning analytics.     

Understanding privacy and data protection issues in learning analytics using a systematic review

The field of learning analytics has advanced from infancy stages into a more practical domain, where tangible solutions are being implemented. Nevertheless, the field has encountered numerous privacy and data protection issues that have garnered significant and growing attention. In this systematic review, four databases were searched concerning privacy and data protection issues of learning analytics. A final corpus of 47 papers published in top educational technology journals was selected after running an eligibility check. An analysis of the final corpus was carried out to answer the following three research questions: (1) What are the privacy and data protection issues in learning analytics? (2) What are the similarities and differences between the views of stakeholders from different backgrounds on privacy and data protection issues in learning analytics? (3) How have previous approaches attempted to address privacy and data protection issues? The results of the systematic review show that there are eight distinct, intertwined privacy and data protection issues that cut across the learning analytics cycle. There are both cross-regional similarities and three sets of differences in stakeholder perceptions towards privacy and data protection in learning analytics. With regard to previous attempts to approach privacy and data protection issues in learning analytics, there is a notable dearth of applied evidence, which impedes the assessment of their effectiveness. The findings of our paper suggest that privacy and data protection issues should not be relaxed at any point in the implementation of learning analytics, as these issues persist throughout the learning analytics development cycle. One key implication of this review suggests that solutions to privacy and data protection issues in learning analytics should be more evidence-based, thereby increasing the trustworthiness of learning analytics and its usefulness.

Practitioner notes

What is already known about this topic

• Research on privacy and data protection in learning analytics has become a recognised challenge that hinders the further expansion of learning analytics.
• Proposals to counter the privacy and data protection issues in learning analytics are blurry; there is a lack of a summary of previously proposed solutions.

What this study contributes

• Establishment of what privacy and data protection issues exist at different phases of the learning analytics cycle.
• Identification of how different stakeholders view privacy, similarities and differences, and what factors influence their views.
• Evaluation and comparison of previously proposed solutions that attempt to address privacy and data protection in learning analytics.

Implications for practice and/or policy

• Privacy and data protection issues need to be viewed in the context of the entire cycle of learning analytics.
• Stakeholder views on privacy and data protection in learning analytics have commonalities across contexts and differences that can arise within the same context. Before implementing learning analytics, targeted research should be conducted with stakeholders.
• Solutions that attempt to address privacy and data protection issues in learning analytics should be put into practice as far as possible to better test their usefulness.

     

The effectiveness of learning analytics for identifying at-risk students in higher education

Abstract

In this article we investigate the effectiveness of learning analytics for identifying at-risk students in higher education institutions using data output from an in-situ learning analytics platform. Amongst other things, the platform generates ‘no-engagement’ alerts if students have not engaged with any of the data sources measured for 14 consecutive days. We tested the relationship between these alerts and student outcomes for two cohorts of first-year undergraduate students. We also compared the efficiency of using these alerts to identify students at risk of poorer outcomes with the efficiency of using demographic data, using widening participation status as a case study example. The no-engagement alerts were found to be more efficient at spotting students not progressing and not attaining than demographic data. In order to investigate the efficacy of learning analytics for addressing differential student outcomes for disadvantaged groups, the team also analysed the likelihood of students with widening participation status generating alerts compared with their non-widening participation counterparts. The odds of students with widening participation status generating an alert were on average 43% higher, demonstrating the potential of such a system to preferentially target support at disadvantaged groups without needing to target directly based on immutable factors such as their socio-economic background.     

Being more human: rooting learning analytics through resistance and reconnection with the values of higher education

ABSTRACT

Universities are now compelled to attend to metrics that (re)shape our conceptualisation of the student experience. New technologies such as learning analytics (LA) promise the ability to target personalised support to profiled ‘at risk’ students through mapping large-scale historic student engagement data such as attendance, library use, and virtual learning environment activity as well as demographic information and typical student outcomes. Yet serious ethical and implementation issues remain. Data-driven labelling of students as ‘high risk’, ‘hard to reach’ or ‘vulnerable’ creates conflict between promoting personal growth and human flourishing and treating people merely as data points. This article argues that universities must resist the assumption that numbers and algorithms alone can solve the ‘problem’ of student retention and performance; rather, LA work must be underpinned by a reconnection with the agreed values relating to the purpose of higher education, including democratic engagement, recognition of diverse and individual experience, and processes of becoming. Such a reconnection, this article contends, is possible when LA work is designed and implemented in genuine collaboration and partnership with students.     

Three in-course assessment reforms to improve higher education learning outcomes

A current international concern is that, for too large a proportion of graduates, their higher order cognitive and practical capabilities are below acceptable levels. The constituent courses of academic programmes are the most logical sites for developing these capabilities. Contributing to patchy attainment are deficiencies in three particular aspects of assessment practice: the design and specifications of many assessment tasks; the minimum requirements for awarding a passing grade in a course and granting credit towards the degree; and the accumulation of points derived from quizzes, assessments or activities completed during the teaching period. Rethinking and reforming these would lead to improvements for significant sub-populations of students. Pursuing such a goal would also have significant positive implications for academic teachers, but be contingent on favourable contextual settings including departmental and institutional priorities.     

Multilevel analysis of student pathways to higher education

This study examined the relationship between academic preparation and post-secondary educational outcomes. To uncover the factors that influence secondary-level school course-taking patterns and outcomes, multi-level modelling was conducted using data from a nationally representative sample of US secondary-level school students from the Education Longitudinal Study of 2002. Results indicated (1) significant effects from student, family and school predictors on course-taking patterns and (2) significant effects from course-taking patterns on enrollment in post-secondary institution types with math course-taking pattern having the largest impact. Implications for educators and administrators are discussed.     

Exploring learning and fit in the transition from higher education to the labour market: A systematic review

The transition from higher education to the labour market is an important period for youngsters, characterised by extensive changes which act as triggers for learning. Furthermore, students’ educational background and the (in)congruence with their work context is important. Accordingly, the aim of this systematic review is to explore the role of learning and fit in the transition process. Results indicate that most emphasis is put on theoretical knowledge, communication, problem-solving, and learning skills. Although the perception on what has to be learned differs for employers, educators, and graduates, each group valued generic competences most. Results show that transfer can be experienced in three ways and the need for learning at work is stressed. Concerning fit, four types of fit are distinguished: vertical, horizontal, competence, and person-environment fit. Several personal background characteristics are shown to influence fit and findings indicate that fit has an influence on career progress and personal resources.     

Supporting the less-adaptive student: the role of learning analytics,formative assessment and blended learning

Abstract

How can we best facilitate students most in need of learning support, entering a challenging quantitative methods module at the start of their bachelor programme? In this empirical study into blended learning and the role of assessment for and as learning, we investigate learning processes of students with different learning profiles. Specifically, we contrast learning episodes of two cluster analysis-based profiles, one profile more directed to deep learning and self-regulation, the other profile more directed toward stepwise learning and external regulation. In a programme based on problem-based learning, where students are supposedly being primarily self-directed, this first profile is regarded as being of an adaptive type, with the second profile less adaptive. Making use of a broad spectrum of learning and learner data, collected in the framework of a dispositional learning analytics application, we compare these profiles on learning dispositions, such as learning emotions, motivation and engagement, learning performance and trace variables collected from the digital learning environments. Outcomes suggest that the blended design of the module with the digital environments offering many opportunities for assessment of learning, for learning and as learning together with actionable learning feedback, is used more intensively by students of the less adaptive profile.     

Using learning analytics to assess student engagement and academic outcomes in open access enabling programmes

Curriculum design, teaching methods, assessments and range of academic support need to be inclusive in Open Access Enabling courses. The findings of this study confirm a correlation between student access to online learning materials and a positive impact on grades in science courses. More specifically, students who frequently use the online learning system to access materials have better assessment and exam results.     

Learning design and learning analytics in mobile and ubiquitous learning: A systematic review

Mobile and Ubiquitous Learning (m/u-learning) are finding an increasing adoption in education. They are often distinguished by hybrid learning environments that encompass elements of formal and informal learning, in activities that happen in distributed settings (indoors and outdoors), across physical and virtual spaces. Despite their purported benefits, these environments imply additional complexity in the design, monitoring and evaluation of learning activities. The research literature on learning design (LD) and learning analytics (LA) has started to deal with these issues. This paper presents a systematic literature review of LD and LA, in m/u-learning. Apart from providing an overview of the current research in the field, this review elicits elements of common ground between both communities, as shown by the similar learning contexts and complementary research contributions, and based on the research gaps, proposes to: address m/u-learning beyond higher education settings, reinforce the connection between physical and virtual learning spaces, and more systematically align LD and LA processes.     

A large-scale test of the relationship between procrastination and performance using learning analytics

Abstract

Many studies have found a relationship between students’ self-reported procrastination and their grades. Few studies have used learning analytic data as a behavioural measure of procrastination in order to predict performance, and there is no systematic research on how this relationship may differ across assessments or disciplines. In this study we analyse nine years’ worth of institutional electronic submission records, a total of 73,608 assignment submissions, to examine the relationship between submission time and grades across assignments, students, courses, and disciplines in higher education. A significant negative relationship was found overall, with students who submitted closer to the deadline obtaining lower grades, however the size of the relationship was negligible, accounting for less than 1% of the variance in grades. The relationship varied significantly depending on student, assignment, course and discipline.     

A systematic review of evidence on the effectiveness of interventions and strategies for widening participation in higher education

Widening participation (WP) in higher education (HE) is an increasingly important policy issue, with interventions to increase participation from minority ethnic, low-income and other under-represented groups undertaken in HE sectors in many countries. In the UK there is a large amount of WP activity but a lack of robust evidence of its effectiveness. This article presents a systematic review in the topic area of WP in HE. We included studies of systematic review, randomised controlled trial (RCT) and quasi-experimental (QE) designs, and assessed evidence of the effectiveness of university access strategies and approaches in relation to the participation of disadvantaged students at university. We searched for, quality appraised and synthesised the international evidence, that is, evidence published in any country, in the English language. The findings from 4 systematic reviews and 12 experimental studies (4 RCTs, 4 RDDs and 4 QEDs) are presented as narrative syntheses in a series of thematic sub-topics. We found some evidence of effectiveness for a number of university access interventions. ‘Black box’ WP programmes (those with multiple elements in a single programme) and financial incentives were found to be effective. However, much of the evidence had design limitations and the majority were conducted in the US. The article concludes with research recommendations in relation to UK interventions, including suggested designs for future quasi-experimental evaluation.     

A review of twenty-first century higher education

This article is predominantly concerned with the global challenges associated with managing an academic workforce in an era characterised by increased demand for higher education. In scrutinising global trends in higher education and academic workforce management, the article will address two research questions. First, what are the global trends that most affect education in the twenty-first century? Second, what are the transformations that have taken place in the academic workforce amid these trends? Last, the article calls for a need for researchers to profoundly explore the management of the academic workforce cross-nationally between developed and developing economies, with a particular focus on Australia, Malaysia and Singapore.     

Assessment,technology and democratic education in the age of data

This paper contends that powerful techniques to manipulate data, enabled by technological and economic developments, can be easily co-opted to serve the restrictive frameworks of hyper-controlling, managerial accountability that characterise current cultures of summative assessment in education. In response to these challenges, research is urgently needed to increase our understanding of the impact that assessments have on individuals and society. The paper concludes that social research ought to contribute to the identification of responses – educational, technological and political – that can minimise inequalities and potential abuses through the encouragement of data literacy across society.     

A systematic review of interventions embedded in curriculum to improve university student wellbeing

BackgroundAs concerns around student mental health increase, universities are looking at preventative and universal interventions. The aim was to conduct a systematic review of curriculum-embedded interventions that target student mental health and wellbeing at university.MethodThis was a systematic review of longitudinal pre-/post-studies of curriculum-embedded interventions to improve the mental health and wellbeing of students. Seven electronic databases were searched from June 2015 to May 2020. The vote counting method was used to synthesise studies.ResultsForty-six studies were included in the review. Studies were heterogeneous, and mostly underpowered and rated ‘poor’ in the risk of bias assessment due to poor and inconsistent reporting. Overall, most curriculum-embedded interventions did not influence stress or anxiety.DiscussionThere is no strong evidence to support the impact of curriculum-embedded interventions for improving student mental health or wellbeing. Greater funding opportunities would allow for multi-programme and inter-institutional collaboration to improve the power of studies. Improved quality of reporting would enable high-quality meta-analyses, optimizing conclusions being drawn.