A systematic review on the deployment and effectiveness of data analytics in higher education to improve student outcomes

Abstract

This is a systematic review conducted of primary research literature published between 2007 and 2018 on the deployment and effectiveness of data analytics in higher education to improve student outcomes. We took a methodological approach to searching databases; appraising and synthesising results against predefined criteria. We reviewed research on the effectiveness of three differentiated forms of data analytics: learning, academic and learner analytics. Student outcomes are defined as retention, academic performance and engagement. Our results find that three quarters of studies report the use of educational data analytics to be effective in improving student outcomes but their relationship with student outcomes requires further and more robust investigation and assessment. We argue that research must interpret and communicate effectiveness qualitatively, as well as quantitatively, by including the student voice in assessments of impact.     

大数据时代学习分析与高职院校教育改革

面对来势汹涌的大数据时代,如何有效度量、发现、评估与操纵信息化学习管理系统中积累的海量数据,已成为未来高职院校教育改革过程中不可回避的棘手问题。学习分析是大数据时代高职院校教育改革的助推器,是解决这一现实问题的有效途径之一。从高职院校教育改革在大数据时代所遭遇的挑战出发,阐述学习分析技术对高职院校教育改革的推动作用,并从高职院校教育管理、教师教学与学生学习三个方面分析学习分析技术对高职院校教育改革的影响。     

Learning Analytics: at the Nexus of Big Data,Digital Innovation,and Social Justice in Education

We are still designing educational experiences for the average student, and have room to improve. Learning analytics provides a way forward. This commentary describes how learning analytics-based applications are well positioned to meaningfully personalize the learning experience in diverse ways. In so doing, learning analytics has the potential to contribute to more equitable and socially just educational outcomes for students who might otherwise be seen through the lens of the average student. Utilizing big data, good design, and the input of the stakeholders, learning analytics techniques aim to develop applications for the sole purpose of reducing the classroom size to 1. Over time, these digital innovations will enable us to do away with a model of education that teaches toward the non-existent average student, replacing it with one that is more socially just—one that addresses the individual needs of every student.     

Challenging Prospective Computer Engineers to Design Educational Software by Engaging Them in a Constructivist Learning Environment

This study focuses on the development of prospective computer engineers' knowledge of educational software design through their involvement in a constructivist learning environment emphasizing project based learning. Within this environment prospective computer engineers (PCE) adopted a variety of roles namely: learners, teachers, users, designers, implementers and evaluators of educational software dealing with concepts of computer science while also taking into account theoretical educational considerations regarding constructivism and social views of knowledge construction. The analysis of the data shows that PCE frequently start by considering traditional behavioristic views regarding teaching and learning as well as regarding the design of educational software. The PCE progressed to accept more constructivist views regarding teaching and learning as well as to designing educational software by communicating their knowledge with their colleagues and the teacher in the project based context. PCE also progressed through the evaluation of the educational software using real classrooms. By exploiting the given feedback, PCE improved the quality of software specifications.     

学习分析技术:挖掘大数据时代下教育数据的价值

当前,大数据时代已经来临,教育领域同样积累了海量数据。教育领域已经部署了众多的学习管理系统,在这些软件系统中存储着海量的学习者信息及学习过程数据。如何利用这些数据,使这些数据转变为信息、知识,并为教学决策、学习优化服务,已成为教育工作者以及学习者们所关注的内容。学习分析技术有助于发挥学习过程数据的价值,使数据成为审慎决策、过程优化的重要依据。该文介绍了国内外学习分析技术研究现状,归纳出学习分析技术的关键技术及分析模式,并以实例从不同用户视角包括管理者、辅导教师、学习者展示了学习分析技术在网络学习过程分析中的应用过程。     

Construction Direction and Development Path of Digital Educational Resources in the Era of New Educational Infrastructure Construction

Digital educational resources have become essential elements of modern educational public services. China’s national program on new educational infrastructure construction has proposed to take new infrastructure construction of digital educational resources as one of the critical tasks. This paper aims to provide a theoretical basis for advancing the reform and development of digital educational resources in the new era. It classifies the current development and realistic challenges of China’s digital educational resources, and summaries and interprets the directions of the three construction projects of new type resources and tools, the resource supply system, and the resource supervisory system. It depicts the prospect of digital educational resource construction in the intelligent era in the combination of the strategic layout of the policies on new educational infrastructure construction. It also discusses such key technologies supporting new infrastructure construction of digital educational resources as multimodal learning analytics, disciplinary knowledge graphs, machine learning, and blockchain. It finally proposes a specific development path for new infrastructure construction of digital educational resources.     

Netflixing human capital development: personalized learning technology and the corporatization of K-12 education

Abstract

Advanced by powerful venture philanthropies, educational technology companies, and the US Department of Education, a growing movement to apply ‘big data’ through ‘learning analytics’ to create ‘personalized learning’ is currently underway in K-12 education in the United States. While scholars have offered various critiques of the corporate school reform agenda, the role of personalized learning technology in the corporatization of public education has not been extensively examined. Through a content analysis of US Department of Education reports, personalized learning advocacy white papers, and published research monographs, this paper details how big data and adaptive learning systems are functioning to redefine educational policy, teaching, and learning in ways that transfer educational decisions from public school classrooms and teachers to private corporate spaces and authorities. The analysis shows that all three types of documents position education within a reductive set of economic rationalities that emphasize human capital development, the expansion of data-driven instruction and decision-making, and a narrow conception of learning as the acquisition of discrete skills and behavior modification detached from broader social contexts and culturally relevant forms of knowledge and inquiry. The paper concludes by drawing out the contradictions inherent to personalized learning technology and corporatization of schooling. It argues that these contradictions necessitate a broad rethinking of the value and purpose of new educational technology.     

The Givenness of the Human Learning Experience and Its Incompatibility with Information Analytics

Abstract

The rise of learning analytics, the application of complex metrics developed to exploit the proliferation of ‘Big Data’ in educational work, raises important moral questions about the nature of what is measurable in education. Teachers, schools and nations are increasingly held to account based on metrics, exacerbating the tendency for fine-grained measurement of learning experiences. In this article, the origins of learning analytics ontology are explored, drawing upon core ideas in the philosophy of computing, such as the general definition of information and the information-theoretic account of knowledge. Drawing upon a reading of Descartes Meditatio II, which extends the phenomenology of Jean-Luc Marion into a pedagogy of intentionality, the article identifies a fundamental incompatibility between the subjective experience of learning and the information-theoretic account of knowledge. Human subjects experience and value their own information incommensurably with the ways in which computers measure and quantify information. The consequences of this finding for the design of online learning environments, and the necessary limitations of learning analytics and measurement are explored.     

An overview of learning analytics

Learning analytics, the analysis and representation of data about learners in order to improve learning, is a new lens through which teachers can understand education. It is rooted in the dramatic increase in the quantity of data about learners and linked to management approaches that focus on quantitative metrics, which are sometimes antithetical to an educational sense of teaching. However, learning analytics offers new routes for teachers to understand their students and, hence, to make effective use of their limited resources. This paper explores these issues and describes a series of examples of learning analytics to illustrate the potential. It argues that teachers can and should engage with learning analytics as a way of influencing the metrics agenda towards richer conceptions of learning and to improve their teaching.     

Analytics for learning design: A layered framework and tools

The field of learning design studies how to support teachers in devising suitable activities for their students to learn. The field of learning analytics explores how data about students' interactions can be used to increase the understanding of learning experiences. Despite its clear synergy, there is only limited and fragmented work exploring the active role that data analytics can play in supporting design for learning. This paper builds on previous research to propose a framework (analytics layers for learning design) that articulates three layers of data analytics—learning analytics, design analytics and community analytics—to support informed decision-making in learning design. Additionally, a set of tools and experiences are described to illustrate how the different data analytics perspectives proposed by the framework can support learning design processes.     

Empowering learners with personalised learning approaches? Agency,equity and transparency in the context of learning analytics

Abstract

The emergence of personalised data technologies such as learning analytics is framed as a solution to manage the needs of higher education student populations that are growing ever more diverse and larger in size. However, the current approach to learning analytics presents tensions between increasing student agency in making learning-related decisions and ‘datafying’ students in the process of collecting, analysing and interpreting data. This article presents a study that explores staff and student experience of agency, equity and transparency in existing data practices and expectations towards learning analytics in a UK university. The results show a number of intertwined factors that have contributed to the tensions between enhancing a learner’s control of their studies and, at the same time, diminishing their autonomy as an active agent in the process of learning analytics. This article argues that learner empowerment should not be automatically assumed to have taken place as part of the adoption of learning analytics. Instead, the interwoven power relationships in a complex educational system and the interactions between humans and machines need to be taken into consideration when presenting learning analytics as an equitable process to enhance student agency and educational equity.     

Effective usage of learning analytics: what do practitioners want and where should distance learning institutions be going?

ABSTRACT

The implementation of learning analytics may empower distance learning institutions to provide real-time feedback to students and teachers. Given the leading role of the Open University UK (OU) in research and application of learning analytics, this study aims to share the lessons learned from the experiences of 42 participants from a range of faculty, academic and professional positions, and expertise with learning analytics. Furthermore, we explored where distance learning institutions should be going next in terms of learning analytics adoption. The findings from the Learning Analytics User Stories (LAUS) workshop indicated four key areas where more work is needed: communication, personalisation, integrated design, and development of an evidence-base. The workshop outputs signalled the aspiration for an integrated analytics system transcending the entire student experience, from initial student inquiry right through to qualification completion and into life-long learning. We hope that our study will spark discussion on whether (or not) distance learning institutions should pursue the dream of an integrated, personalised, and evidence-based learning analytics system that clearly communicates useful feedback to staff and students, or whether this will become an Orwellian nightmare.     

There is no business model for open educational resources: a business model approach

The economic proverb ‘There is no such thing such as a free lunch’ applies also to open educational resources (OER). In recent years, several authors have used revenue models and business models to analyse the different sources of possible funding for OER. In this article the business models of Osterwalder and Chesbrough are combined with research on the motives of the participants of OER to analyse possible funding models. If the motives of governments (knowledge economy), educational institutions (efficiency, marketing), individual producers (reputation, academic interests) and users (intermediary educational products, learning) are combined, it is shown that the only long-term sustainable independent business model is based on subsidies. However, this conclusion depends both on the definition of openness (in the sense of at no cost) and on motives. More research on both aspects could alter these conclusions.     

What changes,and for whom? A study of the impact of learning analytics-based process feedback in a large course

Recent developments in educational technologies have provided a viable solution to the challenges associated with scaling personalised feedback to students. However, there is currently little empirical evidence about the impact such scaled feedback has on student learning progress and study behaviour. This paper presents the findings of a study that looked at the impact of a learning analytics (LA)-based feedback system on students' self-regulated learning and academic achievement in a large, first-year undergraduate course. Using the COPES model of self-regulated learning (SRL), we analysed the learning operations of students, by way of log data from the learning management system and e-book, as well as the products of SRL, namely, performance on course assessments, from three years of course offerings. The latest course offering involved an intervention condition that made use of an educational technology to provide LA-based process feedback. Propensity score matching was employed to match a control group to the student cohort enrolled in the latest course offering, creating two equal-sized groups of students who received the feedback (the experimental group) and those who did not (the control group). Growth mixture modelling and mixed between-within ANOVA were also employed to identify differences in the patterns of online self-regulated learning operations over the course of the semester. The results showed that the experimental group showed significantly different patterns in their learning operations and performed better in terms of final grades. Moreover, there was no difference in the effect of feedback on final grades among students with different prior academic achievement scores, indicating that the LA-based feedback deployed in this course is able to support students’ learning, regardless of prior academic standing.     

学习分析:正在浮现中的数据技术

随着教育信息化的普及与逐渐深入,学习管理系统已经获取并存储了大量的有关学生复杂学习行为的数据,从这些数据中挖掘出改进教学系统、提升学习效果的信息,在教育信息化领域一直有着巨大的吸引力。因此,有必要从分析数据以改进学习的角度,对日益受到关注的学习分析技术进行解读。首先,学习分析技术是测量、收集、分析和报告有关学生的学习行为以及学习环境的数据,用以理解和优化学习及其产生的环境的技术。接着,综述学习分析技术的发展,指出其在教育中有着广泛的应用前景和巨大的发展潜力:学习分析技术可作为教师教学决策、优化教学的有效支持工具,也可为学生的自我导向学习、学习危机预警和自我评估提供有效数据支持,还可为教育研究者的个性化学习设计和增进研究效益提供数据参考。最后,提出学习分析技术也存在隐私、准确性和兼容性等诸多挑战和问题。     

教育思维:教育观和教育操作思路的统一体

教育思维是一个新教育理论范畴。所谓教育思维,是人类的教育实践理性,是教育理论认识在教育实践面前的凝结,也是教育实践经验在人们认识中的凝结;就其实质来说,是一定的教育观及其支配下的教育操作思路的统一体。教育观和教育操作思路在教育思维中的统一是必然的,其实质是人的意识功能在教育认识中的反映。     

教育效益的追问：从学习分析技术的视角

随着我国教育信息化步伐的不断加快,信息化环境下的未来教育生态逐步成型,全面覆盖的网络化基础设施、广泛共享的优质资源使得人们日益关注技术应用于教育的效益问题。当前,高投入的信息化设施建设并没有获得教育效益的高产出。如何在有限的投资中获取更大的教育效益,成为越来越多教育工作者和管理者热议的话题。学习分析技术作为一种能够在海量数据中提取隐含、潜在应用价值信息的工具,可以对学习者、教师、管理部门等相关利益方进行干预支持,形成助力,改善对教与学的理解,促进和优化学习效果和环境,进而提升数字时代的教育内外部效益。当前学习分析技术还处于发展阶段,主要着力于内部效益瓶颈的突破,在外部效益方面还没有显现出足够强大的潜力。随着学习分析技术的不断发展,其教育分析能力也需要从基础性分析、推测性分析发展到更加成熟的预测性分析,集成分散式服务、嵌入式共享服务以及独立式共享服务三种运营模型,实现对教育干预的决策洞察和预测水平。我国应合理规划,形成良好的数据管理机制,深入挖掘、综合利用有价值的数据信息,将数据转化为知识,实现教育效益的真正提高。     

Improving workplace-based assessment and feedback by an E-portfolio enhanced with learning analytics

Electronic portfolios (E-portfolios) are crucial means for workplace-based assessment and feedback. Although E-portfolios provide a useful approach to view each learner’s progress, so far options for personalized feedback and potential data about a learner’s performances at the workplace often remain unexploited. This paper advocates that E-portfolios enhanced with learning analytics, might increase the quality and efficiency of workplace-based feedback and assessment in professional education. Based on a 5-phased iterative design approach, an existing E-portfolio environment was enhanced with learning analytics in professional education. First, information about crucial professional activities for professional domains and suited assessment instruments were collected (phase 1). Thereafter probabilistic student models were defined (phase 2). Next, personalized feedback and visualization of the personal development over time were developed (phase 3). Then the prototype of the E-portfolio—including the student models and feedback and visualization modules—were implemented in professional training-programs (phase 4). Last, evaluation cycles took place and 121 students and 30 supervisors from five institutes for professional education evaluated the perceived usefulness of the design (phase 5). It was concluded that E-portfolios with learning analytics were perceived to assist the development of students’ professional competencies and that the design is only successful when developed and implemented through the eyes of the users. Feedback and assessment methods based upon learning analytics can stimulate learning at the workplace in the long run. Practical, technological and ethical challenges are discussed.     

“实践性学习”：一种新型的职前教师教育模式

基于教育问题的“实践性学习”强调教育实践在教师教育过程中的作用，是培养师范生教育胜任力的一种新型模式。在教育理念上，“实践性学习“强调“知行合一”，侧重“做中学”，认为职前教师教育仅仅传授教育知识是远远不够的，还必须提供教育实践机会，在“实践场景”下让他们尽早进入专业领域，参与教育实践活动，才能掌握了教育的程序和方法，获得教育胜任力。     

Understanding privacy and data protection issues in learning analytics using a systematic review

The field of learning analytics has advanced from infancy stages into a more practical domain, where tangible solutions are being implemented. Nevertheless, the field has encountered numerous privacy and data protection issues that have garnered significant and growing attention. In this systematic review, four databases were searched concerning privacy and data protection issues of learning analytics. A final corpus of 47 papers published in top educational technology journals was selected after running an eligibility check. An analysis of the final corpus was carried out to answer the following three research questions: (1) What are the privacy and data protection issues in learning analytics? (2) What are the similarities and differences between the views of stakeholders from different backgrounds on privacy and data protection issues in learning analytics? (3) How have previous approaches attempted to address privacy and data protection issues? The results of the systematic review show that there are eight distinct, intertwined privacy and data protection issues that cut across the learning analytics cycle. There are both cross-regional similarities and three sets of differences in stakeholder perceptions towards privacy and data protection in learning analytics. With regard to previous attempts to approach privacy and data protection issues in learning analytics, there is a notable dearth of applied evidence, which impedes the assessment of their effectiveness. The findings of our paper suggest that privacy and data protection issues should not be relaxed at any point in the implementation of learning analytics, as these issues persist throughout the learning analytics development cycle. One key implication of this review suggests that solutions to privacy and data protection issues in learning analytics should be more evidence-based, thereby increasing the trustworthiness of learning analytics and its usefulness.

Practitioner notes

What is already known about this topic

• Research on privacy and data protection in learning analytics has become a recognised challenge that hinders the further expansion of learning analytics.
• Proposals to counter the privacy and data protection issues in learning analytics are blurry; there is a lack of a summary of previously proposed solutions.

What this study contributes

• Establishment of what privacy and data protection issues exist at different phases of the learning analytics cycle.
• Identification of how different stakeholders view privacy, similarities and differences, and what factors influence their views.
• Evaluation and comparison of previously proposed solutions that attempt to address privacy and data protection in learning analytics.

Implications for practice and/or policy

• Privacy and data protection issues need to be viewed in the context of the entire cycle of learning analytics.
• Stakeholder views on privacy and data protection in learning analytics have commonalities across contexts and differences that can arise within the same context. Before implementing learning analytics, targeted research should be conducted with stakeholders.
• Solutions that attempt to address privacy and data protection issues in learning analytics should be put into practice as far as possible to better test their usefulness.