The role of learning theory in multimodal learning analytics

This study presents the outcomes of a semi-systematic literature review on the role of learning theory in multimodal learning analytics (MMLA) research. Based on previous systematic literature reviews in MMLA and an additional new search, 35 MMLA works were identified that use theory. The results show that MMLA studies do not always discuss their findings within an established theoretical framework. Most of the theory-driven MMLA studies are positioned in the cognitive and affective domains, and the three most frequently used theories are embodied cognition, cognitive load theory and control–value theory of achievement emotions. Often, the theories are only used to inform the study design, but there is a relationship between the most frequently used theories and the data modalities used to operationalize those theories. Although studies such as these are rare, the findings indicate that MMLA affordances can, indeed, lead to theoretical contributions to learning sciences. In this work, we discuss methods of accelerating theory-driven MMLA research and how this acceleration can extend or even create new theoretical knowledge.

Practitioner notes

What is already known about this topic

• Multimodal learning analytics (MMLA) is an emerging field of research with inherent connections to advanced computational analyses of social phenomena.
• MMLA can help us monitor learning activity at the micro-level and model cognitive, affective and social factors associated with learning using data from both physical and digital spaces.
• MMLA provide new opportunities to support students' learning.

What this paper adds

• Some MMLA works use theory, but, overall, the role of theory is currently limited.
• The three theories dominating MMLA research are embodied cognition, control–value theory of achievement emotions and cognitive load theory.
• Most of the theory-driven MMLA papers use theory ‘as is’ and do not consider the analytical and synthetic role of theory or aim to contribute to it.

Implications for practice and/or policy

• If the ultimate goal of MMLA, and AI in Education in general, research is to understand and support human learning, these studies should be expected to align their findings (or not) with established relevant theories.
• MMLA research is mature enough to contribute to learning theory, and more research should aim to do so.
• MMLA researchers and practitioners, including technology designers, developers, educators and policy-makers, can use this review as an overview of the current state of theory-driven MMLA.

     

Temporal analysis of multimodal data to predict collaborative learning outcomes

The analysis of multiple data streams is a long-standing practice within educational research. Both multimodal data analysis and temporal analysis have been applied successfully, but in the area of collaborative learning, very few studies have investigated specific advantages of multiple modalities versus a single modality, especially combined with temporal analysis. In this paper, we investigate how both the use of multimodal data and moving from averages and counts to temporal aspects in a collaborative setting provides a better prediction of learning gains. To address these questions, we analyze multimodal data collected from 25 9–11-year-old dyads using a fractions intelligent tutoring system. Assessing the relation of dual gaze, tutor log, audio and dialog data to students' learning gains, we find that a combination of modalities, especially those at a smaller time scale, such as gaze and audio, provides a more accurate prediction of learning gains than models with a single modality. Our work contributes to the understanding of how analyzing multimodal data in temporal manner provides additional information around the collaborative learning process.     

What does physiological synchrony reveal about metacognitive experiences and group performance?

There is a growing body of research on physiological synchrony (PS) in Collaborative Problem Solving (CPS). However, the current literature presents inconclusive findings about the way in which PS is reflected in cognitive and affective group processes and performance. In light of this, this study investigates the relationship between PS and metacognitive experiences (ie, judgement of confidence, task interest, task difficulty, mental effort and emotional valence) that are manifested during CPS. In addition, the study explores the association between PS and group performance. The participants were 77 university students who worked together on a computer-based CPS simulation in groups of three. Participants’ electrodermal activity (EDA) was recorded as they worked on the simulation and metacognitive experiences were measured with situated self-reports. A Multidimensional Recurrence Quantification Analysis was used to calculate the PS among the collaborators. The results show a positive relationship between continuous PS episodes and groups’ collective mental effort. No relationship was found between PS and judgement of confidence, task interest, task difficulty or emotional valence. The relationship between PS and group performance was also non-significant. The current work addresses several challenges in utilising multimodal data analytics in CPS research and discusses future research directions.     

Enhancing consent forms to support participant decision making in multimodal learning data research

Advances in the field of multimodal learning analytics (MMLA) research is often accomplished by actively exploring new technologies and techniques related to the collection and analysis of data. Exploration of ethical principles and procedures for governing the use of new technologies and techniques, however, is not as readily pursued. As collected data grow in complexity and invasiveness, potentially, a growing need is arising to scrutinize ethical aspects of MMLA research. In our study, we introduce an informed consent comprehension test for educational technology research and assess the effects of enhancing MMLA consent forms on comprehension of informed consent and on rates of enrollment in a MMLA study. One form is written from a researcher perspective and the other from a participant perspective. Results of the study involving first-year undergraduate students suggest that the overall level of comprehension did not differ between conditions. Yet, the participant-oriented consent form resulted in significantly lower rates of enrollment. Implications for MMLA researchers are discussed.     

Learning analytics to uncover inequality in behavioural engagement and academic attainment in a distance learning setting

Abstract

Although the attainment gap between black and minority ethnic (BME) students and white students has persisted for decades, the potential causes of these disparities are highly debated. The emergence of learning analytics allows researchers to understand how students engage in learning activities based on their digital traces in a naturalistic setting. This study investigates the attainment gap by analysing the differences in behavioural engagement between different ethnic groups. Using multilevel models of academic performance, demographics, and online traces of 149,672 students enrolled in 401 modules in a distance learning setting, we confirmed the existing attainment gap. After controlling for other demographics, module characteristics and engagement, BME students were between 19% and 79% less likely to complete, pass or achieve an excellent grade compared to white students. Given the same academic performance, BME students spent 4-12% more time on studying than white students. While the attainment gap remained persistent after controlling for academic engagement, our study further highlighted the inequality of attainment between BME and white students.     

The differential impact of observational learning and practice-based learning on the development of oral presentation skills in higher education

The present study focuses on the design and evaluation of an innovative instructional approach to developing oral presentation skills. The intervention builds on the observational learning theoretical perspective. This perspective is contrasted with the traditional training and practice approach. Two sequencing approaches – learners starting with observational learning versus learners starting with practice opportunities only – were compared. It was hypothesised that learners starting with observational learning would outperform learners in the practice only condition. The results suggest a significant differential impact on development of oral presentation skills. This impact of the observational learning training approach is only found in a limited number of evaluation criteria. Results additionally suggest that students are highly motivated to learn this type of skill. Interaction effects between student characteristics and instructional interventions were not significant.     

Using log variables in a learning management system to evaluate learning activity using the lens of activity theory

As the advance of learning technologies and analytics tools continues, learning management systems (LMSs) have been required to fulfil the growing expectations for smart learning. However, the reality regarding the level of technology integration in higher education differs considerably from such expectations or the speed of advances in educational technologies. This research aimed to evaluate the current activation levels and usage patterns of a LMS. A large data-set was analysed, which included the online activity information from 7940 courses. Through data pre-processing, general indicators reflecting login frequencies of the virtual campus and activity-based indicators presenting the activation patterns of diverse functions provided by Moodle were derived. Activity theory was applied to interpret the results of analysis, since it has been recognised as a powerful framework to understand phenomena encompassing interactive systems. Further, time-series investigation over three consecutive semesters allowed observation of historical changes. The results revealed considerably low use of the virtual campus with only slight changes, as well as significantly different activity patterns across course attributes and colleges. Contradictions among components in the activity system are discussed, along with the implications for improving teaching and learning with LMS in higher education.     

Courseware development with animated pedagogical agents in learning system to improve learning motivation

The addition of animated pedagogical agents (APAs) in computer-assisted learning (CAL) systems could successfully enhance students' learning motivation and engagement in learning activities. Conventionally, the APA incorporated multimedia materials are constructed through the cooperation of teachers and software programmers. However, the thinking and technical gap between two roles, a concept/content provider (teacher) and an implementer (software programmer), may lead to the concept distortion and loss in the final teaching material implementations. This often results in preventing a teacher from preparing his or her own APA-based multimedia learning systems. This study aims to develop a courseware to facilitate the CAL through the combination of the two roles into a single one and then teachers can use it freely by themselves. The courseware can help teachers to develop appropriate multimedia instructional materials incorporating APAs, which attracts students' attention and enhances their interest in learning. Furthermore, in this study two separate trial experiments were conducted to measure the teachers' satisfaction with the courseware and students' learning motivation after using APA-based multimedia learning systems. The results show that the participating teachers were satisfied with the courseware and would like to apply it in their instruction. Moreover, the participating students were strongly motivated by the customized APA-based multimedia learning systems. Therefore, the proposed courseware could give teachers a good opportunity to add pedagogical value to the curriculum, so that students' learning interest and motivation could be enhanced.     

The process,dialogues, and attitudes of vocational engineering high school students in a web problem-based learning (WPBL) system

This study aims to explore how high school students collaboratively solve problems in a web problem-based learning (WPBL) system in an 8-week digital logic course using discourse analysis. Employing in-depth interviews, this study also investigated the students' attitudes toward the WPBL system. The number of teaching assistants' responses had a negative relationship with the number of peer responses. Regarding dialogue quality, extended questions and brainstorming stimulated each other. Elaboration facilitated problem solving, whereas organizing knowledge from books or websites decreased reflection. Students affirmed the usefulness and effectiveness of the web system because it created an interactive, flexible, comfortable and collaborative learning environment.     

The impact of AI and robotics on physical,social-emotional and intellectual learning outcomes: An integrated analytical framework

This qualitative study examines the use of artificial intelligence (AI) and robotics in learning designs from the perspective of learning sciences. The literature on the topic indicates that there is not enough research on including diverse learning outcomes in the designs for learning. Therefore, the purpose of this study was to understand how AI and robots impact physical, social-emotional and intellectual learning outcomes through the implementation of learning designs that are guided by selected design principles. In this study, the design-based research (DBR) methodology was employed for investigating learning in naturalistic contexts. The intervention was implemented in a primary school in which learners used educational robots. The main findings reveal that the development of an integrated analytical framework, which considers a broader spectrum of human potential, allows for analyzing students’ learning outcomes in a more integral, inclusive and balanced way. This, in turn, promotes students’ learning by using AI and robots. Another finding reveals that the impact of using AI and robotics on learning designs is reflected in learners’ personal trajectories having different pathways and paces. Finally, the lessons learned and the challenges to be overcome are summarized, and recommendations are made for future research for the enhancement of learning experiences that use AI and robotics.     

Cultivating student skills in self-regulated learning through evaluation of task complexity

In order to self-regulate, students need to honestly reflect on their learning and to take appropriate corrective action. A simple procedure to cultivate student skills in self-regulated learning, known as the Task Evaluation and Reflection Instrument for Student Self-Assessment (TERISSA) is discussed in this paper. TERISSA guides students through two evaluations of the complexity of a task: the first is undertaken just before solving the task and the second straight after completing the task. This study involved 63 undergraduate students and observed a statistically significant difference (p = 0.007) in performance between the students who did (6.1/10) and did not (4.1/10) use TERISSA during tutorials leading up to an assessment task.     

It's story time!: exploring the potential of multimodality in oral storytelling to support children's vocabulary learning

Although many studies have been done on the benefits of parent/teacher‐child interactions during shared storybook reading or read‐aloud sessions, very few have examined the potential of professional storytellers' oral discourse to support children's vocabulary learning. In those storytelling sessions conducted by professional storytellers, the process of telling a story is typically not accompanied by a book, but only by the teller's well‐coordinated gestures, facial expressions and voice modulations. In this study, I perform a multimodal analysis of storytellers' oral discourse recorded during two storytelling sessions for four‐to‐five‐year‐old children. The study aims to (1) find out the specific types of vocal and visual features accompanying the spoken words which were unlikely to be known by the children but used by the storytellers for representations of events and characters, and (2) explore the potential of these multimodal features in oral storytelling to support children's inferring of word meanings. The study offers insights into multimodality in oral storytelling and implications for exploring the potential of multimodal features in this form of literacy practice to support children's vocabulary learning.     

Selecting a learning management system (LMS) in developing countries: instructors' evaluation

Learning management systems (LMSs) contain hidden costs, unclear user environments, bulky developer and administration manuals, and limitations with regard to interoperability, integration, localization, and bandwidth requirements. Careful evaluation is required in selecting the most appropriate LMS for use, and this is a general problem in web-based education. One possibility is to automate this evaluation process using computer-aided techniques. In this article, the researcher has focused on the idea of automating the evaluation process and has developed a computer-aided system named Easy Way to Evaluate LMSs (EW-LMS), for the quick and efficient evaluation of LMSs, especially for developing countries, where the idea of using an LMS may be relatively new. The investigation described in this article was carried out with 88 voluntary higher education instructors to test the usefulness of the developed system. Two types of quantitative questionnaires were used during the study. The results obtained indicate that the instructors were highly satisfied using the developed system, and the EW-LMS is technically sound and well developed. Theevaluation system can help instructors to select the best LMS appropriate to their educational needs and their type of usage in the shortest possible time, with little effort, without any specialized technical knowledge/training and by following simple, user-friendly steps. The system is oriented to anyone interested in evaluating and using an LMS in web-based education. Those individuals who may be interested in using this software are instructors, students, or anyone else who may have an interest in LMSs, and educational organizations such as universities, schools, and institutes.     

Solfeggio learning and the influence of a mobile application based on visual,auditory and tactile modalities

Traditional methods of learning solfeggio (music theory) generally do not take advantage of computer-based support, meaning that, when learning individually, students cannot receive instantaneous feedback on their activities. The aim of this study is to examine the effectiveness of an interactive mobile application, mySolfeggio, for learning solfeggio. Using a mobile device, students can take advantage of visual, auditory and tactile modalities to recognise musical notes. Students can also practice and learn notation, rhythm and melody, for which the mobile application provides corrective feedback. To evaluate students’ perceptions of the mobile application and its effect on knowledge, we conducted an experiment with 42 students, from 9 to 13 years old. After learning a particular song during a regular lesson, one group of students practiced it individually with only the musical notation, while the other group used both the musical notation and the mobile application. The results of the experiment illustrated only a small effect on students’ performance in singing and tapping when using the mobile application. However, they demonstrated higher scores in terms of musical intervals and rhythmic accuracy when compared to students in the control group. The students did not find the use of the application difficult, thus allowing it to be used as a tool for improvement of their homework practice.     

The effects of multimodal texts on science reading comprehension in Chilean fifth-graders: text scaffolding and comprehension skills

ABSTRACT

Reading comprehension is a crucial skill that elementary school students must develop in order to learn science. However, there is not yet enough research about the role that multimodal texts play in scaffolding student reading comprehension of complex scientific processes, such as energy transfer. This study explored how verbal and visual resources (scaffolding level) and individual differences (reading skills) contribute to science reading comprehension. One-hundred and sixty Chilean fifth-graders were assessed on reading skills, vocabulary, and prior science knowledge. A counterbalanced design was used to test two groups: Group 1 reads a text with low multimodal scaffolding and Group 2 reads a text with high multimodal scaffolding. Level of text scaffolding was determined by (1) image function, (2) visual-verbal relations, (3) presence of an explicit explanatory structure, and (4) lexico-grammatical resources. General monomodal and multimodal science reading comprehension were assessed with multiple-choice tests. An ANCOVA analysis revealed non-significant differences between groups after controlling for prior knowledge, fluency, and vocabulary. Likewise, a two-factor ANCOVA analysis showed that the high-multimodal scaffolding text significantly boosted science reading comprehension for low-skilled comprehenders. The paper discusses the implications of these findings for pedagogy and research, aiming to foster multimodal literacy for learning in content areas.     

Adaptation and evaluation of a learning analytics dashboard to improve academic support at three Latin American universities

Despite the success of academic advising dashboards in several higher educational institutions (HEI), these dashboards are still under-explored in Latin American HEI's. To close this gap, three different Latin American universities adapted an existing advising dashboard, originally deployed at the KU Leuven to their own context. In all three cases, the context was the main ruling factor to these adaptations. In this paper, we describe these adaptions using a framework that focuses on four different elements of the context: Objectives, Stakeholders, Key moment and Interactions. Evaluation of the adapted dashboards in the three different Latin American universities is conducted through pilots. This evaluation shows the value of the dashboard approach in different contexts in terms of satisfaction, usefulness and impact in academic decision-making and advising tasks. The main contribution of this paper is the systematic reporting of the adaptations to an academic advising dashboard and showing the value of an academic advising dashboard on academic decision-making and advising tasks.     

Leveraging complexity frameworks to refine theories of engagement: Advancing self-regulated learning in the age of artificial intelligence

Capturing evidence for dynamic changes in self-regulated learning (SRL) behaviours resulting from interventions is challenging for researchers. In the current study, we identified students who were likely to do poorly in a biology course and those who were likely to do well. Then, we randomly assigned a portion of the students predicted to perform poorly to a science of learning to learn intervention where they were taught SRL study strategies. Learning outcome and log data (257 K events) were collected from n = 226 students. We used a complex systems framework to model the differences in SRL including the amount, interrelatedness, density and regularity of engagement captured in digital trace data (ie, logs). Differences were compared between students who were predicted to (1) perform poorly (control, n = 48), (2) perform poorly and received intervention (treatment, n = 95) and (3) perform well (not flagged, n = 83). Results indicated that the regularity of students' engagement was predictive of course grade, and that the intervention group exhibited increased regularity in engagement over the control group immediately after the intervention and maintained that increase over the course of the semester. We discuss the implications of these findings in relation to the future of artificial intelligence and potential uses for monitoring student learning in online environments.

Practitioner notes

What is already known about this topic

• Self-regulated learning (SRL) knowledge and skills are strong predictors of postsecondary STEM student success.
• SRL is a dynamic, temporal process that leads to purposeful student engagement.
• Methods and metrics for measuring dynamic SRL behaviours in learning contexts are needed.

What this paper adds

• A Markov process for measuring dynamic SRL processes using log data.
• Evidence that dynamic, interaction-dominant aspects of SRL predict student achievement.
• Evidence that SRL processes can be meaningfully impacted through educational intervention.

Implications for theory and practice

• Complexity approaches inform theory and measurement of dynamic SRL processes.
• Static representations of dynamic SRL processes are promising learning analytics metrics.
• Engineered features of LMS usage are valuable contributions to AI models.