Human and artificial intelligence collaboration for socially shared regulation in learning

Artificial intelligence (AI) has generated a plethora of new opportunities, potential and challenges for understanding and supporting learning. In this paper, we position human and AI collaboration for socially shared regulation (SSRL) in learning. Particularly, this paper reflects on the intersection of human and AI collaboration in SSRL research, which presents an exciting prospect for advancing our understanding and support of learning regulation. Our aim is to operationalize this human-AI collaboration by introducing a novel trigger concept and a hybrid human-AI shared regulation in learning (HASRL) model. Through empirical examples that present AI affordances for SSRL research, we demonstrate how humans and AI can synergistically work together to improve learning regulation. We argue that the integration of human and AI strengths via hybrid intelligence is critical to unlocking a new era in learning sciences research. Our proposed frameworks present an opportunity for empirical evidence and innovative designs that articulate the potential for human-AI collaboration in facilitating effective SSRL in teaching and learning.

Practitioner notes

What is already known about this topic

• For collaborative learning to succeed, socially shared regulation has been acknowledged as a key factor.
• Artificial intelligence (AI) is a powerful and potentially disruptive technology that can reveal new insights to support learning.
• It is questionable whether traditional theories of how people learn are useful in the age of AI.

What this paper adds

• Introduces a trigger concept and a hybrid Human-AI Shared Regulation in Learning (HASRL) model to offer insights into how the human-AI collaboration could occur to operationalize SSRL research.
• Demonstrates the potential use of AI to advance research and practice on socially shared regulation of learning.
• Provides clear suggestions for future human-AI collaboration in learning and teaching aiming at enhancing human learning and regulatory skills.

Implications for practice and/or policy

• Educational technology developers could utilize our proposed framework to better align technological and theoretical aspects for their design of adaptive support that can facilitate students' socially shared regulation of learning.
• Researchers and practitioners could benefit from methodological development incorporating human-AI collaboration for capturing, processing and analysing multimodal data to examine and support learning regulation.

     

Assessing approaches to learning with nonparametric multidimensional scaling

This article reports on a trace-based assessment of approaches to learning used by middle school aged children who interacted with NASA Mars Mission science, technology, engineering and mathematics (STEM) games in Whyville, an online game environment with 8 million registered young learners. The learning objectives of two games included awareness and knowledge of NASA missions, developing knowledge and skills of measurement and scaling, applying measurement for planetary comparisons in the solar system. Trace data from 1361 interactions were analysed with nonparametric multidimensional scaling methods, which permitted visual examination and statistical validation, and provided an example and proof of concept for the multidimensional scaling approach to analysis of time-based behavioural data from a game or simulation. Differences in approach to learning were found illustrating the potential value of the methodology to curriculum and game-based learning designers as well as other creators of online STEM content for pre-college youth. The theoretical framework of the method and analysis makes use of the Epistemic Network Analysis toolkit as a post hoc data exploration platform, and the discussion centres on issues of semantic interpretation of interaction end-states and the application of evidence centred design in post hoc analysis.

Practitioner notes

What is already known about this topic

• Educational game play has been demonstrated to positively affect learning performance and learning persistence.
• Trace-based assessment from digital learning environments can focus on learning outcomes and processes drawn from user behaviour and contextual data.
• Existing approaches used in learning analytics do not (fully) meet criteria commonly used in psychometrics or for different forms of validity in assessment, even though some consider learning analytics a form of assessment in the broadest sense.
• Frameworks of knowledge representation in trace-based research often include concepts from cognitive psychology, education and cognitive science.

What this paper adds

• To assess skills-in-action, stronger connections of learning analytics with educational measurement can include parametric and nonparametric statistics integrated with theory-driven modelling and semantic network analysis approaches widening the basis for inferences, validity, meaning and understanding from digital traces.
• An expanded methodological foundation is offered for analysis in which nonparametric multidimensional scaling, multimodal analysis, epistemic network analysis and evidence-centred design are combined.

Implications for practice and policy

• The new foundations are suggested as a principled, theory-driven, embedded data collection and analysis framework that provides structure for reverse engineering of semantics as well as pre-planning frameworks that support creative freedom in the processes of creation of digital learning environments.

     

Analytics-supported reflective assessment for 6th graders' knowledge building and data science practices: An exploratory study

Preparing data-literate citizens and supporting future generations to effectively work with data is challenging. Engaging students in Knowledge Building (KB) may be a promising way to respond to this challenge because it requires students to reflect on and direct their inquiry with the support of data. Informed by previous studies, this research explored how an analytics-supported reflective assessment (AsRA)-enhanced KB design influenced 6th graders' KB and data science practices in a science education setting. One intact class with 56 students participated in this study. The analysis of students' Knowledge Forum discourse showed the positive influences of the AsRA-enhanced KB design on students' development of KB and data science practices. Further analysis of different-performing groups revealed that the AsRA-enhanced KB design was accessible to all performing groups. These findings have important implications for teachers and researchers who aim to develop students' KB and data science practices, and general high-level collaborative inquiry skills.

Practitioner notes

What is already known about this topic

• Data use becomes increasingly important in the K-12 educational context.
• Little is known about how to scaffold students to develop data science practices.
• Knowledge Building (KB) and learning analytics-supported reflective assessment (AsRA) show premises in developing these practices.

What this paper adds

• AsRA-enhanced KB can help students improve KB and data science practices over time.
• AsRA-enhanced KB design benefits students of different-performing groups.
• AsRA-enhanced KB is accessible to elementary school students in science education.

Implications for practice and/or policy

• Developing a collaborative and reflective culture helps students engage in collaborative inquiry.
• Pedagogical approaches and analytic tools can be developed to support students' data-driven decision-making in inquiry learning.

     

Formative assessment strategies for students' conceptions—The potential of learning analytics

Formative assessment is considered to be helpful in students' learning support and teaching design. Following Aufschnaiter's and Alonzo's framework, formative assessment practices of teachers can be subdivided into three practices: eliciting evidence, interpreting evidence and responding. Since students' conceptions are judged to be important for meaningful learning across disciplines, teachers are required to assess their students' conceptions. The focus of this article lies on the discussion of learning analytics for supporting the assessment of students' conceptions in class. The existing and potential contributions of learning analytics are discussed related to the named formative assessment framework in order to enhance the teachers' options to consider individual students' conceptions. We refer to findings from biology and computer science education on existing assessment tools and identify limitations and potentials with respect to the assessment of students' conceptions.

Practitioner notes

What is already known about this topic

• Students' conceptions are considered to be important for learning processes, but interpreting evidence for learning with respect to students' conceptions is challenging for teachers.
• Assessment tools have been developed in different educational domains for teaching practice.
• Techniques from artificial intelligence and machine learning have been applied for automated assessment of specific aspects of learning.

What does the paper add

• Findings on existing assessment tools from two educational domains are summarised and limitations with respect to assessment of students' conceptions are identified.
• Relevent data that needs to be analysed for insights into students' conceptions is identified from an educational perspective.
• Potential contributions of learning analytics to support the challenging task to elicit students' conceptions are discussed.

Implications for practice and/or policy

• Learning analytics can enhance the eliciting of students' conceptions.
• Based on the analysis of existing works, further exploration and developments of analysis techniques for unstructured text and multimodal data are desirable to support the eliciting of students' conceptions.

     

The role of learning theory in multimodal learning analytics

This study presents the outcomes of a semi-systematic literature review on the role of learning theory in multimodal learning analytics (MMLA) research. Based on previous systematic literature reviews in MMLA and an additional new search, 35 MMLA works were identified that use theory. The results show that MMLA studies do not always discuss their findings within an established theoretical framework. Most of the theory-driven MMLA studies are positioned in the cognitive and affective domains, and the three most frequently used theories are embodied cognition, cognitive load theory and control–value theory of achievement emotions. Often, the theories are only used to inform the study design, but there is a relationship between the most frequently used theories and the data modalities used to operationalize those theories. Although studies such as these are rare, the findings indicate that MMLA affordances can, indeed, lead to theoretical contributions to learning sciences. In this work, we discuss methods of accelerating theory-driven MMLA research and how this acceleration can extend or even create new theoretical knowledge.

Practitioner notes

What is already known about this topic

• Multimodal learning analytics (MMLA) is an emerging field of research with inherent connections to advanced computational analyses of social phenomena.
• MMLA can help us monitor learning activity at the micro-level and model cognitive, affective and social factors associated with learning using data from both physical and digital spaces.
• MMLA provide new opportunities to support students' learning.

What this paper adds

• Some MMLA works use theory, but, overall, the role of theory is currently limited.
• The three theories dominating MMLA research are embodied cognition, control–value theory of achievement emotions and cognitive load theory.
• Most of the theory-driven MMLA papers use theory ‘as is’ and do not consider the analytical and synthetic role of theory or aim to contribute to it.

Implications for practice and/or policy

• If the ultimate goal of MMLA, and AI in Education in general, research is to understand and support human learning, these studies should be expected to align their findings (or not) with established relevant theories.
• MMLA research is mature enough to contribute to learning theory, and more research should aim to do so.
• MMLA researchers and practitioners, including technology designers, developers, educators and policy-makers, can use this review as an overview of the current state of theory-driven MMLA.

     

Understanding privacy and data protection issues in learning analytics using a systematic review

The field of learning analytics has advanced from infancy stages into a more practical domain, where tangible solutions are being implemented. Nevertheless, the field has encountered numerous privacy and data protection issues that have garnered significant and growing attention. In this systematic review, four databases were searched concerning privacy and data protection issues of learning analytics. A final corpus of 47 papers published in top educational technology journals was selected after running an eligibility check. An analysis of the final corpus was carried out to answer the following three research questions: (1) What are the privacy and data protection issues in learning analytics? (2) What are the similarities and differences between the views of stakeholders from different backgrounds on privacy and data protection issues in learning analytics? (3) How have previous approaches attempted to address privacy and data protection issues? The results of the systematic review show that there are eight distinct, intertwined privacy and data protection issues that cut across the learning analytics cycle. There are both cross-regional similarities and three sets of differences in stakeholder perceptions towards privacy and data protection in learning analytics. With regard to previous attempts to approach privacy and data protection issues in learning analytics, there is a notable dearth of applied evidence, which impedes the assessment of their effectiveness. The findings of our paper suggest that privacy and data protection issues should not be relaxed at any point in the implementation of learning analytics, as these issues persist throughout the learning analytics development cycle. One key implication of this review suggests that solutions to privacy and data protection issues in learning analytics should be more evidence-based, thereby increasing the trustworthiness of learning analytics and its usefulness.

Practitioner notes

What is already known about this topic

• Research on privacy and data protection in learning analytics has become a recognised challenge that hinders the further expansion of learning analytics.
• Proposals to counter the privacy and data protection issues in learning analytics are blurry; there is a lack of a summary of previously proposed solutions.

What this study contributes

• Establishment of what privacy and data protection issues exist at different phases of the learning analytics cycle.
• Identification of how different stakeholders view privacy, similarities and differences, and what factors influence their views.
• Evaluation and comparison of previously proposed solutions that attempt to address privacy and data protection in learning analytics.

Implications for practice and/or policy

• Privacy and data protection issues need to be viewed in the context of the entire cycle of learning analytics.
• Stakeholder views on privacy and data protection in learning analytics have commonalities across contexts and differences that can arise within the same context. Before implementing learning analytics, targeted research should be conducted with stakeholders.
• Solutions that attempt to address privacy and data protection issues in learning analytics should be put into practice as far as possible to better test their usefulness.

     

Contextualizing privacy with wearable data in higher education

This conceptual study uses dynamic systems theory (DST) and phenomenology as lenses to examine data privacy implications surrounding wearable devices that incorporate stakeholder, contextual and technical factors. Wearable devices can impact people's behaviour and sense of self, and DST and phenomenology provide complementary approaches for emphasizing the subjective experiences of individuals that occur with the use of wearable data. Privacy is approached through phenomenology as an individual's lived bodily experience and DST emphasizes the self-regulation and feedback loops of individuals and their uses of wearable data. The data collection, analysis and communication of wearable data to support learning systems alongside privacy implications for each are examined. The IoT, cloud computing, metadata and algorithms are discussed as they relate to wearable data, pointing out privacy risks and strategies to minimize harm.

Practitioner notes

What is already known about this topic

• Data privacy is a complex topic and is approached through different perspectives, influencing the degree of an individual's data autonomy.
• Wearable technology is increasing in the consumer market and offers great potential to learning environments.

What this paper adds

• Extends extant literature on dynamic systems theory and phenomenology, contributing these perspectives to educational research in the context of student data privacy and wearable technologies.
• Provides a framework to understand the complex and contingent ways that privacy can be understood in the collection, analysis, and communication of wearable data to support learning.

Implications for practice and/or policy

• Higher education faculty and educational policymakers should consider various interactions in systems and among systems of how wearable data collection may be analysed, communicated and stored, potentially exposing students to privacy harms.
• Multiple actors in learning systems must engage in continuous and evolving feedback loops around data security, consent, ownership and control to determine who has access to student data, how it is used and for what purposes.
• The EU's General Data Protection and Regulation offers one of the most comprehensive frameworks for higher education institutions and faculty around the world to follow for protecting student data privacy.

     

How pedagogical relations in early years settings are reconfigured by interactive touchscreens

While interactive touchscreens are currently entering into educational practice, little is known about what this means for learning in early childhood and, in particular, how touchscreens shape action and communication. In this paper, we examine the interactions of 2-year-olds and their teachers in a multilingual preschool in Sweden. We analyse the communicative environment between the children, teachers and shared touchscreens and books in the context of reading. A mixed-methods analysis was used, taking a concept of action that includes both verbal, non-verbal utterances and digital touch. The analysis shows a reconfiguration to the interactional dynamic where children perform comparable amounts of actions in sessions with the touchscreen and book reading but less talk during the touchscreen sessions. However, while talking less, children display other types of communicative actions. We analyse the changing interactional dynamic that follows, its implications to learning and early childhood pedagogical practice and how interaction can be reconceptualised as cycles of communication and action in which educational scaffolding unfolds.

Practitioner notes

What is already known about this topic

• Touchscreens are a significant part of children's lives and educational curricula.
• There is considerable uncertainty on how touchscreens can be incorporated into early childhood education.
• Little is known about how educational social interaction changes with touchscreens such as iPads.

What this paper adds

• A mixed methods multimodal analysis of the changing actions and dynamics of iPads as compared with bookreading.
• Children's patterns of communication change towards less talk and more bodily communication, while teachers’ actions remain somewhat similar.
• Touch actions change the dynamics of interaction, can alter the pedagogical situation and bring a reconceptualisation towards a cyclical and embodied view of interaction.

Implications for practice and/or policy

• New patterns of action may require a recalibration of educational practices.
• Teachers need to attend to new sets of touch actions that children use to communicate and act with as displays of knowledge.
• The use of touch screens should be seen as complementary to established practices of language and literacy training (such as book reading) rather than replacing them.

     

Pedagogical features of interactive apps for effective learning of foundational skills

Interactive apps are commonly used to support the acquisition of foundational skills. Yet little is known about how pedagogical features of such apps affect learning outcomes, attainment and motivation—particularly when deployed in lower-income contexts, where educational gains are most needed. In this study, we analyse which app features are most effective in supporting the acquisition of foundational literacy and numeracy skills. We compare five apps developed for the Global Learning XPRIZE and deployed to 2041 out-of-school children in 172 remote Tanzanian villages. A total of 41 non-expert participants each provided 165 comparative judgements of the five apps from the competition, across 15 pedagogical features. Analysis and modelling of these 6765 comparisons indicate that the apps created by the joint winners of the XPRIZE, who produced the greatest learning outcomes over the 15-month field trial, shared six pedagogical features—autonomous learning, motor skills, task structure, engagement, language demand and personalisation. Results demonstrate that this combination of features is effective at supporting learning of foundational skills and has a positive impact on educational outcomes. To maximise learning potential in environments with both limited resources and deployment opportunities, developers should focus attention on this combination of features, especially for out-of-school children in low- and middle-income countries.

Practitioner notes

What is already known about this topic

• Interactive apps are becoming common to support foundational learning for children both in and out of school settings.
• The Global Learning XPRIZE competition demonstrates that learning apps can facilitate learning improvements in out-of-school children living in sub-Saharan Africa.
• To understand which app features are most important in supporting learning in these contexts, we need to establish which pedagogical features were shared by the winning apps.

What this paper adds

• Effective learning of foundational skills can be achieved with a range of pedagogical features.
• To maximise learning, apps should focus on combining elements of autonomous learning, motor skills, task structure, engagement, language demand and personalisation.
• Free Play is not a key pedagogical feature to facilitate learning within this context.

Implications for practice and/or policy

• When developing learning apps with primary-aged, out-of-school children in low-income contexts, app developers should try to incorporate the six key features associated with improving learning outcomes.
• Governments, school leaders and parents should use these findings to inform their decisions when choosing an appropriate learning app for children.

     

Recognizing socially shared regulation by using the temporal sequences of online chat and logs in CSCL

The increasing amount of empirical research shows that the role of regulatory processes is critical in CSCL and collaborative learning settings. However, the current conceptual definitions and specificity of the findings vary. This is most probably because of limitations in the methods investigating regulated learning in a collaborative learning context. This study aimed to provide empirical evidence for how self- and shared regulation activities are used and whether they are useful for collaborative learning outcomes. Eighteen graduate students worked in collaborative groups for seven weeks in a CSCL course and the data of this study focuses on three one week online collaborative learning phases in the course. Temporal and sequential analysis of chat discussions and log file traces were matched to find evidence about whether the students' collaboratively planned regulatory activities became shared in practice. The results show evidence that collaborative planned regulatory activities become shared in practice. The groups that achieved good learning results used multiple regulatory processes to support their learning and also reached shared regulation. The four microlevel examples demonstrate simplified patterns of the activation of self-regulation and shared regulation. In conclusion, individual socially shared regulation plays a critical role in successful collaborative learning.     

Using causal models to bridge the divide between big data and educational theory

An extraordinary amount of data is becoming available in educational settings, collected from a wide range of Educational Technology tools and services. This creates opportunities for using methods from Artificial Intelligence and Learning Analytics (LA) to improve learning and the environments in which it occurs. And yet, analytics results produced using these methods often fail to link to theoretical concepts from the learning sciences, making them difficult for educators to trust, interpret and act upon. At the same time, many of our educational theories are difficult to formalise into testable models that link to educational data. New methodologies are required to formalise the bridge between big data and educational theory. This paper demonstrates how causal modelling can help to close this gap. It introduces the apparatus of causal modelling, and shows how it can be applied to well-known problems in LA to yield new insights. We conclude with a consideration of what causal modelling adds to the theory-versus-data debate in education, and extend an invitation to other investigators to join this exciting programme of research.

Practitioner notes

What is already known about this topic

• ‘Correlation does not equal causation’ is a familiar claim in many fields of research but increasingly we see the need for a causal understanding of our educational systems.
• Big data bring many opportunities for analysis in education, but also a risk that results will fail to replicate in new contexts.
• Causal inference is a well-developed approach for extracting causal relationships from data, but is yet to become widely used in the learning sciences.

What this paper adds

• An overview of causal modelling to support educational data scientists interested in adopting this promising approach.
• A demonstration of how constructing causal models forces us to more explicitly specify the claims of educational theories.
• An understanding of how we can link educational datasets to theoretical constructs represented as causal models so formulating empirical tests of the educational theories that they represent.

Implications for practice and/or policy

• Causal models can help us to explicitly specify educational theories in a testable format.
• It is sometimes possible to make causal inferences from educational data if we understand our system well enough to construct a sufficiently explicit theoretical model.
• Learning Analysts should work to specify more causal models and test their predictions, as this would advance our theoretical understanding of many educational systems.

     

Self-regulatory and demographic predictors of grades in online and face-to-face university cohorts: A multi-group path analysis

Predictors of academic success at university are of great interest to educators, researchers and policymakers. With more students studying online, it is important to understand whether traditional predictors of academic outcomes in face-to-face settings are relevant to online learning. This study modelled self-regulatory and demographic predictors of subject grades in 84 online and 80 face-to-face undergraduate students. Predictors were effort regulation, grade goal, academic self-efficacy, performance self-efficacy, age, sex, socio-economic status (SES) and first-in-family status. A multi-group path analysis indicated that the models were significantly different across learning modalities. For face-to-face students, none of the model variables significantly predicted grades. For online students, only performance self-efficacy significantly predicted grades (small effect). Findings suggest that learner characteristics may not function in the same way across learning modes. Further factor analytic and hierarchical research is needed to determine whether self-regulatory predictors of academic success continue to be relevant to modern student cohorts.

Practitioner Notes

What is already known about this topic

• Self-regulatory and demographic variables are important predictors of university outcomes like grades.
• It is unclear whether the relationships between predictor variables and outcomes are the same across learning modalities, as research findings are mixed.

What this paper adds

• Models predicting university students' grades by demographic and self-regulatory predictors differed significantly between face-to-face and online learning modalities.
• Performance self-efficacy significantly predicted grades for online students.
• No self-regulatory variables significantly predicted grades for face-to-face students, and no demographic variables significantly predicted grades in either cohort.
• Overall, traditional predictors of grades showed no/small unique effects in both cohorts.

Implications for practice and/or policy

• The learner characteristics that predict success may not be the same across learning modalities.
• Approaches to enhancing success in face-to-face settings are not automatically applicable to online settings.
• Self-regulatory variables may not predict university outcomes as strongly as previously believed, and more research is needed.

     

Leveraging complexity frameworks to refine theories of engagement: Advancing self-regulated learning in the age of artificial intelligence

Capturing evidence for dynamic changes in self-regulated learning (SRL) behaviours resulting from interventions is challenging for researchers. In the current study, we identified students who were likely to do poorly in a biology course and those who were likely to do well. Then, we randomly assigned a portion of the students predicted to perform poorly to a science of learning to learn intervention where they were taught SRL study strategies. Learning outcome and log data (257 K events) were collected from n = 226 students. We used a complex systems framework to model the differences in SRL including the amount, interrelatedness, density and regularity of engagement captured in digital trace data (ie, logs). Differences were compared between students who were predicted to (1) perform poorly (control, n = 48), (2) perform poorly and received intervention (treatment, n = 95) and (3) perform well (not flagged, n = 83). Results indicated that the regularity of students' engagement was predictive of course grade, and that the intervention group exhibited increased regularity in engagement over the control group immediately after the intervention and maintained that increase over the course of the semester. We discuss the implications of these findings in relation to the future of artificial intelligence and potential uses for monitoring student learning in online environments.

Practitioner notes

What is already known about this topic

• Self-regulated learning (SRL) knowledge and skills are strong predictors of postsecondary STEM student success.
• SRL is a dynamic, temporal process that leads to purposeful student engagement.
• Methods and metrics for measuring dynamic SRL behaviours in learning contexts are needed.

What this paper adds

• A Markov process for measuring dynamic SRL processes using log data.
• Evidence that dynamic, interaction-dominant aspects of SRL predict student achievement.
• Evidence that SRL processes can be meaningfully impacted through educational intervention.

Implications for theory and practice

• Complexity approaches inform theory and measurement of dynamic SRL processes.
• Static representations of dynamic SRL processes are promising learning analytics metrics.
• Engineered features of LMS usage are valuable contributions to AI models.

     

A systematic review of empirical studies using log data from open-ended learning environments to measure science and engineering practices

Technology-based, open-ended learning environments (OELEs) can capture detailed information of students' interactions as they work through a task or solve a problem embedded in the environment. This information, in the form of log data, has the potential to provide important insights about the practices adopted by students for scientific inquiry and problem solving. How to parse and analyse the log data to reveal evidence of multifaceted constructs like inquiry and problem solving holds the key to making interactive learning environments useful for assessing students' higher-order competencies. In this paper, we present a systematic review of studies that used log data generated in OELEs to describe, model and assess scientific inquiry and problem solving. We identify and analyse 70 conference proceedings and journal papers published between 2012 and 2021. Our results reveal large variations in OELE and task characteristics, approaches used to extract features from log data and interpretation models used to link features to target constructs. While the educational data mining and learning analytics communities have made progress in leveraging log data to model inquiry and problem solving, multiple barriers still exist to hamper the production of representative, reproducible and generalizable results. Based on the trends identified, we lay out a set of recommendations pertaining to key aspects of the workflow that we believe will help the field develop more systematic approaches to designing and using OELEs for studying how students engage in inquiry and problem-solving practices.

Practitioner notes

What is already known about this topic

• Research has shown that technology-based, open-ended learning environments (OELEs) that collect users' interaction data are potentially useful tools for engaging students in practice-based STEM learning.
• More work is needed to identify generalizable principles of how to design OELE tasks to support student learning and how to analyse the log data to assess student performance.

What this paper adds

• We identified multiple barriers to the production of sufficiently generalizable and robust results to inform practice, with respect to: (1) the design characteristics of the OELE-based tasks, (2) the target competencies measured, (3) the approaches and techniques used to extract features from log files and (4) the models used to link features to the competencies.
• Based on this analysis, we can provide a series of specific recommendations to inform future research and facilitate the generalizability and interpretability of results:
  • Making the data available in open-access repositories, similar to the PISA tasks, for easy access and sharing.
  • Defining target practices more precisely to better align task design with target practices and to facilitate between-study comparisons.
  • More systematic evaluation of OELE and task designs to improve the psychometric properties of OELE-based measurement tasks and analysis processes.
  • Focusing more on internal and external validation of both feature generation processes and statistical models, for example with data from different samples or by systematically varying the analysis methods.

Implications for practice and/or policy

• Using the framework of evidence-centered assessment design, we have identified relevant criteria for organizing and evaluating the diverse body of empirical studies on the topic and that policy makers and practitioners can use for their own further examinations.
• This paper identifies promising research and development areas on the measurement and assessment of higher-order constructs with process data from OELE-based tasks that government agencies and foundations can support.
• Researchers, technologists and assessment designers might find useful the insights and recommendations for how OELEs can enhance science assessment through thoughtful integration of learning theories, task design and data mining techniques.

     

Learning analytics application to examine validity and generalizability of game-based assessment for spatial reasoning

Game-based assessment (GBA), a specific application of games for learning, has been recognized as an alternative form of assessment. While there is a substantive body of literature that supports the educational benefits of GBA, limited work investigates the validity and generalizability of such systems. In this paper, we describe applications of learning analytics methods to provide evidence for psychometric qualities of a digital GBA called Shadowspect, particularly to what extent Shadowspect is a robust assessment tool for middle school students' spatial reasoning skills. Our findings indicate that Shadowspect is a valid assessment for spatial reasoning skills, and it has comparable precision for both male and female students. In addition, students' enjoyment of the game is positively related to their overall competency as measured by the game regardless of the level of their existing spatial reasoning skills.

Practitioner notes

What is already known about this topic:

• Digital games can be a powerful context to support and assess student learning.
• Games as assessments need to meet certain psychometric qualities such as validity and generalizability.
• Learning analytics provide useful ways to establish assessment models for educational games, as well as to investigate their psychometric qualities.

What this paper adds:

• How a digital game can be coupled with learning analytics practices to assess spatial reasoning skills.
• How to evaluate psychometric qualities of game-based assessment using learning analytics techniques.
• Investigation of validity and generalizability of game-based assessment for spatial reasoning skills and the interplay of the game-based assessment with enjoyment.

Implications for practice and/or policy:

• Game-based assessments that incorporate learning analytics can be used as an alternative to pencil-and-paper tests to measure cognitive skills such as spatial reasoning.
• More training and assessment of spatial reasoning embedded in games can motivate students who might not be on the STEM tracks, thus broadening participation in STEM.
• Game-based learning and assessment researchers should consider possible factors that affect how certain populations of students enjoy educational games, so it does not further marginalize specific student populations.

     

How student perceptions about online learning difficulty influenced their satisfaction during Canada's Covid-19 response

The COVID-19 pandemic has posed a significant challenge to higher education and forced academic institutions across the globe to abruptly shift to remote teaching. Because of the emergent transition, higher education institutions continuously face difficulties in creating satisfactory online learning experiences that adhere to the new norms. This study investigates the transition to online learning during Covid-19 to identify factors that influenced students' satisfaction with the online learning environment. Adopting a mixed-method design, we find that students' experience with online learning can be negatively affected by information overload, and perceived technical skill requirements, and describe qualitative evidence that suggest a lack of social interactions, class format, and ambiguous communication also affected perceived learning. This study suggests that to digitalize higher education successfully, institutions need to redesign students' learning experience systematically and re-evaluate traditional pedagogical approaches in the online context.

Practitioner notes

What is already known about this topic

• University transitions to online learning during the Covid-19 pandemic were undertaken by faculty and students who had little online learning experience.
• The transition to online learning was often described as having a negative influence on students' learning experience and mental health.
• Varieties of cognitive load are known predictors of effective online learning experiences and satisfaction.

What this paper adds

• Information overload and perceptions of technical abilities are demonstrated to predict students' difficulty and satisfaction with online learning.
• Students express negative attitudes towards factors that influence information overload, technical factors, and asynchronous course formats.
• Communication quantity was not found to be a significant factor in predicting either perceived difficulty or negative attitudes.

Implications for practice and/or policy

• We identify ways that educators in higher education can improve their online offerings and implementations during future disruptions.
• We offer insights into student experience concerning online learning environments during an abrupt transition.
• We identify design factors that contribute to effective online delivery, educators in higher education can improve students' learning experiences during difficult periods and abrupt transitions to online learning.

     

Algorithmic versus teacher-led sequencing in a digital serious game and the development of second language reading fluency and accuracy

Recent years have seen a surge of calls for personalization of education. Automatised adaptivity in serious games has been advocated as a potential instantiation of such calls. Yet little is known about the extent to which personalised learning through automatised adaptivity poses an advantage for language learning over generalised teacher-led sequencing in digital, game-based learning environments. The goal of this paper is to address this question by comparing the learning outcomes in reading accuracy and fluency of didactic sequences designed by EFL teachers or by an adaptive algorithm. A total of 67 participants completed several proficiency and reading skills pretest and posttest and used the iRead system for 6 months. Results showed that all learners made progress in reading skills, but no significant differences were found between the two sequences in relation to the development of reading skills. It was also shown that adaptivity works best if it leads to increase in the number of games per feature. Results are discussed in the context of previous findings, and the role of adaptivity and sequencing is critically assessed.

Practitioner notes

What is already known about this topic?

• Serious games have the potential to aid learning but empirical research is needed.
• Findings about the efficiency of serious games are mixed.
• Current and reviewed versions of the Simple View of Reading constitute a suitable framework to measure reading acquisition.

What this paper adds?

• It contributes to the growing corpus of research on digital serious games.
• It provides empirical evidence on the use of an adaptive system in formal education.
• Comparing a teacher-led sequence to an algorithmic adaptive sequence on the same digital serious game has never been done before.
• The paper shows the need to obtain both system-internal and system-external data in order to capture the impact of gameplay on the development of L2 reading skills.

Implications for practise and/or policy

• It sheds some light on how certain game designs may actually help practise with different degrees of intervention by teachers.
• It is interesting for teachers to use an adaptive sequence that they can check and intervene in if needed.

     

Drivers and barriers to the utilisation of gamification and game-based learning in universities: A systematic review of educators' perspectives

While gamification and game-based learning have both been demonstrated to have a host of educational benefits for university students, many university educators do not routinely use these approaches in their teaching. Therefore, this systematic review, conducted using the PRISMA guidelines, sought to identify the primary drivers and barriers to the use of gamification and game-based learning by university educators. A search of multiple databases (Web of Science, Scopus and EBSCO (Business Source Complete; ERIC; Library, Information Science & Technology Abstracts)) identified 1330 articles, with 1096 retained after duplicates were removed. Seventeen articles (11 quantitative, two mixed-methods and four qualitative) were included in the systematic review. The primary drivers described by the educators that positively influenced their gamification and game-based learning usage were their beliefs that it encourages student interactions and collaborative learning; provides fun and improves engagement; and can easily be used by students. Alternatively, the university educators' major barriers included a lack of time to develop gamification approaches, lack of proven benefits and classroom setting issues. Many of these and other less commonly reported drivers and barriers can be categorised as attitudinal, design-related or administrative in nature. Such categorisations may assist university educators, teaching support staff and administrators in better understanding the primary factors influencing the utilisation of gamification and game-based learning and develop more effective strategies to overcome these barriers to its successful implementation.

Practitioner notes

What is already known about this topic

• Gamification and game-based learning may have many benefits for university students.
• The majority of university educators do not routinely use gamification and game-based learning in their teaching.

What this paper adds

• University educators' major drivers that positively influence the use of gamification and game-based learning include their perceptions that it encourages student interactions and collaborative learning, provides fun and improves engagement and can easily be used by students.
• University educators' major barriers that negatively influence the use of gamification and game-based learning include their perceptions of a lack of time to develop gamification approaches, lack of proven benefits and classroom setting issues.
• These drivers and barriers may be classified as attitudinal, design-related and administrative, with these categories providing a useful way for universities to develop strategies to better support educators who wish to use these approaches in their teaching.

Implications for practice and policy

• Attitudinal factors such as university educators' intention to use gamification and game-based learning are influenced by a host of their perceptions including attitude, perceived usefulness and ease of use.
• A range of design-related and administrative barriers may need to be overcome to increase the use of gamification and game-based learning in the university sector.

     

Understanding how educational maths apps can enhance learning: A content analysis and qualitative comparative analysis

Educational applications (apps) are ubiquitous within children's learning environments and emerging evidence has demonstrated their efficacy. However, it remains unclear what the active ingredients (ie, mechanisms), or combination of ingredients, of successful maths apps are. The current study developed a new, open-access, three-step framework for assessing the educational value of maths apps, comprised of type of app, mathematical content and app design features. When applied to a selection of available maths apps previously evaluated with children in the first 3 years of school (the final sample included 23 apps), results showed that practice-based apps were the most common app type tested (n = 15). Basic number skills, such as number representation and relationships, were the most common area of mathematics targeted by apps (n = 21). A follow-up qualitative comparative analysis showed observed learning outcomes with maths apps were enhanced when apps combined the following: a scaffolded and personalised learning journey (programmatic levelling) and explanations of why answers were right or wrong (explanatory feedback), as well as praise, such as ‘Great job!’ (motivational feedback). This novel evidence stresses the significance of feedback and levelling design features that teaching practitioners and other stakeholders should consider when deciding which apps to use with young children. Directions for future research are discussed.

Practitioner notes

What is already known about this topic

• Educational apps have been shown to support maths attainment in the first 3 years of school.
• Several existing frameworks have attempted to assess the educational value of some of these maths apps.
• Emerging experimental evidence also demonstrates the benefits of specific app design features, including feedback and levelling.

What this paper adds

• Practice-based maths apps are the most common type of app previously evaluated with young children.
• These evaluated maths apps have mostly focused on basic number skills.
• The combination of explanatory and motivational feedback, with programmatic levelling (either dynamic or static), was a necessary condition for enhancing learning outcomes with maths apps.

Implications for practice and policy

• The inclusion of feedback and levelling in maths apps should be considered by app developers when designing apps, and by educational practitioners and parents when deciding which apps to use with their children.
• Further consideration is also needed for the development of educational apps that include a broad range of maths skills.