Automated monitoring of online news accuracy with change classification models

In the past decade, news consumption has shifted from printed news media to online alternatives. Although these come with advantages, online news poses challenges as well. Notable here is the increased competition between online newspapers and other online news providers to attract readers. Hereby, speed is often favored over quality. As a consequence, the need for new tools to monitor online news accuracy has grown. In this work, a fundamentally new and automated procedure for the monitoring of online news accuracy is proposed. The approach relies on the fact that online news articles are often updated after initial publication, thereby also correcting errors. Automated observation of the changes being made to online articles and detection of the errors that are corrected may offer useful insights concerning news accuracy. The potential of the presented automated error correction detection model is illustrated by building supervised classification models for the detection of objective, subjective and linguistic errors in online news updates respectively. The models are built using a large news update data set being collected during two consecutive years for six different Flemish online newspapers. A subset of 21,129 changes is then annotated using a combination of automated and human annotation via an online annotation platform. Finally, manually crafted features and text embeddings obtained by four different language models (TF-IDF, word2vec, BERTje and SBERT) are fed to three supervised machine learning algorithms (logistic regression, support vector machines and decision trees) and performance of the obtained models is subsequently evaluated. Results indicate that small differences in performance exist between the different learning algorithms and language models. Using the best-performing models, $F_2$-scores of 0.45, 0.25 and 0.80 are obtained for the classification of objective, subjective and linguistic errors respectively.     

Learning theories for artificial intelligence promoting learning processes

This paper discusses a three-level model that synthesizes and unifies existing learning theories to model the roles of artificial intelligence (AI) in promoting learning processes. The model, drawn from developmental psychology, computational biology, instructional design, cognitive science, complexity and sociocultural theory, includes a causal learning mechanism that explains how learning occurs and works across micro, meso and macro levels. The model also explains how information gained through learning is aggregated, or brought together, as well as dissipated, or released and used within and across the levels. Fourteen roles for AI in education are proposed, aligned with the model's features: four roles at the individual or micro level, four roles at the meso level of teams and knowledge communities and six roles at the macro level of cultural historical activity. Implications for research and practice, evaluation criteria and a discussion of limitations are included. Armed with the proposed model, AI developers can focus their work with learning designers, researchers and practitioners to leverage the proposed roles to improve individual learning, team performance and building knowledge communities.

Practitioner notes

What is already known about this topic

• Numerous learning theories exist with significant cross-over of concepts, duplication and redundancy in terms and structure that offer partial explanations of learning.
• Frameworks concerning learning have been offered from several disciplines such as psychology, biology and computer science but have rarely been integrated or unified.
• Rethinking learning theory for the age of artificial intelligence (AI) is needed to incorporate computational resources and capabilities into both theory and educational practices.

What this paper adds

• A three-level theory (ie, micro, meso and macro) of learning that synthesizes and unifies existing theories is proposed to enhance computational modelling and further develop the roles of AI in education.
• A causal model of learning is defined, drawing from developmental psychology, computational biology, instructional design, cognitive science and sociocultural theory, which explains how learning occurs and works across the levels.
• The model explains how information gained through learning is aggregated, or brought together, as well as dissipated, or released and used within and across the levels.
• Fourteen roles for AI in education are aligned with the model's features: four roles at the individual or micro level, four roles at the meso level of teams and knowledge communities and six roles at the macro level of cultural historical activity.

Implications for practice and policy

• Researchers may benefit from referring to the new theory to situate their work as part of a larger context of the evolution and complexity of individual and organizational learning and learning systems.
• Mechanisms newly discovered and explained by future researchers may be better understood as contributions to a common framework unifying the scientific understanding of learning theory.

     

Early prediction of student knowledge in game-based learning with distributed representations of assessment questions

Game-based learning environments hold significant promise for facilitating learning experiences that are both effective and engaging. To support individualised learning and support proactive scaffolding when students are struggling, game-based learning environments should be able to accurately predict student knowledge at early points in students' gameplay. Student knowledge is traditionally assessed prior to and after each student interacts with the learning environment with conventional methods, such as multiple choice content knowledge assessments. While previous student modelling approaches have leveraged machine learning to automatically infer students' knowledge, there is limited work that incorporates the fine-grained content from each question in these types of tests into student models that predict student performance at early junctures in gameplay episodes. This work investigates a predictive student modelling approach that leverages the natural language text of the post-gameplay content knowledge questions and the text of the possible answer choices for early prediction of fine-grained individual student performance in game-based learning environments. With data from a study involving 66 undergraduate students from a large public university interacting with a game-based learning environment for microbiology, Crystal Island , we investigate the accuracy and early prediction capacity of student models that use a combination of gameplay features extracted from student log files as well as distributed representations of post-test content assessment questions. The results demonstrate that by incorporating knowledge about assessment questions, early prediction models are able to outperform competing baselines that only use student game trace data with no question-related information. Furthermore, this approach achieves high generalisation, including predicting the performance of students on unseen questions.

Practitioner notes

What is already known about this topic

• A distinctive characteristic of game-based learning environments is their capacity to enable fine-grained student assessment.
• Adaptive game-based learning environments offer individualisation based on specific student needs and should be able to assess student competencies using early prediction models of those competencies.
• Word embedding approaches from the field of natural language processing show great promise in the ability to encode semantic information that can be leveraged by predictive student models.

What this paper adds

• Investigates word embeddings of assessment question content for reliable early prediction of student performance.
• Demonstrates the efficacy of distributed word embeddings of assessment questions when used by early prediction models compared to models that use either no assessment information or discrete representations of the questions.
• Demonstrates the efficacy and generalisability of word embeddings of assessment questions for predicting the performance of both new students on existing questions and existing students on new questions.

Implications for practice and/or policy

• Word embeddings of assessment questions can enhance early prediction models of student knowledge, which can drive adaptive feedback to students who interact with game-based learning environments.
• Practitioners should determine if new assessment questions will be developed for their game-based learning environment, and if so, consider using our student modelling framework that incorporates early prediction models pretrained with existing student responses to previous assessment questions and is generalisable to the new assessment questions by leveraging distributed word embedding techniques.
• Researchers should consider the most appropriate way to encode the assessment questions in ways that early prediction models are able to infer relationships between the questions and gameplay behaviour to make accurate predictions of student competencies.