Modeling Student Cognition in Digital and Nondigital Assessment Environments

Inferences about student knowledge, skills, and attributes based on digital activity still largely come from whether students ultimately get a correct result or not. However, the ability to collect activity stream data as individuals interact with digital environments provides information about students’ processes as they progress through learning activities. These data have the potential to yield information about student cognition if methods can be developed to identify and aggregate evidence from diverse data sources. This work demonstrates how data from multiple carefully designed activities aligned to a learning progression can be used to support inferences about students’ levels of understanding of the geometric measurement of area. The article demonstrates evidence identification and aggregation of activity stream data from two different digital activities, responses to traditional assessment items, and ratings based on observation of in-person non-digital activity aligned to a common learning progression using a Bayesian Network approach.     

Using API Data to Understand Learning in League of Legends: A Mixed Methods Study

This mixed-methods research was conducted to understand the impact of learning and player growth in a League of Legends summer camp. Eighteen adolescents engaged in a three-day sleep-over summer camp with various team building activities. Data collected included API metadata from pre- and post-camp as well as semi-structured interview data with youth who attended the camp. Exposing the participants to teamwork methods had a significant impact on changing how players approached competitive game play and engagement in digital environments. In particular, we find that participation in the summer camp led to significant changes in vision score, the most team-focused aspect of the game for which statistics were available. Furthermore, as demonstrated by the qualitative data, evidence suggests youth have an understanding for how teamwork can positively influence peer interactions within digital environments. These findings are important as they demonstrate that teaching team-focused activities can have a significant impact on the players of competitive esports games, and it also implies that the role of teamwork within various digital platforms needs deeper study. Findings indicate that a transfer of learning occurs between physical and digital spaces and that youth recognize the importance of teamwork and team-building activities in digital learning spaces.     

Murder on Grimm Isle: The impact of game narrative design in an educational game‐based learning environment

The purpose of this research is to investigate the impact of narrative design in a game‐based learning environment. Specifically, this investigation focuses the narrative design in an adventure‐styled, game‐based learning environment for fostering argumentation writing by looking at how the game narrative impacted player/learner (1) intrinsic motivation, (2) curiosity, (3) plausibility and (4) transference of game‐based experiences into prewriting activities. The methodological framework for this qualitative investigation is a case study with grounded theory methodology. The setting is an educational, three‐dimensional, immersive game‐based learning environment titled Murder on Grimm Isle, used to foster argumentation and persuasion writing for Grades 9–14. The participants included 20 college students. The findings of the investigation reveal that intrinsic motivation, curiosity and plausibility were first supported by the game‐like environment, and then sustained through the narrative and the environment. Additionally, game‐based experiences were transferred into prewriting activities. Unanticipated findings revealed some student resistance. The goal of this research is to gain a better understanding of narrative design for game‐based learning environments.     

Content validity of game-based assessment: case study of a serious game for ICT managers in training

Serious games foster the acquisition of complex problem-solving skills. Assessment of such skills should be in line with instruction, and within a serious game environment its content validity should equal face-to-face assessment. Research on assessment in serious gaming has remained rather scarce. This article shows how assessment can be implemented in serious gaming in a way that assures content validity. The core of the authors’ validation method entails mapping learning activities (as contained in the game scenario) on performance indicators and outputs (as derived from formal attainment levels). They present how they have elaborated and applied the method for an assessment game for ICT managers in secondary vocational education. They describe the procedure and extent to which this assessment is content-valid compared to face-to-face assessment.     

Learning Fractions by Splitting: Using Learning Analytics to Illuminate the Development of Mathematical Understanding

The struggle with fraction learning in kindergarten through Grade 12 in the United States is a persistent problem and one of the major stumbling blocks to succeeding in higher mathematics. Research into this problem has identified several areas where students commonly struggle with fractions. While there are many theories of fraction learning, none of the research on these theories employs samples large enough to test theories at scale or nuanced enough to demonstrate how learning unfolds over time during instructional activities based on these theories. The work reported here uses learning analytics methods with fine-grained log data from an online fraction game to unpack how splitting (i.e. partitioning a whole into equal-sized parts) impacts learning. Study 1 demonstrated that playing the game significantly improved students’ fraction understanding. In addition, a cluster analysis suggested that exploring splitting was beneficial. Study 2 replicated the learning results, and a cluster analysis showed that compared to early game play, later game play showed more optimal splitting strategies. In addition, in looking at the types of transitions that were possible between a student’s early cluster categorization and later cluster categorization, we found that some types of transitions were more beneficial for learning than others.     

A digital game-based assessment of middle-school and college students’ choices to seek critical feedback and to revise

A major goal of contemporary education is to teach students how to learn on their own. Assessments have largely lagged behind this goal, because they measure what students have learned and not necessarily their learning processes. This research presents Posterlet, an assessment that collects evidence regarding the strategies that students choose while learning on their own. Posterlet is an educational game-based assessment that measures two design thinking choices: students’ choices to seek critical (ie, negative) feedback and to revise their work while they learn graphic design principles through creating posters. This research also presents an examination of students’ choices to seek feedback and to revise, as well as of students’ learning outcomes based on these choices. This game-based assessment approach is empirically validated with three research studies sampling nearly 300 middle-school and college students who played Posterlet and completed a posttest. Results show that the game helps students learn, as students who play the game before completing the posttest learn more graphic design principles than students who only complete the posttest. Moreover, the choices to seek critical feedback and to revise can predict learning and can be used as valid outcome measures for learning. Findings can be used in developing and evaluating models of instruction and assessment that may help students make informed learning choices. A discussion of present and future trends in theory regarding digital feedback environments is also included.     

Video game learning dynamics: Actionable measures of multidimensional learning trajectories

Valid, accessible, reusable methods for instructional video game design and embedded assessment can provide actionable information enhancing individual and collective achievement. Cyberlearning through game‐based, metaphor‐enhanced learning objects (CyGaMEs) design and embedded assessment quantify player behavior to study knowledge discovery and application. CyGaMEs is grounded by analogical reasoning theory, cognitive task analysis and knowledge representation. A construct representation argument for validity using evidence‐centered design warrants CyGaMEs, its web‐based learning environment, Selene: A Lunar Construction GaME, its embedded assessment, and a learning dynamics approach to student, measurement and statistical models. Two studies (US volunteers, Study 1: n = 267, mean (M)_age = 15; Study 2: n = 90, M_age = 12) cross‐validate learning dynamics (learner progress, rate of progress and changes in that rate) for Selene's multidimensional goals while players learn and apply standard‐based science about fundamental geology and space science concepts. Gameplay data analyzed using regression, calculus and hierarchical linear modeling exhibit overall relatively high standardized rates of progress toward each goal statistically higher than zero. For example, adjusting for nesting within individual players, average rate of progress toward the goal of accreting lunar mass is a z‐score of 1.4 (99% confidence interval = 1.40_lower, 1.48_upper) or 1.4 standard deviations above zero.     

Developing a community of thinking: assessment of environmental education

This paper discusses an oral assessment intervention in environmental education at two tertiary institutions in South Africa. A qualitative study grounded in a social constructivist framework, the inquiry locates learning and assessment of environmental education based on practical activities and first‐hand experience within the framework of situated learning, and explores processes of constructing and assessing knowledge within the learner's community of practice. The paper focuses on how communities of practice can be used as a possible learning approach in environmental education to address the diversity and challenges of classrooms, and on the efficacy of oral assessments structured to encourage thinking through dialog in such settings.     

Researchers and teachers learning together and from each other using video‐based multimodal analysis

This paper discusses a year‐long technology integration project during which teachers and researchers joined forces to explore children's collaborative activities through the use of touch screens. In the research project discussed in this paper, 16 touch screens were integrated into teaching and learning activities in two separate classrooms; the learning and collaborative processes were captured by using a video, collecting over 150 hours of footage. By using digital research technologies and a longitudinal design, the authors of the research project studied how teachers and children gradually integrated touch screens into their teaching and learning. This paper examines the methodological usefulness of video‐based multimodal analysis. Through reflection on the research project, we discuss how, by using video‐based multimodal analysis, researchers and teachers can study children's touch‐screen supported collaboration and how researchers and teachers can learn together.     

Role playing games for scientific citizenship

Research has shown that video games can be good for learning, particularly for STEM topics. However, in order for games to be scalable and sustainable, associated research must move beyond considerations of efficacy towards theories that account for classroom ecologies of students and teachers. This study asks how a digital game called Citizen Science, built using tropes and conventions from modern games, might help learners develop identities as citizen scientists within the domain of lake ecology. We conducted an expert-novice study, revealing that games literacy was a mediating variable for content understanding. In a follow-up classroom implementation, games literacy also operated as a variable, although students drove the activity, which mediated this concern. The teacher devised a number of novel pedagogies, such as a field trip, in response to the unit. We found evidence for the most powerful learning occurring through these activities that were reinforced via the curriculum. Students were most engaged by Citizen Science??s most ??gamelike?? features, and learners took up the core ideas of the game. Users also reported the experience was short of commercial gaming experiences, suggesting a tension between game cultures for learning and schools.     

Teachers’ perceptions of boys’ and girls’ shared activities in the school context: towards a theory of collaborative play

The Finnish educational system and curricula lay emphasis on play, collaboration and equality. Modern educational practices allow the learning environment to be enlarged from indoor classrooms to outdoor playful learning environments (PLEs). PLEs have been constructed in schoolyards in Finland with the goal of increasing learning through play in curriculum‐based education and augmenting collaborative play (ColPlay) between boys and girls. In order to better understand and describe such developments, the author set out to ascertain how teachers perceive mixed‐gender play activities in pre‐primary and basic education. Fourteen teachers were interviewed and the obtained data were analysed using the grounded theory as an analytical approach. The research suggests five premises for ColPlay: (1) the most suitable forms of ColPlay are outdoor games and role‐play, (2) gender roles adjust in contemporary play culture, (3) teachers’ pedagogical thinking on ColPlay includes various practices to promote collaborative relationships between girls and boys, (4) teachers have confidence in ColPlay and (5) learning to collaborate with both genders requires practice and reflection. The study offers useful insights for teachers, teacher educators and designers of game content and learning environments.     

Beat the Instructor: An Introductory Forecasting Game

This teaching brief describes a 30‐minute game where student groups compete in‐class in an introductory time‐series forecasting exercise. The students are challenged to “beat the instructor” who competes using forecasting techniques that will be subsequently taught. All forecasts are graphed prior to revealing the randomly generated actual demand. Experiential learning and forecasting exercises are reviewed before detailing how the game is administered and debriefed. Student survey results (n = 247) provide evidence of teaching effectiveness, including that 78% of students said that playing the game increased their interest in learning the forecasting techniques that were used by the instructor.     

REJOINDER

In this paper, we explore the influence of the teacher on the language games being played in a Grade 3 mathematics classroom. The kinds of listening strategies used by the teacher seemed to have an impact on the kind of language game being played as a small group of students engaged in a problem‐solving activity. Understanding classroom conversation from a language‐games perspective is used to explore the kinds of sense‐making, student engagement, and new understandings evolving through classroom discourse. Language games may also be important for encouraging students to extend and transform their understandings. We describe how learning to ‘teach etcetera’ as a way of helping students ‘see‐as’ and move beyond narrowly defined predetermined content may entail playing more open‐ended language games and learning how to ‘listen’.     

Connecting assessment practices with curricula and pedagogy via implementation fidelity data

Globally, higher education institutions aim to assess and improve students’ learning. However, assessment practices often do not culminate in improved learning. For instance, in the United States, empirical examples of learning improvement in higher education are exceptionally rare. A disconnect between assessment practices and instruction at the classroom level likely contributes to scarce evidence of improvement. In the current study, we provide a didactic example of how outcomes assessment data can be integrated with data concerning the educational intervention students received. This type of information – referred to as implementation fidelity data – allows faculty to more accurately interpret outcomes assessment results, make informed modifications to educational interventions, and link learning improvements back to specific interventions. As more faculty purposefully integrate outcomes assessment data and implementation fidelity data, higher education will better demonstrate its worth.     

Examining socially shared regulation and shared physiological arousal events with multimodal learning analytics

Socially shared regulation contributes to the success of collaborative learning. However, the assessment of socially shared regulation of learning (SSRL) faces several challenges in the effort to increase the understanding of collaborative learning and support outcomes due to the unobservability of the related cognitive and emotional processes. The recent development of trace-based assessment has enabled innovative opportunities to overcome the problem. Despite the potential of a trace-based approach to study SSRL, there remains a paucity of evidence on how trace-based evidence could be captured and utilised to assess and promote SSRL. This study aims to investigate the assessment of electrodermal activities (EDA) data to understand and support SSRL in collaborative learning, hence enhancing learning outcomes. The data collection involves secondary school students (N = 94) working collaboratively in groups through five science lessons. A multimodal data set of EDA and video data were examined to assess the relationship among shared arousals and interactions for SSRL. The results of this study inform the patterns among students' physiological activities and their SSRL interactions to provide trace-based evidence for an adaptive and maladaptive pattern of collaborative learning. Furthermore, our findings provide evidence about how trace-based data could be utilised to predict learning outcomes in collaborative learning.

Practitioner notes

What is already known about this topic

• Socially shared regulation has been recognised as an essential aspect of collaborative learning success.
• It is challenging to make the processes of learning regulation ‘visible’ to better understand and support student learning, especially in dynamic collaborative settings.
• Multimodal learning analytics are showing promise for being a powerful tool to reveal new insights into the temporal and sequential aspects of regulation in collaborative learning.

What this paper adds

• Utilising multimodal big data analytics to reveal the regulatory patterns of shared physiological arousal events (SPAEs) and regulatory activities in collaborative learning.
• Providing evidence of using multimodal data including physiological signals to indicate trigger events in socially shared regulation.
• Examining the differences of regulatory patterns between successful and less successful collaborative learning sessions.
• Demonstrating the potential use of artificial intelligence (AI) techniques to predict collaborative learning success by examining regulatory patterns.

Implications for practice and/or policy

• Our findings offer insights into how students regulate their learning during collaborative learning, which can be used to design adaptive supports that can foster students' learning regulation.
• This study could encourage researchers and practitioners to consider the methodological development incorporating advanced techniques such as AI machine learning for capturing, processing and analysing multimodal data to examine and support learning regulation.

     

Using camels to teach probability and expected value

There is much debate about the place of probability in an introductory statistics course. While students may or may not use probability distributions in their post‐collegiate lives, they will likely be faced with day‐to‐day decisions that require a probabilistic assessment of risk and reward. This paper describes an innovative way to teach students in an introductory statistics course how to reason about probability and expected value through the board game Camel Up. The in‐class activities described here can be scaled to fit the needs and abilities of the students in a wide range of introductory statistics courses at a variety of levels.     

Developing teachers’ capacities in assessment through career-long professional learning

Abstract

In a context of increasing demand for quality and equity in education and a sharp focus on accountability, classroom teachers are also expected to support and improve learning outcomes for pupils in response to their individual needs. This paper explores three issues: how teachers understand assessment in relation to their students’ learning, the curriculum and their pedagogical choices; how teachers’ capacity to use assessment to improve students’ learning can be developed through career-long professional learning (CLPL); and how teachers’ learning can be implemented and sustained in schools, both locally and nationally. In considering these issues, recent thinking about learning and assessment and CLPL are considered alongside empirical evidence from the development and implementation of assessment processes and approaches to professional development in Scotland. The paper emphasises the importance of a dynamic framework of CLPL that recognises the individuality of teachers’ learning needs and the consequent need for tailored professional learning opportunities with different combinations of support and challenge at school, local and national levels.     

The assessment cycle: a model for learning through peer assessment

This paper advances a model describing how peer assessment supports self-assessment. Although prior research demonstrates that peer assessment promotes self-assessment, the connection between these two activities is underspecified. This model, the assessment cycle, draws from theories of self-assessment to elaborate how learning takes place through peer assessment. The model is applied to three activity structures described in the literature to analyse their potential to support learning by promoting self-assessment. Broadly speaking, the model can be used to understand learning that takes place in a variety of peer assessment activities: marking/grading, analysis, feedback, conferencing and revision. This approach contrasts most studies on peer assessment, which have focused on calibration of instructor and peer grades, rather than learning opportunities.     

Teaching and Learning in Multi-Grade Classrooms: The LEPO Framework

Abstract

This article reports on a study that examined teaching and learning in multi-grade classrooms using the Learning Environment, Learning Processes and Learning Outcomes (LEPO) conceptual framework. The study sought to investigate how the learning environment is created; how the processes of teaching and learning take place; and how assessment is used to determine the achievement of learning outcomes in multi-grade classrooms. A qualitative research design was used, where interviews were conducted with nine teachers from schools with multi-grade classrooms. The data was analysed thematically and revealed the following: the learning environment can be created by grouping learners appropriately in classrooms, creating learning stations and reading stations, proper use of time-tables, and adaptation of teaching plans; the learning processes should take place through the differentiated curricular approach or quasi mono-grade, that is, learners should be afforded the opportunity to learn from their family members, teaching and learning should take place through self-directed learning, peer tutoring and cooperative learning, and lessons should cater for different learning styles; to determine the realisation of learning outcomes teachers should assess learners in different grades informally with either the same or grade-specific assessment activities and formally with grade-specific assessment tasks. The article also addresses the main criticisms against the LEPO framework by explaining how teachers and learners should interact with the learning environment, learning processes and learning outcomes in multi-grade classrooms. The article concludes that if the LEPO framework can be implemented in multi-grade classrooms, teaching and learning can be strengthened in such classrooms.