Performance assessment in serious games: Compensating for the effects of randomness

This paper is about performance assessment in serious games. We conceive serious gaming as a process of player-lead decision taking. Starting from combinatorics and item-response theory we provide an analytical model that makes explicit to what extent observed player performances (decisions) are blurred by chance processes (guessing behaviors). We found large effects both theoretically and practically. In two existing serious games random guess scores were found to explain up to 41 % of total scores. Monte Carlo simulation of random game play confirmed the substantial impact of randomness on performance. For valid performance assessments, be it in-game or post-game, the effects of randomness should be included to produce re-calibrated scores that can reasonably be interpreted as the players’ achievements.     

The effects of an online professional development course on teachers’ perceptions,attitudes, self-efficacy,and behavioral intentions regarding digital game-based learning

This study examined the effects of a graduate level, professional development (PD) course on teachers’ perceptions, attitudes, self-efficacy, and behavioral intentions regarding the use of digital games in the classrooms, teacher involvement in educational game design, and constructionist gaming. Twenty-one teachers participated in the study. Both quantitative and qualitative data were collected from pre- and post-surveys and participants’ reflections on major assignments. The results revealed significant changes in participants’ perceptions, attitudes, and self-efficacy. At the end of the course, all participants believed that digital games could be helpful for students’ learning and help students develop real-world skills, including higher-order thinking, problem solving, decision making, and collaboration skills. They also felt more comfortable using digital games in the classroom after completing the course. All participants agreed or strongly agreed that teachers should be involved in the process of educational game design. The participants also realized that constructionist gaming could provide an engaging and meaningful context where students can practice and develop real world skills. This study examined the impact of the PD experience on the participants’ behavioral intentions beyond their perceptions and attitudes. The results showed that the PD experience had a positive influence on the participants’ behavioral intentions. The findings of this study provide useful insights into teacher professional development.     

Learning analytics application to examine validity and generalizability of game-based assessment for spatial reasoning

Game-based assessment (GBA), a specific application of games for learning, has been recognized as an alternative form of assessment. While there is a substantive body of literature that supports the educational benefits of GBA, limited work investigates the validity and generalizability of such systems. In this paper, we describe applications of learning analytics methods to provide evidence for psychometric qualities of a digital GBA called Shadowspect, particularly to what extent Shadowspect is a robust assessment tool for middle school students' spatial reasoning skills. Our findings indicate that Shadowspect is a valid assessment for spatial reasoning skills, and it has comparable precision for both male and female students. In addition, students' enjoyment of the game is positively related to their overall competency as measured by the game regardless of the level of their existing spatial reasoning skills.

Practitioner notes

What is already known about this topic:

• Digital games can be a powerful context to support and assess student learning.
• Games as assessments need to meet certain psychometric qualities such as validity and generalizability.
• Learning analytics provide useful ways to establish assessment models for educational games, as well as to investigate their psychometric qualities.

What this paper adds:

• How a digital game can be coupled with learning analytics practices to assess spatial reasoning skills.
• How to evaluate psychometric qualities of game-based assessment using learning analytics techniques.
• Investigation of validity and generalizability of game-based assessment for spatial reasoning skills and the interplay of the game-based assessment with enjoyment.

Implications for practice and/or policy:

• Game-based assessments that incorporate learning analytics can be used as an alternative to pencil-and-paper tests to measure cognitive skills such as spatial reasoning.
• More training and assessment of spatial reasoning embedded in games can motivate students who might not be on the STEM tracks, thus broadening participation in STEM.
• Game-based learning and assessment researchers should consider possible factors that affect how certain populations of students enjoy educational games, so it does not further marginalize specific student populations.

     

Gaming personality and game dynamics in online discussion instructions

Gamification is the use of game mechanics to drive game-like engagements and actions. It applies game mechanics, dynamics and frameworks to promote desired learning behaviours. Positive and effective gamification could enhance learning and engage learners in more social and context-rich decision-making for problem-solving in learning tasks. Effective gamification design should understand the game personalities of learners, social engagement styles, and intrinsic- and extrinsic-motivated user styles initially and evaluate and select appropriate game mechanics, game dynamics, to create ideal gamification instructions. The research question this study investigated is: How will four types of gaming personalities (i.e. Explorer, Socialiser, Killer and Achiever) predict the level of game dynamics in online discussion environments? This study concluded that examining how gaming personalities relate to game dynamics in a gamified online discussion instruction is not sufficient to design effective gamification. It suggests future studies in game instructions should focus on the relationships between game dynamics, gamification contexts, gaming personalities or preferences, dynamic gaming engagement styles, etc.     

Video games as moral educators?

The growing interest in video gaming is matched by a corresponding increase in concerns about the harmful effects on children and adolescents. There are numerous studies on aggression and addiction which spark debates on the negative effects of video gaming. At the same time, there are also studies demonstrating prosocial effects. This paper focuses on how video games, particularly massively multiplayer online role-playing games (MMOs and MMORPGs for short) that allow interaction with other players, can play a part in players' moral and character development. Although there are many games with moral content, the MMO game World of Warcraft (WoW for short) is used as an illustrative example because of its popularity, to demonstrate how players, through game content and game play, are confronted with moral dilemmas which demand decision-making, social obligations and responsibilities, perspective-taking and empathy, perseverance and delayed gratification.     

Adapting a Face-to-Face Role-Playing Simulation for Online Play

The rapid acceleration of online course offerings presents a design challenge for instructors who want to take materials developed for face-to-face settings and adapt them for asynchronous online usage. Broadcast lectures are relatively easy to transfer, but adapting content is harder when classes use small-group discussions, as in role-playing or negotiation games. To be successful, such environments should address three interrelated design challenges: (a) sustaining engagement, (b) promoting content-focused discussion, and (c) promoting reflection-on-action. This article is a case study of how one interactive role-playing game, Island Telecom, was adapted for online play. We describe eight design features, including automated player roles and a structured team decision-making process, and show how they match with design challenges. Feedback from a recent run of this game shows that, although students still prefer to play face-to-face, they now also give favorable ratings to the online version. Feedback on specific adaptations is also presented.     

Simulating serious games: a discrete-time computational model based on cognitive flow theory

This paper presents a computational model for simulating how people learn from serious games. While avoiding the combinatorial explosion of a games micro-states, the model offers a meso-level pathfinding approach, which is guided by cognitive flow theory and various concepts from learning sciences. It extends a basic, existing model by exposing discrete-time evolution, allowing for failure, drop-out, and revisiting of activities, and accounting for efforts made and time spent on tasks, all of which are indispensable elements of gaming. Three extensive simulation studies are presented involving over 10,000 iterations across a wide range of game instances and player profiles for demonstrating model stability and empirical admissibility. The model can be used for investigating quantitative dependences between relevant game variables, gain deeper understanding of how people learn from games, and develop approaches to improving serious game design.     

The applicability of social network analysis to the study of networked learning

In this study, the aim was to examine how small-group collaboration is shaped by individuals interacting in a virtual multiplayer game. The data were collected from a design experiment in which six randomly divided groups of four university students played a voice-enhanced game lasting about 1 h. The ‘eScape’ game was a social action adventure developed as a part of the study. In the analysis of the video data, students' discourse functions during the game were analysed with content analytic methods for studying the nature of their interaction. An effort was made to analyse the data on both group and individual levels, and therefore the participants' prior social ties and experience in gaming were studied as well. The results showed that the students' main discourse functions in their conversation were Question, Content Statement, Instruction or Order and Response. It was found that individual students, especially those with prior knowledge of gaming or prior social ties, can have a major impact on the social interaction and the outcome of collaboration. It can be concluded that the eScape game allowed the students to engage in true and constructive collaborative activity, and in the future multiplayer games could be used, for example, to promote group cohesion and development, when employed in a pedagogically meaningful manner.     

Operation ARA: A computerized learning game that teaches critical thinking and scientific reasoning

Operation ARA (Acquiring Research Acumen) is a computerized learning game that teaches critical thinking and scientific reasoning. It is a valuable learning tool that utilizes principles from the science of learning and serious computer games. Students learn the skills of scientific reasoning by engaging in interactive dialogs with avatars. They are tutored by avatars with tutoring sessions that vary depending on how well students have responded to questions about the material they are learning. Students also play a jeopardy-like game against a feisty avatar to identify flaws in research and then generate their own questions to determine the quality of different types of research. The research examples are taken from psychology, biology, and chemistry to help students transfer the thinking skills across domains of knowledge. Early results show encouraging learning gains.     

Informal STEM: learning with robotics and game design in an urban context

Abstract

This mixed methods study examined how engagement in robotics and game design influenced students’ self-efficacy, STEM attitudes, and computational thinking (CT) skills. Predominantly African-American students engaged in engineering and computer science tasks during informal learning environments. Results revealed students’ self-efficacy scores on computer gaming increased significantly. Focus group data supported these findings, revealing that computer programing was challenging, but students enjoyed the tasks, added elements of culture in some cases, and valued the agency the tasks provided. Focal students were also able to make connections to STEM-related careers. Observational data demonstrated that focal students exhibited substantive CT during robotics and moderate CT during game design. Results support the idea that robotics and game design may be used to broaden underrepresented students’ participation in STEM.     

The acquisition of skill and expertise in massively multiplayer online games

Educational learning environments have changed dramatically in the last 20 years. Advances in technology have enabled the World Wide Web and a sundry of other tools. In response, many researchers have argued that one way to understand learning in a complex world is to examine user interactions within Massively Multiplayer Online Games (MMOGs) [Gee (2003). What video games have to teach us about learning and literacy. New York: Palgrave/St. Martin's; Squire (2003). Educational Researcher, 35(8), 19–29; Young, Schrader, & Zheng, 2006]. However, few empirical investigations have explored MMOGs as a context for learning. As a result, a 20-item, Likert-type instrument was administered to 2140 participants who actively play MMOGs. Items were designed to measure players’ gaming experiences as they developed requisite skill sets and learned game content. Specifically, this investigation examined how participants’ age ranges and levels of expertise relate to behaviors, strategies, and skills exhibited with an MMOG environment. Although results are not necessarily conclusive, implications for understanding gaming expertise in contemporary educational environments are discussed.     

Can children's instructional gameplay activity be used as a predictive indicator of reading skills?

For children who may face reading difficulties, early intervention is a societal priority. However, early intervention requires early detection. While much research has approached the issue of identification through measuring component skills at single timepoints, an alternative is the utilisation of dynamic assessment. To this point, few initiatives have explored the potential for identification through progress data from play in digital literacy games. This study explored how well growth curves from progress data in a digital intervention can predict reading performance after gameplay compared to measuring component skills at a single timepoint (school entry). 137 six-year-old students played the digital Graphogame for 25 weeks. Latent growth curve analyses showed that variation in trajectories explained variation in literacy performance to a greater extent than risk status at school entry. Findings point to a potential for non-intrusive reading assessment in the application of a serious digital game in first grade.     

The use of assessment in the development of higher-order thinking skills

Abstract

In an increasingly complex and specialised society, it is imperative that individuals think critically and creatively. This kind of thinking is also required to achieve the critical outcomes as stated in the Revised National Curriculum Statement for South Africa. It seems, however, that higher-order thinking is unlikely to occur unless learners are engaged in activities that deliberately promote this kind of thinking. They should also be guided on how to engage in these complex thinking tasks. The aim of this article is to reflect on the use of assessment to promote learners' higher-order thinking skills. The role of assessment has implications for the nature of teacher training programmes. If lecturers model the way assessment can be done to develop higher-order thinking skills, it is hoped that teachers will have little difficulty in adapting it to their unique classroom situations. Examples from language modules of an in-service teacher training programme are used to show how assessment can be implemented to help learners develop higher-order thinking skills within the framework of outcomes-based education and the Revised National Curriculum Statement.     

The Game Pokemon Go as a Crosscultural Phenomenon

The gaming culture of modern childhood takes place not only in real space, but in a virtual one too. These two components (two planes of development) are often viewed in isolation from one another. This study investigates the completely new phenomenon of augmented reality gaming through the lens of Pokemon Go, and it describes how the game has been able to transcend social, physical, and virtual spaces. We focus on the most active group of users of this mobile app, namely teenagers. We describe how the game satisfies the psychological needs of this age group. We attempt to understand the cross-cultural phenomenon of this game and the course of development these new forms of activity by modern teenagers may take in the future.     

The bothersome crow people and the silent princess: exploring the orientations of children as they play a digital narrative game

This study builds on and extends our understanding of literacy through exploring children's encounters with a digital narrative game. The research analyses different stances or orientations that children take as they progress through the game and how they draw on schematic understandings about narratives and digital gaming to support their game‐play. The study extends previous research exploring how children make meaning from visual texts and how we draw on resources across and between modes to understand narratives. Taking a socio‐cultural approach, the research suggests a framework of possible orientations that children take as they engage with the storyworld of the game, showing how this is at times strategic and critical, and at other times immersive and reactive.     

An Instrument for the Assessment of Cross-Curricular Skills

An instrument was constructed to assess the level of competence of students (aged 15-16) on eight cross-curricular skills. The instrument, further to be called the cross-curricular skills test (CCST), consisted of 56 multiple-choice items. In the first section cross-curricular skills are defined as general skills which can be taught and practiced in curricula for different disciplines. In the second section we describe how eight cross-curricular skills were selected for testing. We also explain why we decided to compose the test of multiple-choice items, an item-format that is not a customary one for measuring general skills. In the third section we report administration of the test to 465 students in a pilot study and 9,000 students in the main study. Quantitative data are presented concerning p -values, reliability, internal structure, and construct validity of the CCST. Qualitative data were derived from observations, thinking aloud protocols and analysis of the test content. The data warrant the claim that CCST is a valid and reliable test for cross-curricular skills. It is suitable for students of 15-16 years of age and allows administration in large samples. In the final section we discuss some limitations of the test and suggest directions for future research on assessment of cross-curricular skills.     

Hypermediating the Game Interface: The Alienation Effect in Violent Videogames and the Problem of Serious Play

This article looks at the practices of customization in violent urban videogames (such as Grand Theft Auto: San Andreas) that create “hypermediated” interfaces. Drawing from Bolter and Grusin's theory of hypermediation (which stands in contrast to immediate and immersive interfaces), this study looks at gaming practices that resist immediacy and instead focus on the disjunctions and contrasts created by forming avatars that starkly juxtapose their surroundings. Such an approach offers gamers the ability to create a space of cultural critique while simultaneously interrogating the term “serious game” because this category creates false binary oppositions between serious–trivial and serious–play.     

Assessing Teachers’ Beliefs about Literacy Acquisition: the Development of the Teachers’ Beliefs About Literacy Questionnaire (TBALQ)

In this article the authors describe the development of an instrument to measure change in teachers’ beliefs about how children acquire early literacy skills and how these skills should be taught. The questionnaire was originally designed for use in the evaluation of affective outcomes for teachers from literacy inservice training and development programmes. The content of the questionnaire has also proved to be of value as a focus for discussion in school staff meetings. Information is provided here on the design, development, validity and reliability of the instrument. The authors invite comment and feedback from researchers who may use the scale. The instrument may be particularly useful in countries where teachers’ beliefs and practices have been influenced to varying degrees by the ‘whole language’ versus ‘skills-based instruction’ debate.