Assessing approaches to learning with nonparametric multidimensional scaling

This article reports on a trace-based assessment of approaches to learning used by middle school aged children who interacted with NASA Mars Mission science, technology, engineering and mathematics (STEM) games in Whyville, an online game environment with 8 million registered young learners. The learning objectives of two games included awareness and knowledge of NASA missions, developing knowledge and skills of measurement and scaling, applying measurement for planetary comparisons in the solar system. Trace data from 1361 interactions were analysed with nonparametric multidimensional scaling methods, which permitted visual examination and statistical validation, and provided an example and proof of concept for the multidimensional scaling approach to analysis of time-based behavioural data from a game or simulation. Differences in approach to learning were found illustrating the potential value of the methodology to curriculum and game-based learning designers as well as other creators of online STEM content for pre-college youth. The theoretical framework of the method and analysis makes use of the Epistemic Network Analysis toolkit as a post hoc data exploration platform, and the discussion centres on issues of semantic interpretation of interaction end-states and the application of evidence centred design in post hoc analysis.

Practitioner notes

What is already known about this topic

• Educational game play has been demonstrated to positively affect learning performance and learning persistence.
• Trace-based assessment from digital learning environments can focus on learning outcomes and processes drawn from user behaviour and contextual data.
• Existing approaches used in learning analytics do not (fully) meet criteria commonly used in psychometrics or for different forms of validity in assessment, even though some consider learning analytics a form of assessment in the broadest sense.
• Frameworks of knowledge representation in trace-based research often include concepts from cognitive psychology, education and cognitive science.

What this paper adds

• To assess skills-in-action, stronger connections of learning analytics with educational measurement can include parametric and nonparametric statistics integrated with theory-driven modelling and semantic network analysis approaches widening the basis for inferences, validity, meaning and understanding from digital traces.
• An expanded methodological foundation is offered for analysis in which nonparametric multidimensional scaling, multimodal analysis, epistemic network analysis and evidence-centred design are combined.

Implications for practice and policy

• The new foundations are suggested as a principled, theory-driven, embedded data collection and analysis framework that provides structure for reverse engineering of semantics as well as pre-planning frameworks that support creative freedom in the processes of creation of digital learning environments.

     

Learning analytics application to examine validity and generalizability of game-based assessment for spatial reasoning

Game-based assessment (GBA), a specific application of games for learning, has been recognized as an alternative form of assessment. While there is a substantive body of literature that supports the educational benefits of GBA, limited work investigates the validity and generalizability of such systems. In this paper, we describe applications of learning analytics methods to provide evidence for psychometric qualities of a digital GBA called Shadowspect, particularly to what extent Shadowspect is a robust assessment tool for middle school students' spatial reasoning skills. Our findings indicate that Shadowspect is a valid assessment for spatial reasoning skills, and it has comparable precision for both male and female students. In addition, students' enjoyment of the game is positively related to their overall competency as measured by the game regardless of the level of their existing spatial reasoning skills.

Practitioner notes

What is already known about this topic:

• Digital games can be a powerful context to support and assess student learning.
• Games as assessments need to meet certain psychometric qualities such as validity and generalizability.
• Learning analytics provide useful ways to establish assessment models for educational games, as well as to investigate their psychometric qualities.

What this paper adds:

• How a digital game can be coupled with learning analytics practices to assess spatial reasoning skills.
• How to evaluate psychometric qualities of game-based assessment using learning analytics techniques.
• Investigation of validity and generalizability of game-based assessment for spatial reasoning skills and the interplay of the game-based assessment with enjoyment.

Implications for practice and/or policy:

• Game-based assessments that incorporate learning analytics can be used as an alternative to pencil-and-paper tests to measure cognitive skills such as spatial reasoning.
• More training and assessment of spatial reasoning embedded in games can motivate students who might not be on the STEM tracks, thus broadening participation in STEM.
• Game-based learning and assessment researchers should consider possible factors that affect how certain populations of students enjoy educational games, so it does not further marginalize specific student populations.

     

Oral discourse skills: Dimensionality of comprehension and retell of narrative and expository texts,and the relations of language and cognitive skills to identified dimensions

We examined the dimensionality of oral discourse skills (comprehension and retell of texts) and the relations of language and cognitive skills to the identified dimensions. Data were from 529 English-speaking second graders (M_age = 7.42; 46% female; 52.6% Whites, 33.8% African Americans, 4.9% Hispanics, 4.7% two or more races, .8% Asian Americans, .6% American Indians, .2% Native Hawaiians, 2.5% unknown; data from 2014–2015 to 2016–2017). Confirmatory factor analysis revealed that oral discourse skills are best described as four related but dissociable dimensions of narrative comprehension, narrative retell, expository comprehension, and expository retell (rs = .59–.84). Language and cognitive skills had different patterns of relations to the identified dimensions and explained larger amounts of variance in comprehension than in retell.