Designing for impact: a conceptual framework for learning analytics as self-assessment tools

Abstract

Although it is frequently claimed that learning analytics can improve self-evaluation and self-regulated learning by students, most learning analytics tools appear to have been developed as a response to existing data rather than with a clear pedagogical model. As a result there is little evidence of impact on learning. Even fewer learning analytics tools seem to be informed by an understanding of the social context and social practices within which they would be used. As a result, there is very little evidence that learning analytics tools are actually impacting on practice. This paper draws on research in self-regulated learning and in the social practices of learning and assessment to clarify a series of design issues which should be considered by those seeking to develop learning analytics tools which are intended to improve student self-evaluation and self-regulation. It presents a case study of how these design issues influenced the development of a particular tool: the Learning Companion.     

Creating an alternate reality: Critical,creative, and empathic thinking generated in the Global Village Playground capstone experience

The Global Village Playground (GVP) was a capstone learning experience designed to address institutional assessment needs while providing an integrated and authentic learning experience for students aimed at fostering critical and creative thinking. In the GVP, students work on simulated and real-world problems as a design team tasked with developing an alternate reality game that makes an impact on the United Nations Millennium Development Goals. Researchers employed a qualitative case study approach to evaluate participant reactions to the course, their perceptions of the instructional design methodology, what they learned in the course, and the challenges they experienced during the pilot implementation of this capstone design. This article documents what students reported to have learned in the course as a result of their experience creating an alternate reality game.     

Enhancing Inquiry, Understanding, and Achievement in an Astronomy Multimedia Learning Environment

As an example of design-based research, this study refined an assessment strategy for simultaneously enhancing inquiry-based learning and supporting achievement on conventional assessment measures. Astronomy Village ^® : Investigating the Universe? is a software program designed to engage secondary science students in authentic and inquiry-based learning over core topics in astronomy. The software was enhanced with a 20-hour curriculum and three levels of assessment to ensure successful inquiry experiences and high-stakes achievement. The first year implementation of Astronomy Village^® yielded significant gains on a curriculum-oriented exam but not a standards-oriented test, and provided useful design insights that were integrated into the second year implementations. Significant gains were obtained on the test during the second year as well. It is expected that many existing inquiry-oriented science curricula might be similarly enhanced, and is suggested that a large-scale effort to do so might have a lasting impact on science education.     

Space: the final frontier in the learning of science?

In Space, relations, and the learning of science, Wolff-Michael Roth and Pei-Ling Hsu use ethnomethodology to explore high school interns learning shopwork and shoptalk in a research lab that is located in a world class facility for water quality analysis. Using interaction analysis they identify how spaces, like a research laboratory, can be structured as smart spaces to create a workflow (learning flow) so that shoptalk and shopwork can projectively organize the actions of interns even in new and unfamiliar settings. Using these findings they explore implications for the design of curriculum and learning spaces more broadly. The Forum papers of Erica Blatt and Cassie Quigley complement this analysis. Blatt expands the discussion on space as an active component of learning with an examination of teaching settings, beyond laboratory spaces, as active participants of education. Quigley examines smart spaces as authentic learning spaces while acknowledging how internship experiences all empirical elements of authentic learning including open-ended inquiry and empowerment. In this paper I synthesize these ideas and propose that a narrative structure might better support workflow, student agency and democratic decision making.     

Dealing with errors in mathematics classrooms: Structure and relevance of perceived error climate

Errors are often perceived by students as self-threatening and not as learning opportunities. The present work focuses on contextual influences on reactions to errors and learning processes. Based on prior research, a conceptualization of perceived error climate in the classroom with eight subdimensions and one superordinate uniform factor is proposed and a newly developed student questionnaire for its assessment is presented. Results of a study with N = 1116 students from 56 mathematics classrooms in German secondary schools indicated the validity of the error climate conceptualization and the suitability of the questionnaire. Moreover, the results showed that perceived error climate in the classroom predicted the adaptivity of students’ individual reactions to errors above and beyond perceived classroom goal structures and personal achievement motivation (academic self-concept, mastery goal orientation). In addition, the study provided evidence that perceived error climate affects – partially mediated through students’ individual reactions to errors – the quantity and self-regulation of students’ effort.     

Self-regulated learning by writing learning protocols: Do goal structures matter?

Having students write learning protocols is an established follow-up task to course work that elicits processes of self-regulated learning. To date, a wealth of research has focused on optimizing the learning protocol task itself by integrating instructional support measures such as prompts. By contrast, research that focuses on optimizing the context in which learning protocol writing is embedded is widely missing. In the present study, we approached this gap by focusing on the role of the contextual factor goal structure. Specifically, in an experiment with N = 166 high school students we tested whether embedding the learning protocol task in a mastery goal structure would be more beneficial than embedding it in a performance goal structure. The mastery goal structure fostered metacognitive processes, learning outcomes, and learning efficiency. We conclude that considering contextual factors such as the goal structure is crucial for understanding and optimizing the effects of learning protocols.     

自我评价和人生意义在应付方式影响学习动机中的链式中介效应

采用问卷调查法,对867名在校大学生进行调查。结果发现:（1）大学生学习动机与应付方式、核心自我评价和人生意义呈显著正相关;（2）应付方式对大学生学习动机有显著的正向影响;（3）核心自我评价和人生意义在应付方式影响学习动机中起链式双中介作用,其中应付方式通过影响核心自我评价从而进一步影响学习动机这一中介路径的中介效果量最大。本研究为激发大学生学习动机、提升大学生心理健康水平以及改善大学教学质量提供了实证基础。     

A multi-study examination of intra-individual feedback loops between competence and value beliefs,procrastination, and goal achievement

In the present study, we tested intra-individual feedback loops between competence beliefs, value beliefs, and goal achievement (virtuous circles), and intra-individual feedback loops between goal failure and procrastination (vicious circle). We analyzed data from five independent intensive longitudinal studies with university students (N = 841, k = 23,448 observations). Pre-registered hypotheses were tested across the five studies and aggregated using meta-analytic methods. Results provided support for virtuous circles in self-regulated learning: Students who reported higher competence and value beliefs in one study session reported higher goal achievement, and higher goal achievement predicted higher competence and value beliefs in the subsequent study session. Results provided only partial support for a vicious circle: Procrastination was associated with lower goal achievement but goal achievement did not predict subsequent procrastination. The results have theoretical implications for models of self-regulated learning and methodological implications for the design of experience sampling studies.     

Developing the capacity of farmers to understand and apply seasonal climate forecasts through collaborative learning processes

Purpose: This paper documents and evaluates collaborative learning processes aimed at developing farmer’s knowledge, skills and aspirations to use seasonal climate forecasting (SCF). Methodology: Thirteen workshops conducted in 2012 engaged over 200 stakeholders across Australian sugar production regions. Workshop design promoted participant interaction, stimulated discussion, collected farmer experience of regional climate variability, improved understanding of climate drivers and increased participant skill in interpreting SCF linked to management decisions. Post-workshop surveys collected quantitative and qualitative data for statistical analysis and manual thematic coding. Findings: Over 68% of participants identified improved decision-making and risk reduction as the main benefits of using SCF products. High median self-evaluation ratings for gains in skills, knowledge and understanding of climate forecasts and perceived benefits in using climate forecasts in on-farm decision-making were found across stakeholder groups. No significant differences in self-evaluation rating gains were found between farmers, extension officers and millers, suggesting extension officers are less than optimally informed and skilled to support farmers in understanding and applying SCF. Practical Implications: Developing the capacity of extension officers to understand and interpret SCF may increase adoption of improved climate risk management in farmer networks. Theoretical implications: Collaborative learning activities in the agriculture sector, designed using experiential learning principles and evaluated using a logical framework, provide a robust model for improving the capacity of farmers to manage climate risk. Originality/Value: This paper contributes an example of evaluation of collaborative learning in facilitated agriculture climate risk workshops and discusses the value of learning through small group discussion.     

反思学习:涵义、功能与过程

反思学习是指学习者以自身经验、活动或身心结构为对象,以反身性的自我观察、分析、评价、改造、修炼等方式进行的学习。反思学习的主要功能在于,通过反思提高学习者自我认识、自我评价、自我对待以及自我发展的能力。反思是建构性学习、自主学习、学生主体性发展等的核心环节。反思学习的过程实际上是行动(经验)与探究性的思考之间的关联与互动的过程。这一过程是由多个环节构成的。学生的反思学习可通过多种途径展开,并可利用多种手段对其进行促进。     

Designing Instructional Text in a Conversational Style: A Meta-analysis

This article reviews research on the effects of conversational style on learning. Studies of conversational style have variously investigated “personalization” through changing instances of first-person address to second or third person, including sentences that directly address the learner; including more polite forms of address; and making the views and personality of the author more visible. Meta-analyses provided mixed support for a model of learning processes; statistically reliable average effects were found on self-reports of friendliness (d?=?0.46) and effective cognitive processing (d?=?0.62), but not learning assistance (d?=?0.16) and interest (d?=?0.15). Statistically reliable average effects on retention (d?=?0.30) and transfer (d?=?0.54) learning outcomes supported conversational-style redesigns across a range of potential moderators; the clearest apparent boundary condition for learning outcomes across the moderators under analysis was instructional time, with small, non-significant effects being found in studies longer than 35 min. Recommendations for future investigations are discussed.     

Search time and test anxiety in a visual location task

This study was conducted to investigate whether increasing search time in a visual location task would result in significantly different relationships between worry/emotionality test anxiety component scores and performance. Fifty graduate students were given 3.5 or 8 s to locate seven embedded geometrical figures projected on a screen. Correlation coefficients (anxiety component scores X performance scores) for the two treatment groups were significantly different for the emotionality component, r = ? .73 (3.5 s), r = ? .06 (8 s), z(r) = 2.8, p < .01; but not for the worry component, r = ? .61 (3.5 s), r = ? .26 (8 s). Recommendations include further study utilizing actual classroom situations and commercial instructional materials. Tentative recommendations for the production and utilization of instructional materials are provided.     

Differences in Metacognitive Regulation in Introductory Biology Students: When Prompts Are Not Enough

Strong metacognition skills are associated with learning outcomes and student performance. Metacognition includes metacognitive knowledge—our awareness of our thinking—and metacognitive regulation—how we control our thinking to facilitate learning. In this study, we targeted metacognitive regulation by guiding students through self-evaluation assignments following the first and second exams in a large introductory biology course (n = 245). We coded these assignments for evidence of three key metacognitive-regulation skills: monitoring, evaluating, and planning. We found that nearly all students were willing to take a different approach to studying but showed varying abilities to monitor, evaluate, and plan their learning strategies. Although many students were able to outline a study plan for the second exam that could effectively address issues they identified in preparing for the first exam, only half reported that they followed their plans. Our data suggest that prompting students to use metacognitive-regulation skills is effective for some students, but others need help with metacognitive knowledge to execute the learning strategies they select. Using these results, we propose a continuum of metacognitive regulation in introductory biology students. By refining this model through further study, we aim to more effectively target metacognitive development in undergraduate biology students.     

Assessment of prosocial–altruistic behavior of members and non-members of the scout movement

The aim of this study is to evaluate the differences in prosocial altruistic behavior between children and young students who belong to the scout movement and those who do not belong to this or any other similar movement. The prosocial altruistic behavior has been assessed with questionnaires for the school: self-evaluation, teacher, classmate, and parents. By means of a prospective design of case–control, every scout has been compared with another non-scout matched in variables like age; number of siblings; whether he/she is the oldest, the youngest, etc.; sex; the order between the siblings; if the mother works inside or outside the home; and if he/she lives with his/her two parents. The results show that significant differences exist between one group and another in the questionnaires of self-evaluation, for the teacher and for the classmate. It seems that children and young people who are scout members are better assessed than the non-scouts in prosocial altruistic behavior.     

Individual differences in graduate students' help-seeking process in using a computer coach in problem-based learning

There is a scarcity of research regarding help seeking in the context of computer learning environments providing on-demand help, a context in which help-seeking skills appear critical for learning [Aleven, V., Stahl, E., Schworm, S., Fisher, F., & Wallace, R. (2003). Help seeking and help design in interactive learning environments. Review of Educational Research, 73 (3), 277–320]. Whereas a cognitive model of help seeking has been suggested and tested in this context [Mercier, J., & Frederiksen, C. H. The structure of the help-seeking process in using a computer coach in problem-based learning. Computers and Education, submitted for publication], individual differences remained to be studied from this perspective. Participants were 18 graduate students, who solved a challenging statistics problem in pairs, with the help of a computer coach providing on-demand help. Log linear analyses found individual differences in help seeking with respect to (a) its relative importance in problem solving, (b) the relative importance of its component processes, (c) the modulation of help seeking over a series of sub-tasks, and (d) the relationship between help seeking and the quality of the solutions that the student elaborated. Theoretical implications are discussed.     

Evaluating a Modeling Curriculum by Using Heuristics for Productive Disciplinary Engagement

The BIO2010 report provided a compelling argument for the need to create learning experiences for undergraduate biology students that are more authentic to modern science. The report acknowledged the need for research that could help practitioners successfully create and reform biology curricula with this goal in mind. Our objective in this article was to explore how a set of six design heuristics could be used to evaluate the potential of curricula to support productive learning experiences for science students. We drew on data collected during a long-term study of an undergraduate traineeship that introduced students to mathematical modeling in the context of modern biological problems. We present illustrative examples from this curriculum that highlight the ways in which three heuristics—instructor role-modeling, holding students to scientific norms, and providing students with opportunities to practice these norms—consistently supported learning across the curriculum. We present a more detailed comparison of two different curricular modules and explain how differences in student authority, problem structure, and access to resources contributed to differences in productive engagement by students in these modules. We hope that our analysis will help practitioners think in more concrete terms about how to achieve the goals set forth by BIO2010.     

作文自我评估的元认知方法研究

自主学习过程中要不断地进行自我评估。自我评估的元认知方法在自主学习模式下的作文评估不可或缺。自主评估对错误的补救活动是完整的学习活动,元认知方法是自主学习质量的重要保证。     

Using a false biofeedback methodology to explore relationships between learners’ affect,metacognition, and performance

We used a false-biofeedback methodology to manipulate physiological arousal in order to induce affective states that would influence learners’ metacognitive judgments and learning performance. False-biofeedback is a method used to induce physiological arousal (and resultant affective states) by presenting learners with audio stimuli of false heart beats. Learners were presented with accelerated, baseline, or no heart beat (control) while they completed a challenging learning task. We tested four hypotheses about the effect of false-biofeedback. The alarm vs. alert hypothesis predicted that false biofeedback would be appraised as either a signal of distress and would impair learning (alarm), or as a signal of engagement and would facilitate learning (alert). The differential biofeedback hypothesis predicted that the alarm and alert effects would be dependent on the type of biofeedback (accelerated vs. baseline). The question depth hypothesis predicted that these effects would be more pronounced for challenging inference questions. Lastly, the self vs. recording hypothesis predicted that effects would only occur if participants believed that false biofeedback was indicative of their own physiological arousal. In general, learners experienced more positive/activating affective states, made more confident metacognitive judgments, and achieved higher learning when they received accelerated or baseline biofeedback while answering a challenging inference question, irrespective of the perceived source of the biofeedback. Thus, our findings supported the alert and question depth hypotheses, but not the differential biofeedback or self vs. recording hypotheses. Implications of the findings for the integration of affective processes into models of cognitive and metacognitive processes during learning are discussed.