Supporting the less-adaptive student: the role of learning analytics,formative assessment and blended learning

Abstract

How can we best facilitate students most in need of learning support, entering a challenging quantitative methods module at the start of their bachelor programme? In this empirical study into blended learning and the role of assessment for and as learning, we investigate learning processes of students with different learning profiles. Specifically, we contrast learning episodes of two cluster analysis-based profiles, one profile more directed to deep learning and self-regulation, the other profile more directed toward stepwise learning and external regulation. In a programme based on problem-based learning, where students are supposedly being primarily self-directed, this first profile is regarded as being of an adaptive type, with the second profile less adaptive. Making use of a broad spectrum of learning and learner data, collected in the framework of a dispositional learning analytics application, we compare these profiles on learning dispositions, such as learning emotions, motivation and engagement, learning performance and trace variables collected from the digital learning environments. Outcomes suggest that the blended design of the module with the digital environments offering many opportunities for assessment of learning, for learning and as learning together with actionable learning feedback, is used more intensively by students of the less adaptive profile.     

Assisting transition to university: using assessment as a formative learning tool

This study explores the effectiveness of an intervention involving formative assessment in a first‐year core business subject. Students were invited to receive feedback on a draft of their first written assessment during the early weeks of the semester. Consideration is given to the economic and ethical issues raised by the intervention. A multi‐method approach of qualitative and quantitative data collection and analysis is used. The research finds that the intervention facilitates significantly higher marks in assessments and grades, while assisting student learning overall. Findings are reinforced by comparison with a subject where the intervention was not offered.     

How do the existing fairness metrics and unfairness mitigation algorithms contribute to ethical learning analytics?

With the widespread use of learning analytics (LA), ethical concerns about fairness have been raised. Research shows that LA models may be biased against students of certain demographic subgroups. Although fairness has gained significant attention in the broader machine learning (ML) community in the last decade, it is only recently that attention has been paid to fairness in LA. Furthermore, the decision on which unfairness mitigation algorithm or metric to use in a particular context remains largely unknown. On this premise, we performed a comparative evaluation of some selected unfairness mitigation algorithms regarded in the fair ML community to have shown promising results. Using a 3-year program dropout data from an Australian university, we comparatively evaluated how the unfairness mitigation algorithms contribute to ethical LA by testing for some hypotheses across fairness and performance metrics. Interestingly, our results show how data bias does not always necessarily result in predictive bias. Perhaps not surprisingly, our test for fairness-utility tradeoff shows how ensuring fairness does not always lead to drop in utility. Indeed, our results show that ensuring fairness might lead to enhanced utility under specific circumstances. Our findings may to some extent, guide fairness algorithm and metric selection for a given context.

Practitioner notes

What is already known about this topic

• LA is increasingly being used to leverage actionable insights about students and drive student success.
• LA models have been found to make discriminatory decisions against certain student demographic subgroups—therefore, raising ethical concerns.
• Fairness in education is nascent. Only a few works have examined fairness in LA and consequently followed up with ensuring fair LA models.

What this paper adds

• A juxtaposition of unfairness mitigation algorithms across the entire LA pipeline showing how they compare and how each of them contributes to fair LA.
• Ensuring ethical LA does not always lead to a dip in performance. Sometimes, it actually improves performance as well.
• Fairness in LA has only focused on some form of outcome equality, however equality of outcome may be possible only when the playing field is levelled.

Implications for practice and/or policy

• Based on desired notion of fairness and which segment of the LA pipeline is accessible, a fairness-minded decision maker may be able to decide which algorithm to use in order to achieve their ethical goals.
• LA practitioners can carefully aim for more ethical LA models without trading significant utility by selecting algorithms that find the right balance between the two objectives.
• Fairness enhancing technologies should be cautiously used as guides—not final decision makers. Human domain experts must be kept in the loop to handle the dynamics of transcending fair LA beyond equality to equitable LA.

     

Building personalised homework from a learning analytics based formative assessment: Effect on fifth-grade students' understanding of fractions

This study analyses the potential of a learning analytics (LA) based formative assessment to construct personalised teaching sequences in Mathematics for 5th-grade primary school students. A total of 127 students from Spanish public schools participated in the study. The quasi-experimental study was conducted over the course of six sessions, in which both control and experimental groups participated in a teaching sequence based on mathematical problems. In each session, both groups used audience response systems to record their responses to mathematical tasks about fractions. After each session, students from the control group were given generic homework on fractions—the same activities for all the participants—while students from the experimental group were given a personalised set of activities. The provision of personalised homework was based on the students' errors detected from the use of the LA-based formative assessment. After the intervention, the results indicate a higher student level of understanding of the concept of fractions in the experimental group compared to the control group. Related to motivational dimensions, results indicated that instruction using audience response systems has a positive effect compared to regular mathematics classes.

Practitioner notes

What is already known about this topic

• Developing an understanding of fractions is one of the most challenging concepts in elementary mathematics and a solid predictor of future achievements in mathematics.
• Learning analytics (LA) has the potential to provide quality, functional data for assessing and supporting learners' difficulties.
• Audience response systems (ARS) are one of the most practical ways to collect data for LA in classroom environments.
• There is a scarcity of field research implementations on LA mediated by ARS in real contexts of elementary school classrooms.

What this paper adds

• Empirical evidence about how LA-based formative assessments can enable personalised homework to support student understanding of fractions.
• Personalised homework based on an LA-based formative assessment improves the students' comprehension of fractions.
• Using ARS for the teaching of fractions has a positive effect in terms of student motivation.

Implications for practice and/or policy

• Teachers should be given LA/ARS tools that allow them to quickly provide students with personalised mathematical instruction.
• Researchers should continue exploring these potentially beneficial educational implementations in other areas.

     

Designing for impact: a conceptual framework for learning analytics as self-assessment tools

Abstract

Although it is frequently claimed that learning analytics can improve self-evaluation and self-regulated learning by students, most learning analytics tools appear to have been developed as a response to existing data rather than with a clear pedagogical model. As a result there is little evidence of impact on learning. Even fewer learning analytics tools seem to be informed by an understanding of the social context and social practices within which they would be used. As a result, there is very little evidence that learning analytics tools are actually impacting on practice. This paper draws on research in self-regulated learning and in the social practices of learning and assessment to clarify a series of design issues which should be considered by those seeking to develop learning analytics tools which are intended to improve student self-evaluation and self-regulation. It presents a case study of how these design issues influenced the development of a particular tool: the Learning Companion.     

Linking a learning progression for natural selection to teachers' enactment of formative assessment

Learning progressions, or representations of how student ideas develop in a domain, hold promise as tools to support teachers' formative assessment practices. The ideas represented in a learning progression might help teachers to identify and make inferences about evidence collected of student thinking, necessary precursors to modifying instruction to help students advance in their learning. The study reported in this article took the novel approach of using a learning progression for natural selection to support teachers' enactment of formative assessment. Sources of data include interviews and videotapes of six high school biology teachers leading assessment conversations around the same formative assessment questions. Results indicate that while teachers picked out and made inferences about student ideas related to the learning progression during assessment conversations, they did not use all parts of the learning progression in the same way. Furthermore, several of the teachers seemed to use the learning progressions simply as catalogs of misconceptions to be “squashed” rather than drawing upon the developmental affordances offered by a learning progression. Results are framed in terms of the utility of learning progressions as supports for classroom practice. © 2012 Wiley Periodicals, Inc. J Res Sci Teach 49: 1181–1210, 2012     

Influence of an integrated learning diagnosis and formative assessment-based personalized web learning approach on students learning performances and perceptions

The advancement of computer and communication technologies has enabled students to learn across various real-world contexts with supports from the learning system. In the meantime, researchers have emphasized the necessity of providing personalized learning guidance or support by considering individual students’ status and needs in order to improve their learning performance. Based on this perspective, this study proposes a formative assessment-based approach for improving the learning performance of students in a personalized learning environment. An integrated learning diagnosis and formative assessment-based personalized web learning system was developed based on this approach. To evaluate the effectiveness of the proposed approach, an experiment was conducted in an elementary school mathematics course in Thailand. The experimental results showed that (1) the students learning with the proposed system revealed significantly better learning achievement and learning perceptions than those learning with the conventional learning system and (2) students’ learning perceptions of the system had significant relationships with their learning achievement.     

Formative assessment strategies for students' conceptions—The potential of learning analytics

Formative assessment is considered to be helpful in students' learning support and teaching design. Following Aufschnaiter's and Alonzo's framework, formative assessment practices of teachers can be subdivided into three practices: eliciting evidence, interpreting evidence and responding. Since students' conceptions are judged to be important for meaningful learning across disciplines, teachers are required to assess their students' conceptions. The focus of this article lies on the discussion of learning analytics for supporting the assessment of students' conceptions in class. The existing and potential contributions of learning analytics are discussed related to the named formative assessment framework in order to enhance the teachers' options to consider individual students' conceptions. We refer to findings from biology and computer science education on existing assessment tools and identify limitations and potentials with respect to the assessment of students' conceptions.

Practitioner notes

What is already known about this topic

• Students' conceptions are considered to be important for learning processes, but interpreting evidence for learning with respect to students' conceptions is challenging for teachers.
• Assessment tools have been developed in different educational domains for teaching practice.
• Techniques from artificial intelligence and machine learning have been applied for automated assessment of specific aspects of learning.

What does the paper add

• Findings on existing assessment tools from two educational domains are summarised and limitations with respect to assessment of students' conceptions are identified.
• Relevent data that needs to be analysed for insights into students' conceptions is identified from an educational perspective.
• Potential contributions of learning analytics to support the challenging task to elicit students' conceptions are discussed.

Implications for practice and/or policy

• Learning analytics can enhance the eliciting of students' conceptions.
• Based on the analysis of existing works, further exploration and developments of analysis techniques for unstructured text and multimodal data are desirable to support the eliciting of students' conceptions.

     

A large-scale test of the relationship between procrastination and performance using learning analytics

Abstract

Many studies have found a relationship between students’ self-reported procrastination and their grades. Few studies have used learning analytic data as a behavioural measure of procrastination in order to predict performance, and there is no systematic research on how this relationship may differ across assessments or disciplines. In this study we analyse nine years’ worth of institutional electronic submission records, a total of 73,608 assignment submissions, to examine the relationship between submission time and grades across assignments, students, courses, and disciplines in higher education. A significant negative relationship was found overall, with students who submitted closer to the deadline obtaining lower grades, however the size of the relationship was negligible, accounting for less than 1% of the variance in grades. The relationship varied significantly depending on student, assignment, course and discipline.     

Classroom assessment as the co-regulation of learning

ABSTRACT

Until recently, the classroom assessment literature has emphasized the role of teachers and tests, for example investigating teachers’ assessment practices or the quality of classroom tests and other assessments. In contrast, current understandings of teaching and learning emphasize the role of students, as well as the complex interactions between teachers, students, and contexts. We use the literature review method to give substance to a theory of classroom assessment as the co-regulation of learning by teachers, students, instructional materials, and contexts. We organize the literature using a version of Pintrich and Zusho’s theory of the phases and areas of the self-regulation of learning, expanded to include the co-regulation of learning, in order to demonstrate how classroom assessment is related to all aspects of the regulation of learning. We conclude that this is a useful expansion for the field.     

Comparing student learning experiences of in-text commentary and rubric-articulated feedback: strategies for formative assessment

This study compares students’ experiences of two types of criteria-based assessment: in-text commentary and rubric-articulated feedback, in an assessment design combining the two feedback channels. The main aim is to use students’ responses to shed light on how feedback strategies for formative assessment can be optimised. Following action research methodology, the study discusses key categories of student responses from three sources: reflective texts, a questionnaire, and interviews. Results show that different functions were attributed to the two feedback channels: in-text commentary to lower-order concerns related to language proficiency, and rubric-articulated feedback to higher-order concerns related to an overview of writing achievement. We argue that the different functions have the potential of creating a sufficiently balanced assessment design with the possibility of serving both short-term and continuous learning goals. On the other hand, some students found it difficult to navigate between the two feedback channels. The article therefore ends with a ‘lessons learned’ section where we list possible ways in which the current assessment design can be improved for optimal use of the synergy effects emanating from a combination of in-text commentary and rubric-articulated feedback for formative purposes.     

Effective formative e-assessment of student learning: a study on a statistics course

The process of formative assessment in universities has the potential to engage students in reflection and to take greater ownership of their learning. We report on a study involving 104 students taking a statistics course in a degree programme in rehabilitation science in Hong Kong. The assessment strategy was redesigned to include a weekly online formative assessment activity for 10 weeks. In this case, it was observed that there was a high participation rate across weeks. An end-of-course online survey was conducted to explore the learning experiences during formative assessment of the cohort, and students reported high satisfaction with the process, particularly how this contributed to a change in their learning approach. Students not only showed regular commitment of time on tasks but also took active steps to find answers for themselves and with their peers.     

Feedback as dialogue: exploring the links between formative assessment and social software in distance learning

The paper explores the relationship between formative assessment and social software. Formative assessment practices though beneficial for student learning become marginalised and constrained in open and distance learning environments in higher education. Feedback is a key factor in formative assessment and learners can benefit from the deployment of emerging technologies and the opportunities for participation and dialogue afforded by social software. This paper explores and proposes a conceptual framework for this relationship. The claim is that the social dimensions of emerging technologies – specifically, blogs and wikis – allow for formative assessment practices to be re‐invented or at the very least facilitated by essentially participative and student‐focussed interventions. A comparison of these technologies against formative assessment mechanisms identifies the types of processes that these new tools might best support to encourage effective feedback approaches that both empower the learner and enhance their learning experience.     

Teacher agency within the context of formative teacher assessment: an in-depth analysis

Teachers’ agency has an effect on their own learning process at the workplace. In this study we explored the extent to which teachers participating in a formative teacher assessment procedure developed a sense of agency. We investigated not only whether teachers participating in a such an assessment procedure experienced agency and thus felt in control of the learning process and able to pursue their learning objectives, but also whether agency was visible, by looking at decision-making in real time: did teachers take an active role in their own assessment, especially regarding the learning objectives to be pursued, during the assessment meetings? We found that teachers experienced a high level of agency while participating in the assessment procedure, but did not consistently show this during the assessment procedure.     

Investigating the dynamics of formative assessment: relationships between teacher knowledge,assessment practice and learning

This exploratory study of elementary school science examines questions central to policy, practice and research on formative assessment: What is the quality of teachers’ content-pedagogical and assessment knowledge? What is the relationship between teacher knowledge and assessment practice? What is the relationship between teacher knowledge, assessment practice and student learning? Drawing on multiple measures, hierarchical linear modelling and path analysis, results suggest that despite weaknesses in teachers’ content-pedagogical and assessment knowledge, teachers’ formative assessment practices are positively related to student learning. Relationships between teachers’ knowledge and assessment practices are mixed. Findings underscore both the potential and challenge of bringing effective formative practice to fruition as well as the need for continued research.     

Supporting adaptive learning pathways through the use of learning analytics: developments,challenges and future opportunities

Learning Analytics (LA) and adaptive learning are inextricably linked since they both foster technology-supported learner-centred education. This study identifies developments focusing on their interplay and emphasises insufficiently investigated directions which display a higher innovation potential. Twenty-one peer-reviewed studies are identified via a systematic review and classified based on the dimensions of a proposed framework. The findings of the review suggest that interesting work has been carried out during the last years on the topic. We conclude with a need for more research on the topic in specific learning domains and settings. Recommendations include: a clear strategy for adaptation augmented by LA, the combination of on-task with pre-task measures, and the combination of system-controlled adaptation with user-controlled adaptation. Future trends include the emergence of constructivist-collaborative environments that provide insightful models of complex student behaviour.     

The conceptualisation of a modified formative assessment model for non‐verbal students with autism and severe learning difficulties

The popularity of formative assessment has increased since the publication of work by Black and Wiliam in 1998. Even though it is a useful teaching tool, in most cases it has only been possible to use it for students with high levels of cognitive and communicative ability. The aim of this article is to propose a modified, personalisable model of formative assessment for non‐verbal students with autism and severe learning difficulties. Five students with autism and severe learning difficulties participated in systematic video observations over a period of eight weeks, during which student behaviour and attainment were recorded. The behaviour checklist gave an accurate representation of the students' level of engagement and predicted attainment, but differences in passive learning objectives (those requiring passive co‐operation) and active ones (those requiring active contribution) were noted. The introduction of engaging resources improved engagement, but tangible rewards had a negative effect on attainment. Praise had a positive effect on engagement and attainment.     

形成性评价与终结性评价并重高职英语测评体系构建

高职英语作为专业课程的辅助课程,主要体现出其应用能力,沿用原有的终结式考试已经不能体现出学生对本门课技能掌握情况。鉴于此教师对考核方式进行改革,加强考核引导和过程控制,注重学生英语应用能力培养及提高,建立基于过程控制的高职英语课程形成性考核与终结性考核并重的考核体系,深化高职英语课程改革,促进高职学生英语应用能力的提高。