A methodological framework for practice-based research in networked learning

This paper presents a methodological frameworkfor practice-based research in the field ofnetworked learning, based on the author'sapproach to carrying out a case studyevaluation of a networked professionaldevelopment initiative for information servicesstaff in higher education. The theoreticalcase for the framework is made with referenceto principles associated with constructivistprogramme evaluation as well as withinterpretivist and critical traditions inaction research. The purposes and politics ofthis approach, and the question of validationcriteria for claims about its effectiveness forbuilding theory, as well as for evaluating andimproving practice, are discussed. Issues todo with using computer-mediated communicationas a medium for participatory research are alsoraised. An overview is given of the way thisresearch approach was implemented in practice(including the use of a combination of on-lineand face-to-face data collection methods) andsome methodological issues that arose in thecourse of carrying it out are highlighted. Thepaper concludes by suggesting that as amethodology for evaluating, theorising andimproving practice, `networked action research'also offers an approach to pedagogical design,most obviously in the context of networkedprofessional development for educators andlearner support staff.     

Interactive learning environments and games

Undergraduate psychometrics classes often use computer-intensive active learning projects. However, little research has examined active learning or computer-intensive projects in psychometrics courses. We describe two computer-intensive collaborative learning projects used to teach the design and evaluation of psychological tests. Course evaluations were significantly above the department average, and students perceived the projects as effective in meeting course objectives and improving computer skills. Grades on these projects were unrelated to self-reported computer background, suggesting that all students can succeed at computer-intensive projects when instructors provide adequate support and create a collaborative learning environment.     

Assessing assessment quality: Criteria for quality assurance in design of (peer) assessment for learning - A review of research studies

The interest in assessment for learning (AfL) has resulted in a search for new modes of assessment that are better aligned to students’ learning how to learn. However, with the introduction of new assessment tools, also questions arose with respect to the quality of its measurement. On the one hand, the appropriateness of traditional, psychometric criteria is being questioned in the context of assessment for learning. On the other hand, it has been argued that new criteria need to be formulated to acknowledge the unique features of assessment for learning. The issue dealt with in this review is what quality criteria are specifically relevant to AfL. Studies using (peer) assessment for learning were evaluated with regard to use of quality criteria under two perspectives: their recognition of educational measurement criteria, as well as their consideration of student involvement in the assessment of learning. The selected studies were analyzed to determine what specific criteria were utilized in (successive parts of) the assessment cycle (i.e., the process of construction, administration and follow up of an assessment). Our results indicate that quality criteria are dissimilarly connected to the distinct steps of an assessment cycle, which holds as well for the attention given to student involvement in assessment for learning.     

Measuring learning in serious games: a case study with structural assessment

The effectiveness of serious games is often measured with verbal assessment. As an alternative we propose Pathfinder structural assessment (defined as measuring the learners’ knowledge organization and compare this with a referent structure) which comprises three steps: knowledge elicitation, knowledge representation and knowledge evaluation. We discuss practical and theoretical considerations for the use of structural assessment and showcase its application with the game Code Red: Triage. Results suggest that structural assessment measures an individual’s understanding of a domain at least differently from verbal assessment. While verbal assessment may provide a more nuanced picture regarding declarative and procedural knowledge, structural assessment may add an in-depth understanding of the concepts that are regarded important in a domain. In the Discussion we propose four guidelines to effectively use structural assessment in serious games: (1) Determine the appropriateness of the domain for structural assessment, (2) select an appropriate referent for the target group(s), (3) select the number of concepts needed for structural assessment, and (4) consider the analysis of the graphical knowledge representations to obtain in-depth information about the quality of the knowledge structures.     

The assessment of learning skills in nursing student selection: A scoping review

Abstract

Higher education student selection has significant societal, institutional and individual impacts. Thousands of applicants apply only for nursing, one of the major higher education disciplines. As the nursing profession is characterised by cognitive requirements, higher education institutions assess the learning skills of nursing applicants. However, there has been no comprehensive analysis of learning skills assessment for nursing student selection. The purpose of this scoping review was to describe the assessment of learning skills in undergraduate nursing student selection. Five databases were systematically searched, and 24 studies published between 2006 and 2016 were included. Learning skills were most commonly assessed using standardised tests in the areas of language and communication, reasoning, mathematics and natural sciences. Overall scores of onsite selection methods were found to best predict future academic performance. The results indicate that higher education institutions may benefit from comprehensive assessment of learning skills in their selection processes. This assessment should focus on a wider range of cognitive aptitudes, including reasoning skills. This review focussed on nursing education, but the results may benefit other higher education disciplines due to the generic nature of learning skills and similar cognitive requirements of higher education studies. The results support the development of more comprehensive and valid methods for assessing learning skills.     

Improving assessment of learning and for learning

The focus of this article is recent work by the Assessment Reform Group (ARG) on the role of teachers' judgements in the summative use of assessment. A brief overview of the early work of the ARG is followed by discussion of the desirable properties of assessment for summative uses. The work of the ARG's Assessment Systems for the Future project provided evidence and arguments concerning the validity, reliability, impact and cost of tests and of summative assessment by teachers. Whilst there is ample evidence that the teachers' judgements are more valid than, and equally reliable as, tests, there is a danger of unwanted impact on teaching as long as results are used for ‘high stakes’ evaluation of teachers and schools. Implications for policy include an end to the practice of using the results of pupils' summative assessment, however they are derived, as the sole basis for target setting and school accountability.     

Committing to quality learning through adaptive online assessment

With advances in computer‐based technologies and the emergence of e‐learning, there are unprecedented opportunities to reconsider assessment of learning (and, axiomatically, of teaching) and how this can be undertaken. One approach is adaptive assessment. Although it has existed in the tertiary environment since the time of the oral examination, advanced technologies allow much fuller exploitation of the possibilities inherent in a dynamic system of testing that responds to the user. Having described the characteristics of adaptive assessment, this paper considers how it can achieve significant pedagogical aims within the sector. The paper differentiates between adaptive assessment to assist learning and adaptive assessment to assess achievement. How adaptive assessment can be put in place and salient issues, such as security and system integrity, when such assessment is used for credit, are then discussed. The paper concludes that the capability exists but it has yet to be exploited within higher education as a viable approach to assessment and as a contributor to quality learning.     

Uncertainty and engagement with learning games

Uncertainty may be an important component of the motivation provided by learning games, especially when associated with gaming rather than learning. Three studies are reported that explore the influence of gaming uncertainty on engagement with computer-based learning games. In the first study, children (10–11 years) played a simple maths quiz. Participants chose their preferred reward for a correct answer prior to seeing each question. They could either receive a single point or toss an animated coin to receive 2 points for heads or none for tails. A preference for the uncertain option was revealed and this increased during the quiz. The second study explored the discourse around learning when pairs of participants (13–14 years) competed against the computer in a science quiz. Progress depended on the acquisition of facts but also on the outcomes of throwing dice. Discourse was characterised by a close intermingling of learning and gaming talk without salient problematic constructions regarding fairness when losing points due to gaming uncertainty. A final experiment explored whether, in this type of game, the uncertainty provided by the gaming component could influence players’ affective response to the learning component. Electrodermal activity (EDA) of 16 adults was measured while they played the quiz with and without the element of chance provided by the dice. Results showed EDA when answering questions was increased by inclusion of gaming uncertainty. Findings are discussed in terms of the potential benefits of combining gaming uncertainty with learning and directions for further research in this area are outlined.     

Linking a learning progression for natural selection to teachers' enactment of formative assessment

Learning progressions, or representations of how student ideas develop in a domain, hold promise as tools to support teachers' formative assessment practices. The ideas represented in a learning progression might help teachers to identify and make inferences about evidence collected of student thinking, necessary precursors to modifying instruction to help students advance in their learning. The study reported in this article took the novel approach of using a learning progression for natural selection to support teachers' enactment of formative assessment. Sources of data include interviews and videotapes of six high school biology teachers leading assessment conversations around the same formative assessment questions. Results indicate that while teachers picked out and made inferences about student ideas related to the learning progression during assessment conversations, they did not use all parts of the learning progression in the same way. Furthermore, several of the teachers seemed to use the learning progressions simply as catalogs of misconceptions to be “squashed” rather than drawing upon the developmental affordances offered by a learning progression. Results are framed in terms of the utility of learning progressions as supports for classroom practice. © 2012 Wiley Periodicals, Inc. J Res Sci Teach 49: 1181–1210, 2012     

Passion play: Will Wright and games for science learning

Researchers and instructional designers are exploring the possibilities of using video games to support STEM education in the U.S., not only because they are a popular media form among youth, but also because well-designed games often leverage the best features of inquiry learning. Those interested in using games in an educational capacity may benefit from an examination of the work of video game designer Will Wright. Wright designs through a constructivist lens and his open-ended, sandbox games (SimCity, The Sims, Spore) present wide ??possibility spaces?? that allow players to exercise their critical thinking and problem solving skills. His games invoke a delight in discovery that inspire creative acts and interest-driven learning both during and outside of the game. Finally, he reminds us that failure-based learning is a viable strategy for building expertise and understanding.     

Leveraging competency framework to improve teaching and learning: A methodological approach

A number of engineering education programs have defined learning outcomes and course-level competencies, and conducted assessments at the program level to determine areas for continuous improvement. However, many of these programs have not implemented a comprehensive competency framework to support the actual delivery and assessment of an individual course. This paper highlights how a competency framework can be used across the life cycle of a course to effectively deliver and assess course content, and give valuable, timely feedback to students thus, improving teaching, student motivation and learning. A framework for leveraging course competencies during course design and delivery is presented, and addresses the following five phases of a course, namely, content design, assessment design, content delivery and assessment, assessment results analysis and feedback, and content review. Using a large first-year core course of the BSc (Information Systems Management) program, at School of Information Systems, Singapore Management University, Singapore—called Information Systems Software Foundation (ISSF)—as an example, this paper shows how course competencies support the framework’s five phases. Data from a student survey indicates that the framework has contributed to enhancing their motivation to learn, provides enhanced learning experiences in terms of helping students prepare for each assessment, providing better feedback by raising awareness of what they know and do not know, and revisiting topics that relate to competencies that have not been fully acquired. Results from interviewing instructors revealed that the competency framework provides valuable and timely feedback on how students are performing, and additionally what changes are required to both the content and method of delivery in order to improve teaching. This contributes towards more effectively closing the “teaching and learning loop”.     

A framework for classroom assessment,learning, and self-regulation

ABSTRACT

We present a conceptual framework that leverages synergies between classroom assessment (CA) practices and self-regulated learning (SRL) theory to support academic growth and instruction. We articulate the processes shared by CA and SRL, drawing on a model of SRL with three phases: forethought, performance, and self-reflection. We blend this SRL model with CA to create the CA:SRL framework in four stages: (1) pre-assessment, (2) the cycle of learning, doing, and assessing, (3) formal assessment, and (4) summarizing assessment evidence. We elucidate how SRL processes are involved at each stage and can be drawn on to support learning development and teacher understanding and co-regulation of learning. This framework is important in that it depicts how assessment and learning processes interact dynamically for both teachers and students in classrooms, and demonstrates that such interactions encompass the full breadth of purposes in CA, from planning through summation of evidence.     

Transforming professional learning and practice in assessment for learning

Assessing student learning is a complex process requiring teachers to have deep knowledge of the curriculum, assessment, and pedagogy. Changing political climates mean that teachers are expected to respond to new approaches or systems and adjust their classroom practice accordingly. Teachers often engage in professional learning (PL) to assist their knowledge and classroom practice, but what impact does PL have on their assessment practice as well as student learning and achievement?

This paper examines optimal PL principles, the New Zealand assessment policy context, and application of assessment for learning principles in a nation-wide PL programme. Empirical data presented in relation to the five levels of Guskey's (2002) Guskey, T. R. (2002). Does it make a difference? Evaluating professional development. Educational Leadership, 59(6), 45–51.[Web of Science ®] , [Google Scholar] model evaluating PL indicate successful outcomes at the classroom level. However, application of Carless’ (2005) Carless, D. (2005). Prospects for the implementation of assessment for learning. Assessment in Education: Principles, Policies & Practices, 12(1), 39–45.[Taylor & Francis Online] , [Google Scholar] theoretical framework on embedding PL changes in schools reveals an area of neglect: attending to system (macro) level needs. Collaborative partnerships between schools and the wider community are posited as essential determinants of assessment literacies and transformational PL in times of political change.     

Intrinsic integration in learning games and virtual instruction

This paper responds to a 2016 systematic literature review of the research on learning games by Ke (Educ Technol Res Dev 64(2):219–244, 2016). The review paper unpacked the idea of intrinsic integration in learning games, analyzing important emergent themes. The key ideas and the value of this review are discussed in the context of the recent shift to virtual instruction. The limitations, impact, and future implications are also described, with the perspective of this response focused on education theory. Intrinsic integration is an important consideration when designers create the next generation of digital games, and as researchers try to untangle the affordances of different educational games for student learning.