Small group discussion as a key component in online assessment training for enhanced student learning in web-based peer assessment

The purpose of this study is to examine the effect of online assessment training, with synchronous group discussion as a key component, on subsequent web-based peer assessment results. Participants included 81 college students, mostly women, taking a business writing class. After initial submission of a draft counter-offer letter, they completed online assessment training by engaging in a consensus-seeking, synchronous group discussion of the rubric-based ratings they gave to sample counter-offer letters. They then engaged in web-based assessment of randomly assigned peer counter-offer letters and upon receiving peer feedback, submitted revision of their own counter-offer letter. The discussion groups were randomly assigned to either anonymous (using a pseudo name) or identified conditions (using real name). Findings indicate that the quality of student counter-offer letters improved after online assessment training, and improved even more after web-based peer assessment. There were no significant differences between anonymous discussion groups and identified groups.     

Using seen examination questions in Economics: an evaluation

This paper presents a detailed quantitative analysis of the use of seen questions within examinations in Economics option modules at one UK university. 4622 marks for seen and unseen questions awarded over a period of three years were analysed; the impact of personal characteristics was analysed using a sub-sample. It was found that the number of attempted answers to the seen question was somewhat lower than might be expected if students had engaged in random selection of examination questions and the proportion of students who attempted to answer the seen question varied both across modules and years. The majority of students who answered the seen question did so in only one module, and answering the seen question in more than one module did not have a positive impact on obtaining a good mark for the seen question. Most notably students who answered seen questions were more likely to obtain a good mark for seen than for unseen questions. The personal factors found to correlate with the likelihood of answering a seen question were gender, performance in Economics core modules, and type of school attended. Reasons for the findings are discussed and set within the context of innovation in Economics assessment.     

Using the web to encourage student-generated questions in large-format introductory biology classes

Students rarely ask questions related to course content in large-format introductory classes. The use of a Web-based forum devoted to student-generated questions was explored in a second-semester introductory biology course. Approximately 80% of the enrolled students asked at least one question about course content during each of three semesters during which this approach was implemented. About 95% of the students who posted questions reported reading the instructor's response to their questions. Although doing so did not contribute to their grade in the course, approximately 75% of the students reported reading questions posted by other students in the class. Approximately 60% of the students reported that the Web-based question-asking activity contributed to their learning of biology.     

Development and evaluation of a student-centred multimedia self-assessment instrument for social-communicative competence

Communication plays an important role in manyprofessional contexts. This is especially truefor students in the field of social work. Theaim of this study was to develop formative,self-regulated multimedia self-assessment ofsocial-communicative competencies for socialwork students. First, a pilot study wasconducted to gain insight into the students'specific characteristics. This insight was usedto design guidelines for the development of theassessment instrument in order to tune these tothe students' perceptions, instructionalpreferences, and personalities and thusenhancing the students' enthusiasm to use theself-assessment instrument. This might increasethe chance for successful implementation ofthis new form of assessment.A first version of a multimedia test wasdeveloped. A quality expert meeting wasorganised to gain insight into expertjudgements on the quality of the test and toobtain indications for improvement of theassessment. A second version of the test wasconstructed and put on the Internet. Nearly 400students completed the assessment and expressedtheir opinions on this new way of communicationassessment.We found it was possible to testsocial-communicative competence by means ofmultimedia, with the help of digital video. Theuse of Internet makes assessments available atany time to fit curriculum needs and alsoresolves time and space constraints. Ourconclusion was that the multimedia test isreasonably valid. All students reported havingliked the multimedia assessment.     

Using email for formative assessment with professional doctorate students

This article reports on aspects of a recent research and development project in doctoral education. It focuses on the use of email for tutor’s formative assessment within the early stages of a Professional Doctorate in Education (EdD) in an English university. Its case study methodology included participant observation of the programme workshops, critical discourse analysis of the email texts, and two series of in‐depth, semi‐structured student interviews. The tutor whose feedback was analysed had previously researched and theorised formative assessment, so the research allowed his previous theoretical insights to be explored and developed in an early doctoral context. The article concludes by discussing the problematic power of feedback at this level, given the culturally constructed associations of feedback with summative assessment, and implications for supervisory practice.     

Exploring children's learning stories as an assessment method for research and practice

The purpose of this study was to apply and explore, in the context of an Icelandic preschool, the methods developed by the New Zealand Te Whaariki curriculum to assess children's well‐being and learning dispositions. This article describes the process of piloting learning stories as a research method, where researchers and preschool teachers cooperate in generating data. Data were generated by means of interviews, observations and documentation, following a research process in which learning stories were documented, reflected upon and analysed by the researchers and teachers, where possible in cooperation with the four‐ to five‐year‐old children. The learning stories were constructed and analysed using the five domains of learning dispositions identified in the Te Whaariki curriculum. The study asks how a collaborative approach to assessing children's well‐being and learning dispositions could be used in preschools both for professional learning and for research purposes. The findings show children as skilful communicators who are capable of finding solutions and reacting to various situations arising from the social context of the preschool. The study demonstrates that documenting children's learning stories enables researchers to focus on children's strengths and capabilities, and adds to preschool teachers' knowledge of individual children and groups.     

An empirical exploration of metacognitive assessment activities in a third-year civil engineering hydraulics course

Threshold concepts are transformative, integrative, and provocative; understanding these difficult concepts allows students to be capable of solving advanced problems. This investigation and evaluation of a metacognitive curricular approach explore variation in students' and teachers' discernment of structural complexity of concepts and its potential for enhancing students' learning and conceptual understanding of threshold concepts. Three trials of a metacognitive assessment activity administered to two cohorts of a civil engineering course (n?=?276 and n?=?264) were investigated. Students were presented with several answers (varying in structural complexity) to a question about a threshold concept and asked to mark each response. Quantitative analyses compared students' and teachers' marking schemes within and across trials, and qualitative analyses explored students' written reflections following the activity. Students' justifications for their marking schemes, their reflections on the activity's usefulness, and the convergence of students' and teachers' marking schemes suggest that the activity supported deep forms of student learning.     

A framework for classroom assessment,learning, and self-regulation

ABSTRACT

We present a conceptual framework that leverages synergies between classroom assessment (CA) practices and self-regulated learning (SRL) theory to support academic growth and instruction. We articulate the processes shared by CA and SRL, drawing on a model of SRL with three phases: forethought, performance, and self-reflection. We blend this SRL model with CA to create the CA:SRL framework in four stages: (1) pre-assessment, (2) the cycle of learning, doing, and assessing, (3) formal assessment, and (4) summarizing assessment evidence. We elucidate how SRL processes are involved at each stage and can be drawn on to support learning development and teacher understanding and co-regulation of learning. This framework is important in that it depicts how assessment and learning processes interact dynamically for both teachers and students in classrooms, and demonstrates that such interactions encompass the full breadth of purposes in CA, from planning through summation of evidence.     

Time and Performance in Online Learning: Applying the Theoretical Perspective of Metacognition

Educational psychologists have found that metacognitive calibration predicts learning outcomes in self‐regulated learning. In this research the authors apply theories of metacognition from educational psychology and postulate that metacognitive calibration influences learning time and performance in online learning. Data gathered from 230 college students who completed online self‐regulated assignments in an introductory information systems course confirms that higher calibration accuracy leads to better performance on assignments, which contributes to higher exam grades. In addition, higher calibration accuracy also reduces the amount of time students spent on the online assignments. However, time spent on the assignments is not a mediator of the effect of calibration on learning performance. This research presents pioneering work in examining a holistic model that assesses the impact of metacognition in online learning on both assignment and exam level learning outcomes using field data collected through a natural learning setting. The findings highlight the need to include metacognitive calibration as one of the learner characteristics in research models addressing student performance in online learning.     

Using patchwork texts in assessment: clarifying and categorising choices in their use

An emerging form of assessment is the patchwork text, an application of assessment for learning, as opposed to assessment of learning. A framework for making choices in how to use patchwork texts in higher education courses is developed, drawing on prior research on, and use of, patchwork texts. These choices reflect the fundamental objectives of patchwork texts, namely continuous learning, deep learning, integrated understanding and critical self‐reflection, while also offering options for patchwork texts which can be tailored to particular courses and individual preferences.     

Science assessments for all: Integrating science simulations into balanced state science assessment systems

This article reports on the collaboration of six states to study how simulation‐based science assessments can become transformative components of multi‐level, balanced state science assessment systems. The project studied the psychometric quality, feasibility, and utility of simulation‐based science assessments designed to serve formative purposes during a unit and to provide summative evidence of end‐of‐unit proficiencies. The frameworks of evidence‐centered assessment design and model‐based learning shaped the specifications for the assessments. The simulations provided the three most common forms of accommodations in state testing programs: audio recording of text, screen magnification, and support for extended time. The SimScientists program at WestEd developed simulation‐based, curriculum‐embedded, and unit benchmark assessments for two middle school topics, Ecosystems and Force & Motion. These were field‐tested in three states. Data included student characteristics, responses to the assessments, cognitive labs, classroom observations, and teacher surveys and interviews. UCLA CRESST conducted an evaluation of the implementation. Feasibility and utility were examined in classroom observations, teacher surveys and interviews, and by the six‐state Design Panel. Technical quality data included AAAS reviews of the items' alignment with standards and quality of the science, cognitive labs, and assessment data. Student data were analyzed using multidimensional Item Response Theory (IRT) methods. IRT analyses demonstrated the high psychometric quality (reliability and validity) of the assessments and their discrimination between content knowledge and inquiry practices. Students performed better on the interactive, simulation‐based assessments than on the static, conventional items in the posttest. Importantly, gaps between performance of the general population and English language learners and students with disabilities were considerably smaller on the simulation‐based assessments than on the posttests. The Design Panel participated in development of two models for integrating science simulations into a balanced state science assessment system. © 2012 Wiley Periodicals, Inc. J Res Sci Teach 49: 363–393, 2012     

The assessment cycle: a model for learning through peer assessment

This paper advances a model describing how peer assessment supports self-assessment. Although prior research demonstrates that peer assessment promotes self-assessment, the connection between these two activities is underspecified. This model, the assessment cycle, draws from theories of self-assessment to elaborate how learning takes place through peer assessment. The model is applied to three activity structures described in the literature to analyse their potential to support learning by promoting self-assessment. Broadly speaking, the model can be used to understand learning that takes place in a variety of peer assessment activities: marking/grading, analysis, feedback, conferencing and revision. This approach contrasts most studies on peer assessment, which have focused on calibration of instructor and peer grades, rather than learning opportunities.     

The relationship between student-generated learning issues and self-study in problem-based learning

A major assumption of problem-based learning (PBL) is that learning issues, generated by students while discussing a problem, are used as guides for self-directed learning activities. This assumption, though basic to PBL, has never been tested. At the University of Limburg, the Netherlands, two procedures have been developed that reflect the extent to which students are able to identify important learning issues given a particular problem, and whether subsequent, independent, learning corresponds with these learning issues. The focus of the present article will be on the relationship between the two. We have explored to what extent student-generated learning issues are a major factor influencing the nature of students' self-study, or whether other factors may be involved in decisions on what to study and how much time to spend on topics selected. First, the production of learning issues was studied and represented as the percentage of overlap between learning issues raised by students and pre-set faculty objectives for each problem. The second procedure consisted of the administration of a Topic Checklist (TOC) which purports to measure students' actual self-directed learning activities. The TOC consists of a list of topics specifying the intended course content. Students were asked to indicate on a five-point Likert scale how much time they had spent studying each topic and to what degree they had mastered that topic. Third, learning issues and TOC topics were compared directly in a qualitative sense. Comparisons between the procedures revealed that a low proportion of variance of TOC scores could be predicted from the percentage of faculty objectives identified for each problem and the direct match between learning issues and TOC scores. It is concluded that scrutinizing student-generated learning issues and topics covered during self-study may provide information about what content is covered by students in tutorial groups. The discrepancy between the results of the measurements suggests, however, that learning issues produced during group discussion are not the sole source on which students base self-study decisions. Several other factors may be involved, such as tutor guidance, content already covered in previous units, issues raised during sessions with resource persons, and the nature of the learning resources available. Therefore, the relationship between learning issues and content covered during self-study is not as straightforward as is suggested.     

Promoting student-centred forms of learning across an entire university

Many academics see themselves primarily as experts in their discipline and hold content-oriented conceptions of teaching. It can then be difficult to persuade them to adopt forms of teaching incorporating active student engagement, even though there is evidence for the effectiveness of such forms of learning. This article describes a university-wide initiative to promote student-centred forms of teaching and learning. The campaign included: models of good practice from award-winning teachers, compulsory teachers training courses for new junior teachers and teaching assistants, projects funded by teaching development grants, a diagnostic programme-level survey accompanied by counselling over results and programme reviews. Evidence of impact comes from the overall university-level results of the programme survey, which showed marked increases, over a 2 year period, on each of the nine scales used to obtain feedback on the quality of the teaching and learning environment.     

Trust,distrust and their impact on assessment reform

This paper puts forward the case that one of the factors constraining principled learning‐oriented assessment practices is lack of trust. It examines a number of assessment dimensions in which trust or distrust plays a role. These issues are illustrated via a discussion of two different iterations of the same module taught in a teacher education institution. Through this example, the author analyses how accountability forces and distrust created an atmosphere which constrained the use of innovative assessment methods. The paper discusses how trust might be developed, some of the barriers arising, and the relationship between trust and good assessment practices. It concludes by sketching some possible avenues for further research into stakeholders’ perceptions of the interplay between trust and assessment.     

Sustainable assessment for lifelong learning

This paper explores the alignment of assessment practices in universities and lifelong learning as a key process and outcome for expansive student development. It outlines Boud’s approach to assessment, operationalises this to analyse practices in two contrasting national contexts: the sociology departments of the Midlands University in the UK and Nam Du University in Vietnam, and reframes this framework as a guide to improving practices and better-supporting lifelong learning. The paper thus contributes to recent debates about sustainable assessment and how to change assessment to better support students for lifelong learning in different countries. The findings suggest that although more elements were found at Midlands University than Nam Du University, assessment in both cases was an imperfect realisation of this framework. Nonetheless, the paper argues that the framework can offer guidance for changes to align assessment practices with lifelong learning, but to do this it should be adapted to the education context and also expanded to include explicit social commitments to others for expansive lifelong learning.     

Beyond intelligent tutoring systems: Using computers as METAcognitive tools to enhance learning?

Framed by the existing theoretical andempirical research on cognitive and intelligenttutoring systems (ITSs), this commentaryexplores two areas not directly or extensivelyaddressed by Akhras and Self (this issue). Thefirst area focuses on the lack of conceptualclarity of the proposed constructivist stanceand its related constructs (e.g., affordances,situations). Specifically, it is argued that aclear conceptualization of the novelconstructivist stance needs to be delineated bythe authors before an evaluation of theirambitious proposal to model situationscomputationally in intelligent learningenvironments (ILEs) can be achieved. The secondarea of exploration deals with the similaritiesbetween the proposed stance and existingapproaches documented in the cognitive,educational computing, and AI in educationliterature. I believe that the authors are at acrossroads, and that their article presents aninitial conceptualization of an important issuerelated to a constructivist-based approach tothe computational modeling of situations inILEs. However, conceptual clarity isdefinitively required in order for theirapproach to be adequately evaluated and used toinform the design of ILEs. As such, I invitethe authors to re-conceptualize their frameworkby addressing how their constructivist stancecan be used to address a particular researchagenda on the use of computers as metacognitivetools to enhance learning.