Threats to student evaluative judgement and their management

ABSTRACT

Students’ capacity for making evaluative judgements of their own work is widely acknowledged as central to their learning within programmes as well as being vital to their subsequent professional practice. In higher education literature, the act of evaluative judgement is usually portrayed as a process of deliberative, analytical reasoning requiring student agency and objectivity, typically scaffolded by points of reference such as explicit criteria, rubrics or exemplars. This article challenges this common portrayal of judgement by drawing attention to research from outside higher education on the role of unconscious factors in judgement and decision-making. Drawing from the field of heuristics and bias studies, the article outlines six unconscious factors that have the potential to distort students’ analytical judgement of their work. A recent challenge to the heuristics and bias approach that radically repositions the place of reasoning in judgement is also considered. Since these unconscious factors have received scant attention in higher education literature, the purpose of this article is to draw attention to them, to identify the challenges they pose to current understandings of evaluative judgement, and to outline their implications for enhancing assessment practice.     

Heuristic Reasoning in Chemistry: Making decisions about acid strength

The characterization of students’ reasoning strategies is of central importance in the development of instructional strategies that foster meaningful learning. In particular, the identification of shortcut reasoning procedures (heuristics) used by students to reduce cognitive load can help us devise strategies to facilitate the development of more analytical ways of thinking. The central goal of this qualitative study was thus to investigate heuristic reasoning as used by organic chemistry college students, focusing our attention on their ability to predict the relative acid strength of chemical compounds represented using explicit composition and structural features (i.e., structural formulas). Our results indicated that many study participants relied heavily on one or more of the following heuristics to make most of their decisions: reduction, representativeness, and lexicographic. Despite having visual access to reach structural information about the substances included in each ranking task, many students relied on isolated composition features to make their decisions. However, the specific characteristics of the tasks seemed to trigger heuristic reasoning in different ways. Although the use of heuristics allowed students to simplify some components of the ranking tasks and generate correct responses, it often led them astray. Very few study participants predicted the correct trends based on scientifically acceptable arguments. Our results suggest the need for instructional interventions that explicitly develop college chemistry students’ abilities to monitor their thinking and evaluate the effectiveness of analytical versus heuristic reasoning strategies in different contexts.     

Does authentic assessment assure academic integrity? Evidence from contract cheating data

ABSTRACT

Contract cheating occurs when a student outsources their assessment to a third party, regardless of the third party’s relationship with the student, or whether money is exchanged. In higher education, there is a widespread belief that assessment design is a solution to the problem of contract cheating and that authentic assessment tasks are particularly effective. This research analysed two datasets – 221 assignment orders placed on academic custom writing websites and 198 assessment tasks in which contract cheating was detected – to investigate if authentic assessment can assure academic integrity. The authenticity of assessments was determined using five factors derived from the literature: frequency, fidelity, complexity, real-world impact and feed forward. Our analysis found that assessment tasks with no, some, or all of the five authenticity factors are routinely outsourced by students.     

Support for assessment practice: developing the Assessment Design Decisions Framework

There are many excellent publications outlining features of assessment and feedback design in higher education. However, university educators often find these ideas challenging to realise in practice, as much of the literature focuses on institutional change rather than supporting academics. This paper describes the conceptual development of a practical framework designed to stimulate educators’ thinking when creating or modifying assessments. We explain the concepts that underpin this practical support, including the notions of ‘assessment decisions’ and ‘assessment design phases’, as informed by relevant literature and empirical data. We also present the outcome of this work. The Assessment Design Decisions Framework. This provides key considerations in six categories: purposes, contexts, tasks, interactions, feedback processes and learning outcomes. By tracing the development of the Framework, we highlight complex ways of thinking about assessment that are relevant to those who design and deliver assessment to tertiary students.     

Heuristic principles and cognitive bias in decision making: Implications for assessment in school psychology

Research has indicated that the diagnoses and classification decisions made by professionals are often unreliable. People employ several heuristic strategies when making decisions under conditions of uncertainty. Use of these heuristics can have a detrimental effect on the decision-making process. This paper describes factors that may bias psychoeducational decision making and discusses three heuristic principles that affect decision making. The means by which school psychologists can be made aware of these heuristic principles and encouraged to consider them when making psychoeducational decisions are discussed. Other methods by which bias in the psychoeducational process can be reduced, including the use of statistical and actuarial-based assessment systems, the multitrait multimethod approach to multifactored assessment advanced by Gresham (1983), and direct instruction in reasoning and decision making are also discussed.     

Connecting engagement and focus in pedagogic task design

Teachers of mathematics face a particular tension, which the authors call the planning paradox If teachers plan from objectives, the tasks they set are likely to be unrewarding for the pupils and mathematically impoverished. Planning from tasks may increase pupils' engagement but their activity is likely to be unfocused and learning difficult to assess. By seeking inspiration from research in the areas of curriculum design, the nature of authenticity in the classroom and the use of tools, and by looking retrospectively at the design of computer‐based tasks that have underpinned their research for many years, the authors recognise a theme of purposeful activity, leading to a planned appreciation of utilities for certain mathematical concepts. The authors propose utility as a third dimension of understanding, which can be linked to purpose in the effective design of tasks. The article concludes with a set of heuristics to guide such planning.     

Participation in an intergenerational service learning course and implicit biases

Biases against the elderly and people with disabilities can lead to discriminatory behaviors. One way to conceptualize attitudes toward the elderly and people with disabilities is through the differentiation of explicit (conscious) and implicit (unconscious) factors. Although both explicit and implicit attitudes and biases contribute to the full picture of peoples’ attitudes, explicit ideas alone may not accurately reflect people’s attitudes. This may be due to societal pressure to conceal explicit biases or the fact that often times, people are not aware they hold prejudiced views. The Implicit Association Test is a computer-based categorization task designed to assess implicit or unconscious attitudes. This test was used in in the current study to assess the impact of an intergenerational service-learning course. We designed this study to determine if greater exposure or contact with the elderly or people with disabilities might influence college students’ implicit attitudes. Pre- and post-assessment of participating students, using the Wilcoxon signed-rank test, found statistically significant decreased biases after participation. Results from this study that suggest participating in an intergenerational service-learning course can positively affect implicit attitudes. This is encouraging, especially because these attitudes are developed over a long period of repeated exposure, are consistently reinforced by cultural factors, and are notoriously difficult to change.     

Collaborative Group Versus Individual Assessment in Mathematics: Processes and Outcomes

Large-scale assessment programs are beginning to design group assessment tasks in which small groups of students collaborate to solve problems or complete projects. Little is known, however, about the validity of data from group assessment for making inferences about the competence of individual students. This study compared performance in small-group and individual assessment contexts to determine how well achievement scores from the group setting represented the mathematics skills of individual students and to determine what additional information about students' skills was provided by data on their behavior in the group setting. All students performed at high levels in the group setting; the profile of group interaction was a better predictor of individual achievement than was individual performance in the group setting.     

First-year students’ appraisal of assessment tasks: implications for efficacy,engagement and performance

This study investigated students’ appraisals of assessment tasks and the impact of this on both their task-related efficacy and engagement and subsequent task performance. Two hundred and fifty-seven first-year students rated their experience of an assessment task (essay, oral presentation, laboratory report or exam) that they had previously completed. First-year students evaluated these assessment tasks in terms of two general factors: the motivational value of the task and its manageability. Students’ evaluations were consistent across a range of characteristics and level of academic achievement. Students’ evaluations of motivational value generally predict their engagement and their evaluations of task manageability generally predict their sense of task efficacy. Engagement was a significant predictor of task performance (viz. actual mark) for exam and laboratory report tasks but not for essay-based tasks. Findings are discussed in terms of the implications for assessment design and management.     

Planning and Doing in Professional Teaching Practice. A Study with Early Childhood Education Teachers Working with ICT (3–6 years)

Planning is one of the professional tasks teachers have to carry out before their direct action in the classrooms. This planning is closely interrelated to the way teachers teach. The question about how and why teachers reach their decisions in their pre-class planning is a classical one in the research into curricular design and development. The aim of this paper will therefore be to establish whether there is a relationship between curricular planning and curricular practices, studying how nine early childhood education teachers using an ICT resource plan their actions and execute them. For the research, we obtained video recordings of classroom practices and interviewed the teachers just before they went into class. By applying qualitative data analysis, we have been able to identify the elements taken into consideration when the teachers make decisions in lesson, their conceptions about ICT, and the types of activity that are held in all the classes. The results confirm the conclusions reached by previous studies on the relationships between planning and doing, in the sense that the activities, understood to be teaching strategies, are the ones that link the design of what is to be done and direct action. Likewise, the results ratify prior research on the role of materials and resources as aspects that teachers can rely on for support in the management and presentation of classroom tasks and content. The introduction of ICT does not modify the teachers’ curricular planning and development.     

Directions for research and development: alternative methods of assessing scientific literacy

Educators' current enthusiasm for alternative assessment has roots in political and intellectual concerns. American science students do not do well on international comparisons or on tests of science literacy. Blame for these shortcomings is laid on multiple-choice tests and their focus on isolated facts. But if alternative assessment is to fulfill its promise, research and development need to be done in a coordinated way. This article lays out an agenda for developing assessment tools and researching their effectiveness. A series of design tasks is proposed, followed by related research questions for each task. Needed theoretical research includes studies of the effects of alternative assessment on policy and practice as well as the development of new psychometric techniques. Influences of new assessment forms on learning and motivation theory are also discussed.     

The assessment of risk in child abuse and neglect case investigations

Estimates vary, but statistics indicate that anywhere between one and six million children will be abused and/or neglected this year in the United States. In the process of investigating reports of abuse and neglect, child protective service workers are called upon to make numerous case decisions. Critical to much of this decision making is the assessment by the worker of the potential risk of harm that exists to the child regarding further abuse or neglect. This paper, based on two separate research studies, identifies the criteria child protective service workers use to assess this potential risk of harm to the child. The research also shows that the criteria are not used as individual factors that indicate the presence or absence of risk. Instead the criteria form a constellation of factors that function for the worker in assessing the degree of potential risk to the child. The article concludes with a discussion of the implications of research findings for child welfare practice.     

Curriculum change and contextual realities in South African education: cui bono?

All planned change efforts imply a commitment to a certain view of reality, and acceptance of certain modes of realizing ends. Those assumptions constitute the conscious or unconscious bases for selecting specific courses of action and thus precede all tactical decisions. To the extent that change agents cannot identify those basic assumptions and their implications, they cannot explore the full range of effective strategies for change. (Crowfoot and Chesler 1974)     

Using interactive management to facilitate a student-centred conceptualisation of critical thinking: a case study

Critical thinking (CT) is a metacognitive process, consisting of a number of sub-skills and dispositions, that, when used appropriately, increases the chances of producing a logical solution to a problem or a valid conclusion to an argument. CT has been identified as a fundamental learning objective of third-level education; however, students often report not being given the opportunity to adequately understand and cultivate CT skills. Though most CT interventions are designed based on academic or expert definitions of CT skills, students are rarely, if ever, asked to guide their instruction by describing their perspectives on what constitutes CT. The current case study investigated students’ conceptualisations of what constitutes good CT using a collective intelligence methodology, interactive management. Interactive management (IM) is a computer-assisted process that allows a group to build a structural model describing relations between elements in a system. Though decades of research on group decision-making in educational and social psychology highlight the many limitations associated with group problem solving (e.g. as a result of an over-reliance on heuristics, cognitive biases and ‘groupthink’), a fundamental skill for making decisions and solving problems is the ability to collectively visualise the structure of a shared problem, and use this knowledge to design solutions and strategies for collective action. Results of IM group work from the current case study revealed five core CT skills (clarity of expression, conversational skill, inference, evaluation, and explanation), five CT dispositions (detachment, listening, systematicity, recognising uncertainty, and self-questioning) and fourteen structural relationships among them. The ability to detach, listen and engage in conversation with others, were seen as fundamental drivers of all other competencies in the system. Results are discussed in light of research and theory on CT and best practice for CT instruction.     

DeFT: A conceptual framework for considering learning with multiple representations

Multiple (external) representations can provide unique benefits when people are learning complex new ideas. Unfortunately, many studies have shown this promise is not always achieved. The DeFT (Design, Functions, Tasks) framework for learning with multiple representations integrates research on learning, the cognitive science of representation and constructivist theories of education. It proposes that the effectiveness of multiple representations can best be understood by considering three fundamental aspects of learning: the design parameters that are unique to learning with multiple representations; the functions that multiple representations serve in supporting learning and the cognitive tasks that must be undertaken by a learner interacting with multiple representations. The utility of this framework is proposed to be in identifying a broad range of factors that influence learning, reconciling inconsistent experimental findings, revealing under-explored areas of multi-representational research and pointing forward to potential design heuristics for learning with multiple representations.     

Higher education curriculum orientations and the implications for institutional curriculum change

This research is based on an empirical study exploring how academics make curriculum decisions and their perceptions of the influences that shape their decisions. Interviews were held with 20 academics from diverse disciplines, who were both research active and committed to teaching. The higher education curriculum was conceptualised as a field of decision-making shaped by academics’ beliefs about educational and contextual influences. The study identified five distinctive curriculum orientations representing coherent patterns of curriculum decisions aligned with academics’ beliefs about educational purposes. Case studies are presented to elucidate each of the curriculum orientations. Curriculum orientations were also found to shape academics’ responses to educational change. The following higher education change drivers are explored: graduate employability and the skills agenda, teaching–research relationships, changing understandings about teaching and learning, educational technologies and flexible delivery. The findings suggest implications for institutional curriculum change initiatives and academic development programmes.     

Creativity: The role of unconscious processes in idea generation and idea selection

Today's world of continuous change thrives on creative individuals. Anecdotal reports suggest that creative performance benefits from unconscious processes. Empirical research on the role of the unconscious in creativity, though, is inconsistent and thus far has focused mainly on one aspect of the creative process - idea generation. This is the first study to assess the role of the unconscious mind for both idea generation and idea selection. Participants generated creative ideas immediately, after conscious thought, or after a period of distraction during which unconscious thought was hypothesized to take place. After having listed their ideas, participants selected their most creative idea. Performance in idea generation was similar between conscious and unconscious thought; however, individuals who had unconsciously thought about ideas were better in selecting their most creative idea. These findings shed more light on the role of unconscious processes in creativity, and provide a means to enhance creative performance.     

Heuristic decision-making in foster care matching: Evidence from a think-aloud study

Complex decisions are often based on heuristics, which are shortcuts or simple ‘rules of thumb’. Since the matching decision in family foster care is often made in a less-than-ideal setting and riddled with uncertainty, heuristics are expected to be applied in that field of child and youth care on a daily basis. However, the use of heuristics in the matching decision has not been studied empirically until now. This research explores how decision-making heuristics are used by practitioners to determine which foster family is the best fit for a child. A number of 20 matching practitioners from the Netherlands were interviewed using vignettes and a ‘think-aloud’ methodology to generate an understanding of their reasoning. Two types of vignettes were created: hypothetical children and hypothetical foster families. The interviews were analyzed using a qualitative deductive content analysis focusing on key indicators of three classes of heuristics: recognition heuristics, one-reason heuristics, and trade-off heuristics. The results show that recognition heuristics did not play a decisive role in the matching process; practitioners considered more than one family before making a final decision. The findings for the one-reason heuristics reveal conjunctive decision-making rules; families were rejected based on one negative premise. The analysis of the trade-off heuristics demonstrates that the number of positive premises and the ratio between positive and negative premises predicted the matching decision. However, the total number of premises also predicted the matching decision, which might indicate confirmation bias.     

A conceptual framework for trust in standardised assessments: Commercial,quasi-market and national systems

This article presents a conceptual framework for trust in standardised assessments. Standardised assessments play an important role in many education systems as they inform decisions about students' future schooling career or entry to the labour market. Also, standardised assessments are often used for teacher performance reviews and school accountability, or to monitor learning outcomes on the national level. Various stakeholders rely on the accuracy of assessment outcomes when making decisions about students' competences, or seek to improve the quality of education. Such reliance implies a need for trust in those who design and administer standardised assessments and make decisions on the basis of the outcomes. The framework presented in this article describes the type of relational and macro-level trust that is relevant for three types of assessment systems: national, quasi-market and commercial systems. Throughout the analysis presented, examples are provided to illustrate the ways in which relational and macro-level trust can vary by who is tested and by whom they are assessed; and how trust in evaluations varies by the purpose and consequences of testing, as well as the individual agency of students, their teachers and school leaders.