Qualitatively different ways of experiencing student self‐assessment

In the literature, student self‐assessment as a practice and as a goal in higher education is generally emphasized for academics. This paper reports a study that investigates what academics emphasize in their experiences of student self‐assessment. The investigation focused on the different ways academics described their understanding and practise of self‐assessment. A phenomenographic approach was used to research and identify a set of progressive variations of academics’ ways of experiencing student self‐assessment. Altogether, 16 academics from a variety of disciplines and programs of study in three Australian universities participated in the investigation. The consequent research findings describe five qualitatively different conceptions that depict how academics understand and use student self‐assessment. These findings are subsequently discussed in terms of the potential for academics to understand and use student self‐assessment to enhance students’ self‐assessment ability, to further students’ lifelong learning and to empower, rather than discipline, students.     

Large‐scale mathematics assessment: looking globally to act locally

In this paper, I will argue that it is possible to use data from large‐scale international and national mathematics assessment programmes, whose attention is on summative achievement, to provide formative information that informs teachers about the effects of their classroom practice. However, to have impact on, and be useful for, classroom practitioners, these achievement data need to be reworked and re‐presented in ways that are plausible, provide a basis for inferences about practice, and be appropriate for the intended audience. This paper examines achievement‐focused assessment programmes in terms of their aims and approaches, and develops the argument that formative assessment possibilities are present, within these programmes, although usually hidden. Examples are drawn from several sources to support this argument, and demonstrate a variety of approaches that have been taken in the past. Suggestions for further action are made.     

Developing a play‐based communication assessment through collaborative action research with teachers in northern Canadian indigenous communities

With the goal of developing culturally appropriate approaches for assessing and supporting children's language use, teachers of 4‐to 6‐year‐old children in northern Canadian rural and Indigenous communities are involved in a 6‐year collaborative action research project. Teachers video record children's interactions during dramatic and construction play and then meet with university researchers to carry out inductive analyses of ways in which children use language to achieve social purposes. From these analyses, a Play‐based Communication Assessment has been created. Examples from two teachers' classrooms in one Indigenous community are used to show how play contexts and the still‐evolving play‐based communication assessment provide opportunities for teachers to recognise and build upon the linguistic and cultural resources that children bring to classrooms. Through the play‐based assessment and action research processes, teachers have come to recognise the richness of children's language when they are engaged in play and have gained understandings of their community's culture. Teachers and researchers are exploring ways to capture children's non‐verbal communication abilities through this assessment approach.     

Interpretive validity theory: mapping a methodology for validating educational assessments

Background:?Validity theory has evolved significantly over the past 30 years in response to the increased use of assessments across scientific, social and educational settings. The overarching trajectory of this evolution reflects a shift from a purely quantitative, positivistic approach to a conception of validity reliant on the interpretation of multiple evidence sources integrated into validity arguments. Moreover, within contemporary validity, interpretation has been emphasised as a central process; however, despite this emphasis, there have been few explicit articulations of specific interpretive methodologies applicable to the practice of validation.

Purpose:?To link contemporary theoretical foundations in validity to practical methods and structures to help guide the collection and analysis of interpretive validity evidence. By building upon existing validity theory, this paper aims to provide greater clarity on the practice of validation and contribute toward the larger developing framework for the validation of educational assessments.

Source of evidence:?An interdisciplinary, integrative review of over 60 research articles and sources related to the theory and practice of educational validation and interpretive inquiry approaches. Sources include literature from the fields of educational assessment and more broadly social scientific research.

Main argument:?As assessments in education increasingly aim to measure complex constructs that are value-laden and socially dependant, validity theory must keep pace and evolve in ways that address the inherent complexities associated with contemporary educational assessment. Through this paper, I assert that a greater understanding of interpretive methodologies represents one of the most promising areas for development of validation theory and practice. Specifically, I argue that dialectic, hermeneutic and transgressive forms of inquiry can be integrated within current argument-based structures for the collection, analysis and representation of validity evidence in several useful ways.

Conclusions:?Interpretive inquiry processes, namely dialectic, hermeneutic and transgressive forms of interpretation, serve to expand validation practice to include diverse evidences for the generation of multiple-perspective validity arguments. The paper concludes with specific implications for future research and practice within the field of interpretive validity theory.     

A scholarly approach to solving the feedback dilemma in practice

It is clear from the literature that feedback is potentially the most powerful and potent part of the assessment cycle when it comes to improving further student learning. However, for some time, there has been a growing amount of research evidence that much feedback practice does not fulfil this potential to influence future student learning because it fails in a host of different ways. This dilemma of the disjuncture between theory and practice has been increasingly highlighted by the UK National Student Survey results. This paper uses a model of the assessment process cycle to frame understandings drawn from the literature, and argues that the problem with much current practice resides largely in a failure to effectively engage students with feedback. The paper goes on to explore how best to effectively engage students with assessment feedback, with evidenced examples of feedback strategies that have successfully overcome this problem.     

When Assessment Validation Neglects Any Strand of Validity Evidence: An Instructive Example from PISA

The Standards for Educational and Psychological Testing identify several strands of validity evidence that may be needed as support for particular interpretations and uses of assessments. Yet assessment validation often does not seem guided by these Standards, with validations lacking a particular strand even when it appears relevant to an assessment. Consequently, the degree to which validity evidence supports the proposed interpretation and use of the assessment may be compromised. Guided by the Standards, this article presents an independent validation of OECD's PISA assessment of mathematical self-efficacy (MSE) as an instructive example of this issue. OECD identifies MSE as one of a number of “factors” explaining student performance in mathematics, thereby serving the “policy orientation” of PISA. However, this independent validation identifies significant shortcomings in the strands of validity evidence available to support this interpretation and use of the assessment. The article therefore demonstrates how the Standards can guide the planning of a validation to ensure it generates the validity evidence relevant to an interpretive argument, particularly for an international large-scale assessment such as PISA. The implication is that assessment validation could yet benefit from the Standards as what Zumbo calls “a global force for testing”.     

Establishing the critical elements that determine authentic assessment

This study sought to determine the critical elements of an authentic learning activity, design them into an applicable framework and then use this framework to guide the design, development and application of work-relevant assessment. Its purpose was to formulate an effective model of task design and assessment. The first phase of the study identified from the literature critical elements that determined assessment as being authentic, and presented these to practitioners and experts for feedback. In phase two, it codified the elements into a framework that was then applied to the redesign of assessments in an army course. Phase three involved student evaluation of the redesigned assessment activities. This led to further review and revision of elements in phase four. The study outcomes suggest that it is possible, by identifying and codifying individual elements, to determine the ways in which the authenticity of an individual assessment activity might be enhanced. The paper concludes with a literature update on the framework elements that lead to suggestions for further research.     

Constructing validity arguments for test combinations

The argument-based approach to validation has been widely adopted in validation theory. However, this approach aims to validate the intended interpretation and use of a single test or assessment. This article proposes an extension of the argument-based approach for validation of multiple tests. This extension is illustrated with the validation of a competency assessment program (CAP). This CAP was validated in collaboration with a quality manager of an educational program. In this case study, it became apparent that this approach fosters an in-depth evaluation of the assessment program and that the approach appears suitable for validation efforts of competency assessment programs. The approach guides validation research from a more general perspective, but also guides more detailed validation efforts.     

Dynamic Assessment practice: some suggestions for ensuring follow up

This article examines tensions in Dynamic Assessment (DA) practice that relate ways in which its findings can be followed up and made accessible to school staff, by examining how process skill interventions can be implemented in the context of a content‐focussed curriculum. It briefly outlines some of the theoretical and methodological underpinnings of DA that arise from Feuerstein’s theory of structural cognitive modifiability. It then suggests ways in which links can be made between DA and Department for Education and Skills initiatives that emphasise the development of process skills as well as delivery of curriculum content. Finally, three conditions that are likely to ensure links between assessment and intervention are proposed: sharing common assessment and intervention goals; sharing a common curriculum context; and sharing a common language.     

The assessment of lesson plans in teacher education: a case study in assessment validity and reliability

This paper forms part of an exploration of assessment on one part‐time higher education (HE) course: an in‐service, professional qualification for teachers and trainers in the learning and skills sector which is delivered on a franchise basis across a network of further education colleges in the north of England. This paper proposes that the validity and reliability of portfolio‐based assessment, a key component of many HE programmes in addition to the course being researched here, is contestable. Analysis of the processes of compiling portfolios for assessment, through the conceptual framework of the New Literacy Studies, suggests that the ways in which portfolios are assessed and the ways in which the crucial requisites of validity and reliability are assigned to them, mask complexities and contradictions in their creation by the student. This paper argues for a new, critical analysis of portfolio production and raises a number of questions about the validity, reliability and authenticity of the assessment process that the portfolios reify.     

The impact of the Phonics Screening Check on grouping by ability: A ‘necessary evil’ amid the policy storm

This article examines the impact of a statutory assessment in England, the Phonics Screening Check (PSC), on classroom practices of grouping children by ‘ability’. Bearing in mind the argument that assessment is the rudder that steers the otherwise slow‐moving battleship of educational practice, it is argued that the PSC has altered how teachers organise their classes and curriculum in both the affected year group (Year 1, children aged 5–6) and in earlier and later years. Using data from a nationwide survey of teachers (n = 1,373), focus groups and in‐depth interviews with teachers, the article examines how this relatively new phonics assessment forms part of a ‘policy storm’ of pressures relating to accountability, which encourage teachers to place children in groups on the basis of ability, even when they have doubts about this practice and there is little evidence to suggest grouping improves attainment. Practices include grouping children within classes, across year groups or even across several year groups, by phase of phonics learning, guided by advice from bought‐in private phonics schemes. There is also evidence of ‘educational triage’, where borderline children become the focus, and increased use of interventions which involve withdrawing children. Overall, the article uses the PSC to demonstrate how, in times of multiple policy pressures, assessment can rapidly alter practice, in this case making grouping a ‘necessary evil’, as one teacher respondent argued.     

The role of learner-centred assessment in postsecondary organisational change

For postsecondary institutions, organisational change is an inevitable and ongoing process. Institutions face a growing need to assess student learning assessment as part of an institution's comprehensive assessment plan. While a growing body of literature suggests that learner-centred assessment is a best practice in higher education pedagogy, it is unclear whether faculty members have embraced it fully. Using data from the 1993 and 2004 administrations of the National Study of Postsecondary Faculty, this study examines the extent to which faculty members employed learner-centred assessment in two- and four-year colleges in the USA. Findings show increases in use for some assessment techniques over the decade, as well as differences in use by faculty gender, discipline, and institution level. Increases in the use of these techniques may indicate a continued emphasis on learner-centred assessment and thus one aspect of organisational change. Implications for institutional policy are discussed.     

The Need and Use of Process Recording in Policy Practice: A Learning and Assessment Tool for Macro Practice

This article proposes the use of a structured process recording format for policy practice students, field instructors, and social work faculty as a learning, teaching, and assessment tool. Traditionally, casework and group work have used process recording to facilitate students learning in the field practicum to assess both the process and content of student‐client interactions. Since policy practice is concerned with achieving a preferred course of action to shape policies and interactions affecting large numbers of people, process recordings offer a method to assess tasks, competency skills, and learning needs while examining process and conscious use of self in action.     

Learning in a comfort zone: cultural and social capital inside an outcome‐based assessment regime

Despite numerous problems with outcome‐based assessment systems, claims that they enhance learners' motivation and autonomy resonate with research interest in how young people develop cultural and social capital. However, research has not yet explored the ways in which assessment systems affect the forms of capital embedded within them. This paper applies concepts from a growing body of work on social and cultural capital in education to data from a study of assessment policy and practice in the Advanced General National Vocational Qualification (GNVQ) in two further education colleges. It evaluates how norms, practices and dynamics created by the GNVQ assessment regime interacted with other factors in students' lives and the learning programme itself. These interactions shaped cultural and social capital inside a learning ‘comfort zone’ and affected students' motivation and attitudes to learning. The article evaluates the implications of these factors for the types of cultural and social capital that may be on offer in different assessment regimes. In particular, it raises questions about the extent to which social and cultural capital are empowering or constraining, and offers ideas for further research in this area.     

Test Design with Cognition in Mind

One of the primary themes of the National Research Council's 2001 book Knowing What Students Know was the importance of cognition as a component of assessment design and measurement theory ( NRC, 2001 ). One reaction to the book has been an increased use of sophisticated statistical methods to model cognitive information available in test data. However, the application of these cognitive-psychometric methods is fruitless if the tests to which they are applied lack a formal cognitive structure. If assessments are to provide meaningful information about student ability, then cognition must be incorporated into the test development process much earlier than in data analysis. This paper reviews recent advancements in cognitively-based test development and validation, and suggests various ways practitioners can incorporate similar methods into their own work.     

Educational assessment policy and practice: a matter of ethics

Long-standing concerns within the field of educational assessment consider the impact of assessment policy and practice as matters of equity, inequality and social justice. Yet educational assessment policy and practice continues to have powerful social consequences for key users such as children and young people. This paper re-positions these consequences as a matter of ethics. It uses the work of Messick to frame how ethical matters extend beyond test instruments into the realm of uses and impact. A case study of the 11+ school transfer system in Northern Ireland is presented to illustrate ethical dilemmas emerging as a consequence of actions and decisions of using assessment systems for particular purposes. In looking forward to how we might attend to ethical matters in assessment policy and practice, a consideration of a children’s rights approach is outlined that may provide a moral and legal framework for action.     

Teaching for the Test: Validity, Fairness, and Moral Action

In response to heightened levels of assessment activity at the K-12 level to meet requirements of the No Child Left Behind Act of 2001, measurement professionals are called to focus greater attention on four fundamental areas of measurement research and practice: (a) improving the research infrastructure for validation methods involving judgments of test content; (b) expanding the psychometric definition of fairness in achievement testing; (c) developing guidelines for validation studies of test use consequences; and (d) preparing teachers for new roles in instruction and assessment practice. Illustrative strategies for accomplishing these goals are outlined.     

Accuracy of students' self‐assessment and their beliefs about its utility

The purpose of the two studies presented here was to evaluate the accuracy of students' self‐assessment ability, to examine whether this ability improves over time and to investigate whether self‐assessment is more accurate if students believe that it contributes to improving learning. To that end, the accuracy of the self‐assessments of 3588 first‐year students enrolled in a post‐secondary institution was studied throughout a semester during which each student made approximately 80 self‐assessments about his or her own learning process. These self‐assessments were then compared with multiple judgements by peers and tutors. The overall correlations between the scores of self‐, peer and tutor assessments suggest weak to moderate accuracy of student self‐assessment ability. The findings also reveal an ability effect; students judged as more academically competent were able to self‐assess with higher accuracy than their less competent peers. Comparing the accuracy of student self‐assessment averaged over four consecutive periods indicates that the accuracy does not improve over time. In a second study, a questionnaire aimed at eliciting student's beliefs about the effects of self‐assessment on their learning was administered to 936 first‐year students. Based on their responses, sub‐groups of students were identified: those who either believed in the usefulness of self‐assessment or did not. Results suggest that there is no significant association between student beliefs about the utility of self‐assessment and the accuracy of their self‐assessments.