(Mis)appropriations of criteria and standards‐referenced assessment in a performance‐based subject

This paper draws on semi‐structured interview data and participant observations of senior secondary Physical Education (PE) teachers and students at two school sites across 20 weeks of the school year. The data indicated that the teachers in this study made progressive judgements about students’ level of achievement across each unit of work without explicit or overt reference to the criteria and standards represented in the schools’ work programmes and in the Senior PE syllabus. The teachers’ justification for such an approach was that the criteria and standards had become for them sufficiently ‘internalised’. Determining students’ levels of achievement was for the teachers somewhat ‘intuitive’, being reliant on their memory of students’ performances, and influenced by the construct‐irrelevant affective characteristics of the students. It is argued in this paper that such construct‐irrelevance compromised the construct validity and possible inter‐rater reliability of the decisions made and advantaged some students and marginalised others on the basis of characteristics that were not specifically related to the learning expected from following the syllabus. The potential inequities of such an approach are discussed and suggestions are made for the consolidation of the validity and reliability of teachers’ judgements.     

The application of pairwise comparisons to form scaled exemplars as a basis for setting and exemplifying standards in teacher education

ABSTRACT

This paper details a two-stage process for standard setting using an extended pairwise comparison method. It describes the application of a new process of establishing a set of scaled and ordered performances as a basis for exemplifying standards. In stage one, the method of pairwise comparison was used to generate data from pairwise judgements made at the criterion level and overall grade. In stage two, judges identified where the pass/fail boundary lay in relation to a selected subset of ordered performances generated from stage one. The combined evidence from the two-stage process demonstrates the method’s application to establish exemplified standards using on-balance judgements across multiple criteria. The approach combines expert judgement and statistical data to provide a basis for comparable cross-institution assessments against standards in the case of a summative assessment of professional competence in teacher education. It may be relevant to any discipline with professional standards and complex performance assessments.     

Developing Performance Standards for Teacher Assessment by Policy Capturing

There is a need for assessment of teachers' competencies fostered by a growing attention given to accountability and quality improvement. Important questions are how good the demonstrated competencies of teachers should be for a satisfying assessment and how the different competencies should be weighted. Using a policy capturing method, in two rounds, nine stakeholders developed performance standards (or cut-off scores) for teacher assessment on eight criteria (or content standards) that resulted from an earlier study. Between the rounds, the panellists held a structured group discussion. Policy capturing proved to be a clear and useful method generating consistent judgements that can be described according to both a compensatory model and a conjunctive model. From the first to the second round, the consistency increased. However, while the panellists agreed to a substantial degree on the performance standards, they disagreed on the weights to be assigned to the criteria.     

'It's Pretty Difficult to Fail': The reluctance of lecturers to award a failing grade

Graduates from initial teacher education institutions in New Zealand are assumed to be fit to take up positions as beginning teachers. To this end lecturers make judgements at various stages during the teacher education programme about the quality of student performance with regard to coursework and teaching practice. Findings from a field-based project at a large metropolitan college of education in New Zealand revealed that making such judgements was not a straightforward activity. In particular, there was a reluctance to award failing grades. The college's published guidelines referred to the passing and failing of courses and the programme as dependent on the quality of submitted work or demonstrated performance. The everyday discourse and observed practice of lecturers and student teachers suggested, however, that both groups construed a fail grade as a judgement of personal worth. This shift from judgements about performance to judgements about personal worth was associated with the utilisation of criteria other than those outlined in published documents. The use of 'other' criteria enabled positive judgements to be made about students who did not meet course requirements and/or whose work failed to meet the minimum standard. Lecturers thus appeared to be reluctant to act as gatekeepers to an initial beginning teacher position.     

Standards,criteria and the problematic of action research within an award bearing course[1]

This paper stems from my practical involvement as Course Director for a Masters programme that supports teacher action research within the general area of school management. My starting point was the seeming contradiction between the need to apply criteria to make judgements about higher degree worthiness and my belief that the evaluative criteria and assessment procedures of academia should not take precedence over teachers' insights that were grounded in a rigorous examination of their own practice. This paper outlines my values as a facilitator of action research within the programme and explores possible criteria for judging success. Several possible standards of judgement are considered: to do with ethics, rigour, logic, the ‘practical’ and aesthetics. The paper also suggests criteria that might apply to different stages of the action research process and raises the question concerning ‘competence to judge’ action research.     

Assessment criteria: reflections on current practices

This article reviews the findings of a small‐scale investigation into the criteria used by a number of SACWG departments for assessing final‐year project modules in business and history and other written history assignments. The findings provide the basis for a broader discussion of the issues relating to the formulation and use of assessment criteria. Assessment entails academics making professional judgements about the standards and quality of students' work. However, for the educational value of the work entailed in developing assessment criteria to be fully realized, there needs to be a higher level of shared understanding than currently exists (among students, tutors and other stakeholders) of the language in which criteria are couched and the ways in which criteria are applied.     

Experience is Not The Whole Story: The Integral Role of the Situation in Dewey's Democracy and Education

The central objective of Dewey's Democracy and Education is to explain ‘what is needed to live a meaningful life and how can education contribute?’ While most acquainted with Dewey's educational philosophy know that ‘experience’ plays a central role, the role of ‘situations’ may be less familiar or understood. This essay explains why ‘situation’ is inseparable from ‘experience’ and deeply important to Democracy and Education’s educational methods and rationales. First, a prefatory section explores how experience is invoked and involved in pedagogical practice, especially experience insofar as it is (a) experimental, (b) direct, and (c) social‐moral in character. The second and main section on situations follows. After a brief introduction to Dewey's special philosophical use of ‘situation’, I examine how situations are implicated in (a) student interest and motivation; (b) ‘aims’ and ‘criteria’ in problem‐solving; and (c) moral education (habits, values, and judgements). What should become abundantly clear from these examinations is that there could be no such thing as meaningful education, as Dewey understood it, without educators’ conscious, intentional, and imaginative deployment of experience and situations.     

Policy into practice: internal assessment at 16 +: standardization and moderation procedures

1986 saw a radical change in the structure of examining at 16+ when, with the introduction of the General Certificate of Secondary Education in England and Wales, an element of school‐based assessment was incorporated for the first time. Both the examining groups and the teaching force had to adjust to new constraints and accommodate each other's perspectives. Teachers have had to operate with closely defined criteria which were developed by the examining groups, who also subsequently monitored their application to pupils’ work. This paper examines the procedures deployed to ensure comparability of judgements between teachers, both within and between schools, and highlights some of the tensions which now exist as roles have been adapted. It is suggested that some of the conflicts of interest might have been diminished, and more effective professional relationships developed between the two bodies, had procedures been more open to public scrutiny. The examining groups have a specific responsibility to maintain standards; but by taking more account of the position of teachers and their training needs, more effective long‐term solutions could be evolved. Finally, it is suggested that the experiences of GCSE may be used to develop more effective mechanisms for National Curriculum Assessment, thereby making procedures smoother for all parties involved.     

The distinction between academic standards and quality: implications for transnational higher education

Although the conceptual distinction between academic standards and the quality of learning opportunities is fundamental to an understanding of the role of quality assurance in higher education, the distinction, and its implications, have not always been well understood in the case of transnational higher education. This paper explores both the source of academic standards and the criteria by which the quality of learning opportunities can be judged. It is argued that the distinction is of particular importance in the context of transnational education, because whilst judgements about quality may legitimately be susceptible to a degree of cultural influence, the same is not true of judgements about academic standards. It is suggested that a lack of attention to the logical distinction between standards and quality has hampered efforts by regulatory régimes around the world to develop coherent policies with respect to hosting transnational higher education provision.     

Perspectives on the standards agenda: exploring the agenda's impact on primary teachers' professional identities

ABSTRACT

This study aimed to investigate teachers' perspectives on the practical implementation of the standards agenda and its impact on their professional identities. Q-methodology was used alongside semi-structured interviews with UK primary school teachers. The study explored the views of 25 teachers in six schools, selected through purposive sampling to give a range of individual and institutional demographics. Teachers in this research commented on the impact the standards agenda has had on parental and societal judgements that affect their identity as professionals. Teachers held differing positions on whether they experienced constraint or flexibility when implementing standards objectives. These differing positions were mainly influenced by whether they taught above or below Year 3. Teachers who found flexibility in the agenda's objectives had less occupational stress and increased ownership of their own actions and the standards agenda. Teachers' positions were complex and changed according to situational influences at a classroom level.     

Universities and sustainable development: the necessity for barriers to change

In June 2015, the Higher Education Funding Council for England (HEFCE) proposed a lighter touch quality assessment regime that would place greater reliance on institutions' own governance arrangements. In July 2015 the Minister for Higher Education announced a Teaching Excellence Framework (TEF) parts of which appear inconsistent with HEFCE's proposals. In August the Quality Assurance Agency for Higher Education (QAA) responded to HEFCE's proposals envisaging some developments to its present quality assessment arrangements. This article analyses the key issues and principles that will supposedly govern the design of a new quality assessment regime from 2017. It proposes points of reference against which to evaluate the proposals for that new regime; these are whether the approach to review that is adopted: a) differentiates between institutions on the basis of their capacities to manage their responsibilities; b) shows the regulator's willingness to trust institutions to discharge their responsibilities (with verification) and willingness by institutions to accept their responsibilities; c) adopts an approach to risk and its management that is forward-looking and enables pre-emptive actions to forestall negative outcomes; and d) makes greater use of metrics that are interpreted by experts qualified to make judgements on academic matters.     

标准参照评估的行动实践 ——以香港中文大学通识教育基础课程为案例

香港特别行政区高校从2015年开始，学生学业评估陆续由常模参照评估转为标准参照评估。本文以香港中文大学通识教育基础课程实施标准参照评估的行动为案例，分析了这项新的学业评估方式在实施中所遇到的问题，讨论了解决方法并提出实施模型。该模型包含评分量表的设计与开发、量表使用、教师意见反馈与量表修改，以及评分之后的集体讨论这一套实施流程。文章最后指出，要保证标准参照评估的有效实施，需建立教师实践共同体。实践共同体的行动包括：教师共同开发评估标准，通过集体讨论对评估标准的含义和使用达成共识，共同审视评估结果，根据评估结果进一步修改和完善评估标准。     

Redesigning an advanced master's degree for early childhood professionals in alignment with NBPTS and NAEYC standards

Abstract

This article describes the standards‐based redesign of one university's advanced master's degree program for early childhood practitioners. It discusses the philosophy and goals, degree components, and features that aligned with standards from the National Board for Professional Teaching Standards and the National Association for the Education of Young Children guidelines for advanced degree programs.     

The Use of Exemplars and Formative Feedback when Using Student Derived Marking Criteria in Peer and Self-assessment

This paper reports a study which implemented a method of student self and peer assessment involving student constructed marking criteria in the presence of exemplars. Pairs of first-year undergraduate biology students were asked to complete a poster assignment. The study was designed to allow the evaluation of (1) student self and peer marking and tutor marking for individual marking criteria following the use of exemplars; (2) the role of exemplars in providing a focus for formative feedback about subject standards. The present study shows that: (a) the use of exemplars can help students demonstrate greater understanding of both marking criteria and subject standards; (b) the use of exemplars can help students learning so that higher quality outcomes are produced; (c) the use of exemplars forms a focus for meaningful formative feedback; (d) students may make more objective judgements as a result of peer assessment compared to self-assessment.     

Moderation and consistency of teacher judgement: Teachers’ views

Major curriculum and assessment reforms in Australia have generated research interest in issues related to standards, teacher judgement and moderation. This article is based on one related inquiry of a large‐scale Australian Research Council Linkage project conducted in Queensland. This qualitative study analysed interview data to identify teachers’ views on standards and moderation as a means to achieving consistency of teacher judgement. A complementary aspect of the research involved a blind review that was conducted to determine the degree of teacher consistency without the experience of moderation. Empirical evidence was gained that most teachers, of the total interviewed, articulated a positive attitude towards the use of standards in moderation and perceived that this process produces consistency in teachers’ judgements. Context was identified as an important influential factor in teachers’ judgements and it was concluded that teachers’ assessment beliefs, attitudes and practices impact on their perceptions of the value of moderation practice and the extent to which consistency can be achieved.     

The Experience of Collaborative Assessment in e-Learning

This paper examines the various ways in which students talk about their experience and perceptions of collaborative review and assessment as it occurs in e-learning environments. Collaborative review and assessment involves the student, their peers and tutor in thoughtful and critical examination of each student's course work. The process involves two stages: review and discussion of the student's work with a view to bringing different critical yet supportive perspectives to the work. This is followed by the use of two sets of criteria to make judgements on the student's work: one set provided by the student, the other by the tutor. The purpose of collaborative assessment is to foster a learning approach to assessment and to develop a shared power relationship with students. From analysis of face-to-face interviews, examination of e-learning discussions and student-completed questionnaires, a set of analytic categories was built describing the learners' experiences of collaborative e-assessment. These categories are: (1) the appropriateness of collaborative assessment; (2) collaborative assessment as a learning event; and (3) the focus for assessment. The paper focuses on analysing and discussing these categories of experience. The research shows that a positive social climate is necessary in developing and sustaining collaborative assessment and that this form of assessment helps students to reduce dependence on lecturers as the only or major source of judgement about the quality of learning. Students develop skill and know-how about self- and peer assessment and see themselves as competent in making judgements about their own and each other's work, which are surely good lifelong learning skills.     

英国高等教育质量保证署的学术审查

学术审查是英国最新的、综合性的外部质量评估。作为英国高等教育质量保证署(QAA)评估活动的一种方式,旨在建立、保持和提高学术标准和质量。英国每所学校及其全部专业每六年都要审查一次。其具体运作大致分为三个阶段——提交自评报告、审查小组分析、核实自评报告并作出最后判断。