The development of a criterion‐referenced profile: the Hertfordshire English Project

Many developments in profiling are taking place at present and a considerable number use a criterion‐referenced assessment model. This study reports the experiences of a local curriculum development project which has developed an English course and summative profile for lower‐attaining pupils. The way in which criterion‐referenced assessment was incorporated into the development of the profile is described, together with the results of a small‐scale survey of teachers’ views of the use of such a profile. Key issues for those developing such profiles in English and other subjects are identified.     

Problems in Measuring the Reliability of National Curriculum Assessment in England and Wales

Standard Assessment Tasks (SATs) are used in the assessment of pupil performance relative to the National Curriculum in England and Wales, assigning pupils to one of 10 levels on various Attainment Targets (ATs). This is a criterion‐referenced system, and the conventional reliability measures used in norm‐referenced systems do not apply. This paper discusses the problems involved in aggregating SA T items to give levels, and in measuring the reliability of the outcomes produced. Fitting a simple model to SAT data has allowed reliability measures to be produced which are more relevant to this situation than the single numerical values applied to norm‐referenced tests.     

Measuring learning outcomes

The growing interest for measurement of learning outcomes relates to long lines of development in higher education, the request for accountability, intensified through international reforms and movements such as the development and implementation of qualifications frameworks. In this article, we discuss relevant literature on different approaches to measurement and how learning outcomes are measured, what kinds of learning outcomes are measured, and why learning outcomes are measured. Three dimensions are used to structure the literature: Whether the approaches emphasise generic or disciplinary skills and competence, self‐assessment or more objective test based measures (including grades), and how the issue of the contribution from the education program or institution (the value‐added) are discussed. It is pointed out that large scales initiatives that compare institutions and even nations seem to fall short because of the implicit and explicit differences in context, whilst small‐scale approaches suffer from a lack of relevance outside local contexts. In addition, competence (actual level of performance) is often confused with learning (gain and development) in many approaches, laying the ground for false assumptions about institutional process‐quality in higher education.     

SETTING OBJECTIVES AND ASSESSING COMPETENCE IN PROFESSIONAL LEGAL EDUCATION

Professional Legal Education in the Australian Capital Territory (ACT) is undertaken at the Legal Workshop. A problem‐based learning orientation is a prominent feature of their programme. In this paper the assessment of the problem‐based learning used in the Workshop is described. All stages of this assessment process, including setting the course objectives, defining the conditions under which the problems (tasks) will be undertaken, and setting the criteria which are the standards for the assessment of success in the task, are described. The importance of criterion‐referenced assessment in the stage of setting assessment standards is emphasised. The results of the first attempt to use this assessment method are then described, and problems emerging with the process are discussed. Although this first “reality test” reveals some difficulties with the assessment method, it is concluded that this novel and innovative teaching and assessment technique has been appropriately applied to Professional Legal Education. While some refinements are required in the implementation of the method, the overall assessment concept has been successfully developed.     

Achieving Bologna convergence: is ECTS failing to make the grade?

Transparent and consistent credit transfer procedures are essential if EU Universities are to successfully build the European Higher Education Area and thrive in the emerging global knowledge economy. Currently, the European Credit Transfer System is the most widely used mechanism to enable credit transfer between universities in different EU nations. Using data from 20 universities in four EU member states, this paper examines the problems in calculating and using ECTS grades. The results demonstrate that the alignment of ECTS grades varies within nation states and show that, despite the fact that ECTS grading is a norm‐referenced system, while national systems are usually criterion‐referenced, the ECTS conversion tables provided by universities indicate straight line transference from institutional to ECTS grades. Given the anticipated increase in student mobility following the EU enlargement to 25 nations, this paper proposes a re‐alignment of ECTS towards a criterion‐referenced system. Such a new system would acknowledge and build on the diversity of EU higher education systems, unlike the current mechanistic system. The latter masks this diversity and is flawed in calculation and ad hoc in operation.     

Accountability and Control: a sociological account of secondary school assessment in Queensland

This paper analyses sociologically the current form of school‐based secondary assessment in Queensland which is criterion‐referenced to Year 10 and a hybrid criterion/norm referenced form at the end of Year 12. Habermas’ arguments are used to suggest that this assessment pattern will give the state potentially greater ‘steering capacity’ over education by ‘rationalising’ it — the ‘scientisation of schooling’. This form of assessment fits within the accountability discourse of the economically parsimonious 1980s while meeting selection demands. However, the approach does meet some educational demands. The paper also reflects upon the role of the state and expert knowledge in policy formulation.

     

Higher education learning outcomes – Ambiguity and change in higher education

The emerging interdependent world order poses new challenges for States and citizens alike. For States, interdependence has meant a new concern with integration, whilst for citizens and authorities alike, greater mobility has raised new concerns about recognition of competences, qualifications, quality and transparency. The introduction of learning outcomes is one of the principal instruments to achieve this in higher education. This article analyses how the implementation of higher education learning outcomes (HELOs) can be seen as ambiguous governance and management tools, manifested as parts of international policy development and policy trends. These ambiguous tools intertwine with different disciplinary and stakeholder networks. The desire to implement HELOs in a more or less uniform way across as diverse contexts (countries, disciplines, institutions) as possible has led to a design strategy that favours generic definitions of learning outcomes. In the implementation process, these generic HELOs are experienced as ambiguous, meaning that they are characterised by an openness to different interpretations. This opens up a space of discretionary and interpretational latitude, either because HELOs are assimilated to traditional path dependencies, or because they allow institutional agents (such as institutional leaders and others) the space to introduce change. The ambiguity of HELOs simultaneously provides the flexibility for contextually‐diverse implementation, ensures less comparability than initially envisaged, and opens up the possibility for change, although change is contingent on structures and processes that are external to the policy process itself. HELOs are thus a paradigm case of the centrality of context in policy implementation studies.     

SOAP in practice: learning outcomes of a cross‐institutional innovation project conducted by teachers,student teachers,and teacher educators

This paper reports on a case study investigating learning outcomes at the individual and organisational level of a cross‐institutional innovation project based on the SOAP approach. SOAP integrates Schooling of teachers, Organisational development of schools, Action‐ and development‐oriented research, and Professional development of teachers. The innovation project was aimed at combining teachers, student teachers, and teacher educators in an alliance to design and develop new competence‐based vocational educational arrangements for pupils. An inductive qualitative analysis of 37 semi‐structured interviews among the participants revealed seven main categories of individual learning outcomes: attitudes, project design and management, collaboration, action theory, teaching practice, educational principles, and developments within secondary vocational education. Three main categories of organisational learning outcomes were identified: institution‐level learning, project‐level learning, and combining institution‐level and project‐level learning. A tension was identified between the participants’ individual interests in learning and personal development, and the need for organisational learning aimed at improving organisational processes.     

Collaborative Teaching and Learning through Multi‐Institutional Integrated Group Projects

This teaching brief describes an innovative multi‐institutional initiative through which integrated student groups from different courses collaborate on a common course project. In this integrated group project, students are asked to design a decentralized manufacturing organization for a company that will manufacture industrial Proton‐Exchange Membrane fuel cells. The groups include students from supply chain management, production planning and scheduling, and facility layout and design courses. Empirical results from the implementation suggest that students responded positively to the integrated experience. Lastly, the article presents implementation strategies for multi‐institutional group projects based on the experiences gained through the collaborative experience.     

Meanings and Consequences in Standard Setting

Standard setting is defined as the identification of certain points on a mark scale with particular performance standards, with the intention of enhancing the inferences that are warranted from the test scores. It is argued that the selection of both the points on the mark‐scales and the performance standards with which they are equated are arbitrary and are driven by a set of values (which are often implicit). In ‘high‐stakes’ settings, it is shown how the values implicit in the standard can come to dominate the values inherent in the domain they represent. The validation of standards must therefore include consideration of their consequences as well as their meanings. It is then argued that standards, where they exist, cannot be accounted for purely in terms of norm‐referenced or criterion‐referenced interpretations, but exist rather by virtue of a shared construct in a community of practice. These theoretical positions are then developed to classify standard‐setting methods along two dimensions, the first relating to the role of performance data in the setting of standards and the second relating to the extent to which the meanings or the consequences of the assessment are emphasised in the process.     

The Assessment of GCSE Art: Criterion‐Referencing and Cognitive Abilities

In this paper Cognitive Abilities Test scores are compared directly with moderated GCSE scores awarded to the same group of pupils. For ease of interpretation the comparisons are presented in a graphical form. Whilst some provisional and tentative conclusions are drawn about the reliability of GCSE art, questions are raised about the general validity of criterion‐referenced assessment in this area.     

Criterion-referenced and norm-referenced assessments: compatibility and complementarity

The tension between criterion-referenced and norm-referenced assessment is examined in the context of curriculum planning and assessment in outcomes-based approaches to higher education. This paper argues the importance of a criterion-referenced assessment approach once an outcomes-based approach has been adopted. It further discusses the implementation of criterion-referenced assessment, considering to what extent the criteria and standards adopted are implicitly norm referenced. It introduces a compatible interpretation of criterion-referenced and norm-referenced assessments in higher education, and illustrates how their combined use can avoid grade inflation and also provide useful information to educators, employers and learners. Instead of seeing criterion referencing and norm referencing as a dichotomy, assessment in higher education benefits from their synthesis through a feedback loop that emphasises alignment between learning and assessment; such feedback and alignment are essential features of quality assurance and enhancement.     

Bones,boys, bombs and booze: an exploratory study of the reliability of marking dissertations across disciplines

The primary aim of this study was to evaluate the reliability of the University’s Masters’ level (M‐level) generic assessment criteria when used by lecturers from different disciplines. A further aim was to evaluate if subject‐specific knowledge was essential to marking these dissertations. Four senior lecturers from diverse disciplines participated in this study. The University of Teesside’s generic M‐level assessment criteria were used and formatted into a grid. The assessment criteria related to the learning outcomes, the depth of understanding, the complexity of analysis and synthesis and the structure and academic presentation of the work. As well as a quantitative mark, a qualitative statement for the reason behind the judgement was required. Each lecturer provided a dissertation that had previously been marked. All participants then marked each of the four projects using the M‐level grid and comments sheet. The study found very good inter‐rater reliability. For any one project, the variation in marks from the original mark was no more than 6% on average. This study also found that subject‐specific knowledge was not essential to marking when using generic assessment criteria in terms of the reliability of marks. The authors acknowledge the exploratory nature of these results and hope other lecturers will join in the exploration to test the robustness of generic assessment criteria across disciplines.     

THE STUDENT ASSESSMENT PROJECT

Under a grant from Education Research and Development Committee, researchers at Macquarie University, Sydney, Australia, developed a set of instructional materials aimed at the inservice education of teachers on the topic of student assessment. The Student Assessment Project (SAP) now comprises seven modules in slide‐tape format covering the topics of test design, item writing, analysis of norm‐referenced and criterion‐referenced tests, combining scores from different components, moderation of test results, and grading and reporting. The kit also contains appropriate computer software (for an Apple II microcomputer), manuals and supplementary materials. This article gives some details of the project and its development and describes the widespread use of the first 20 kits from which evaluation data are being sought. Although SAP originally focussed on inservice education of secondary teachers, the present applicability to higher education and the further potential for modification and use at this level is discussed.     

A Mastery Rubric: guiding curriculum design,admissions and development of course objectives

This article describes a ‘Mastery Rubric’ (MR) used to design both the curriculum and the assessments in a new two‐year certificate programme intended to train physicians in clinical research skills. The MR for clinical research skills is built around a set of core research skills: critical review of literature; articulation of research objective; development of research design; development of analysis plan; implementation of the study; implementation of the analysis plan and presentation of results. Four distinct levels of performance are described for each skill: beginning, novice, competent and proficient. This rubric outlines and provides a path to mastery of the clinical research skills the certificate programme was designed and funded to target. Using the rubric to design the curriculum ensures that courses will provide instruction in key domains, promotes assessment that demonstrates development in the target skills and knowledge, and encourages reflection and cognitive self‐monitoring in the students. It is a flexible, criterion‐referenced definition of ‘success’ for students as well as the programme itself. The criteria are characterised in terms of the skills, habits of mind and organisational principles that can foster excellence in clinical research, but the approach can be generalised.     

Theoretical and Practical Integration of Literacy and Numeracy in a University Academic Programme

Abstract

This paper reports on research into the enhancement of generic skills in academic numeracy in the context of an Australian university teacher education programme. The project addressed issues of what constitutes academic numeracy, the design and implementation of appropriate numeracy strategies, and how best to integrate tertiary literacy and numeracy in university courses. This paper presents an overview of the key stages of the project, and the development of its outcomes. The major outcome is a framework for academic numeracy, which is a means for identifying and describing the numeracy demands of academic texts and tasks.     

JOB DUTIES OF INSTITUTIONAL RESEARCHERS IN SOUTHERN TWO‐YEAR COLLEGES

The objective of this study was to determine the types of duties being performed by institutional researchers in Southern two‐year colleges. Research was based on a mail survey submitted to all two‐year colleges accredited by the Southern Association of Colleges and Schools. Of 326 colleges surveyed, a total of 301 (92 percent) responded. Respondents completed a survey designed with the assistance of a panel of experts. This included a section posing statements about institutional research tasks in a discrepancy model, and a second section consisting of statements about the overall research function organized in a five‐part, Likert‐type scale

The survey found that less than half of Southern two‐year colleges employed an institutional researcher on a half‐time or greater basis. It also identified a number of discrepancies in the job duties now being performed by institutional researchers and the duties those individuals felt they should be performing

Among those duties which should not be performed were external reporting, program supervision, grant development, and clerical functions. Tasks identified as needing to be performed included increased research project completion, academic program evaluation, planning studies, outcomes assessment, financial analysis, and policy development     

Personalized learning for the post‐mechanical age

This study describes an approach to learning and teaching that is structured as a project‐based context‐driven inquiry. The approach is positioned at the interface between knowledge‐generation and use, and grounded in a generic notion of responsibility for the future of bodily life. The intention is to move the debate beyond the exhausted language of rigid oppositions between the academic and vocational, the universal and contextual. The purpose is to identify and nurture a personal portfolio of competencies responding to the contemporary material condition of humanity. It is expressed in terms of the student's learning power, a manifold of new assessment criteria and methodological steps constitutive of what a student could achieve having progressed through a given course. This is an approach in which competencies are outcomes supported rather than led by subject knowledge. The course structure combines traditional instruction with innovative project and assessment components and also provides an opportunity for the student to get acquainted with an employment niche. The practical applications of this approach at university and secondary‐school levels have led to encouraging results for both staff and learners.