On-demand testing and maintaining standards for general qualifications in the UK using item response theory: possibilities and challenges

Background:?Although on-demand testing is being increasingly used in many areas of assessment, it has not been adopted in high stakes examinations like the General Certificate of Secondary Education (GCSE) and General Certificate of Education Advanced level (GCE A level) offered by awarding organisations (AOs) in the UK. One of the major issues with on-demand testing is that some of the methods used for maintaining the comparability of standards over time in conventional testing are no longer available and the development of new methods is required.

Purpose:?This paper proposes an item response theory (IRT) framework for implementing on-demand testing and maintaining the comparability of standards over time for general qualifications, including GCSEs and GCE A levels, in the UK and discusses procedures for its practical implementation.

Sources of evidence:?Sources of evidence include literature from the fields of on-demand testing, the design of computer-based assessment, the development of IRT, and the application of IRT in educational measurement.

Main argument:?On-demand testing presents many advantages over conventional testing. In view of the nature of general qualifications, including the use of multiple components and multiple question types, the advances made in item response modelling over the past 30 years, and the availability of complex IRT analysis software systems, coupled with increasing IRT expertise in awarding organisations, IRT models could be used to implement on-demand testing in high stakes examinations in the UK. The proposed framework represents a coherent and complete approach to maintaining standards in on-demand testing. The procedures for implementing the framework discussed in the paper could be adapted by people to suit their own needs and circumstances.

Conclusions:?The use of IRT to implement on-demand testing could prove to be one of the viable approaches to maintaining standards over time or between test sessions for UK general qualifications.     

Teachers' estimates of candidates' grades: Curriculum 2000 Advanced Level Qualifications

In the UK, estimated grades have long been provided to higher education establishments as part of their entry procedures. Since 1994 they have also been routinely collected by awarding bodies to facilitate the grade‐awarding process. Analyses of required estimates to a British awarding body revealed that teachers' estimates of candidates' Curriculum 2000 A level grades in the first year of awarding demonstrated an unprecedented degree of accuracy. Estimates of AS level grades conversely showed relative imprecision. Accuracy of the A‐level estimates was most likely bolstered by feedback inherent to the modularisation of the examination while the weakness of AS level estimates may have been a consequence of the comparatively unfamiliar standard at which this qualification was set. As in previous research, and for both GCE qualifications, when estimates were inaccurate they were more commonly optimistic than pessimistic.     

Does workplace training participation vary by type of secondary level qualification? England and Germany in comparison

ABSTRACT

Existing evidence shows that the higher the level of education the higher the likelihood to participate in workplace training. However, we know little about training participation of individuals educated to the secondary level, and whether this may vary by the type of qualification attained, i.e. vocational or general. Vocational qualification holders are known to find employment sooner after school than those with a general qualification but we do not know whether they are also more likely to participate in workplace training. Using data from the 2012 Programme of International Assessment of Adult Competencies (PIAAC) and logistic regressions, we investigate whether vocational qualification holders are more likely to participate in workplace training than general ones in Germany and England. Germany is a coordinated labour market economy with a large emphasis on the provision of vocational qualifications which facilitate school-to-work transitions whereas England is an example of liberal labour markets where the focus is on general qualifications. Results show that in Germany general qualification holders are more likely to participate in training compared to vocational ones; however, differences reduce when controlling for endogeneity. In contrast, in England there is no difference between types of qualification.     

The Validity and Value of National Vocational Qualifications: the case of purchasing

Abstract

This paper outlines the process through which competence‐based National Vocational Qualifications in Purchasing have been developed against the context of criticisms of the concept itself and the methodology adopted. It is argued that the validity of such qualifications lies in practitioner involvement in their development and assessment. They also have value through their incorporation in organisational personnel selection and development processes. Ultimately, however, their validity and value remain to be established through empirical research.     

Adopting National Vocational Qualifications in the Offshore Industry: A Case Study

Abstract

This paper explores the feasibility of introducing National Vocational Qualifications into the Offshore industry. A case study is used to examine the changing perceptions of trainers as they work through the implications of adopting the new framework. Conceptualisations developed in the educational innovation literature are used to help understand the change process. Three areas of concern are identified which illustrate a gap between the aims and practicalities of implementing the qualifications. The conclusion differentiates four emerging research issues.     

Purpose,Value and Competence: Contextualising competence based assessment in the civil aviation sector

Abstract

This paper outlines some of the issues emerging from research in to competence based training and assessment, and National Vocational Qualifications. A model for analysing the key purposes of formal assessment systems is suggested and a conceptual framework is presented which uses the notions of use and exchange value for gauging the worth of qualifications. The author presents material collected during a study of the training and assessment of student air traffic controllers to illustrate how contextual features, particularly at the occupational level, can affect the purpose and design of a qualification as well as the value attached to it by recipients and employers. It is suggested that elements such as: ‘purpose’, ‘perceived value’ ‘qualification design’ and ‘context’ provide a conceptual framework within which further understanding of the role of qualifications can be achieved. The longer term aspiration of the analysis begun in this paper is the development of a context‐sensitive theory of occupational qualifications.     

A Model for Wholesale and Retail Assessment Centres in Regulated Occupational Learning in South Africa

ABSTRACT

South Africa's new occupational and vocational qualifications require assessment centres with a national footprint to conduct national summative assessments which are convenient and cost effective for learners. This new assessment process requires specialised facilities, but few current facilities can provide these. The research project proposed a model which is able to serve the needs of the quality assurance body and industry and be appropriate to the retail sector occupational qualifications. A qualitative method was used, involving a literature review and secondary data analysis; small focus groups; semi-structured interviews; and follow-up workshops to peer review the findings. The participants included industry experts, education providers and quality assurance experts. Two models were identified and considered, namely: Model A, which involves private businesses that can be accredited to become assessment centres; and Model B, which uses Technical Vocational Education and Training (TVET) colleges for all national assessments. The overall finding was that Model B is more suitable.     

The Role of Functional Analysis in National Vocational Qualifications: A Critical Appraisal

Abstract

The article provides a critical evaluation of functional analysis and its centrality in the competence philosophy underpinning reform of the UK's national system of vocational qualifications. Definitions and meanings of competence are examined to locate functional analysis in its theoretical context. The origins of the use and application of the term competence to the vocational qualification (VQ) system are also examined to provide an historical context. Finally, the conceptual model adopted to apply notions of competence and the process of functional analysis is evaluated as part of the critical appraisal.

The theoretical and conceptual analysis is supported by the findings of a small piece of primary research and published evidence from other sources. Both the conceptual analysis and the empirical evidence, in the view of the authors, argue towards a serious questioning of the appropriateness of functional analysis. The results of applying functional analysis are argued to be technical and narrow specifications of competence which fail to address the broader skills and social contexts influencing performance in the workplace.     

New Qualifications in IT: A Pilot Study at Hull College

Abstract

The General National Vocational Qualification (GNVQ) is enjoying growing popularity in the further education (EFE) sector in England and Wales. GNVQs have been recently introduced in the Information Technology (IT) vocational area, to augment or replace existing qualifications. These new courses have been piloted (academic year 1994‐199S) at several centres, including Hull College. This paper focuses on the Advanced level GNVQ in IT, contrasting it with existing qualifications (A level and National Diploma); reflections on the Hull College experience are also provided

The GNVQ (Advanced) is seen not only as leading directly to employment (like the traditional National Diploma), but also as a route into higher education (HE). Growing numbers of universities now accept the GNVQ as an entry qualification. For the first time, a vocational course is now perceived increasingly as a genuine alternative to the A level. The implications for universities are examined, with predictions as to the likely strengths and weaknesses of different categories of university entrant. The importance of choosing the right HE course, now that university provision has become increasingly diverse in the computing area is also highlighted. This paper extends and provides full discussion of the preliminary findings introduced at the 3rd Annual Conference on the Teaching of Computing held at Dublin City University in August 1995.     

Portfolio Assessment as National Policy: the National Council for Vocational Qualifications and its quest for a pedagogical revolution

The National Council for Vocational Qualifications established assessment procedures for qualifications in England, Wales and Northern Ireland in which portfolios play a central role. The relevant qualifications (National Vocational Qualifications and General National Vocational Qualifications) have been taken by large numbers of young people in upper secondary schooling, and young people and adults in work or in government training schemes; and are intended to have labour market currency and to provide routes into further and higher education. The use of portfolios derives from a commitment to direct and comprehensive assessment of the whole ‘syllabus’, but also from a belief in certain pedagogical approaches and in enhancing students' independence and initiative. There is evidence that the approach has affected learning styles, but major problems have arisen relating to the manageability of the approach and the reliability of assessor judgements.     

中外合作开办高等教育课程的政策与现状分析

中外合作办学已经成为我国教育事业的一个组成部分。然而,我国高等教育领域中的中外合作办学现状如何呢?这是国内外教育机构都十分关注的话题。本首先介绍了我国中外合作办学的管理和条例,这些规定也适用于香港、澳门和台湾;进而以国内25所高校与境外147个教育机构的116项合作办学项目为例,概括了中外合作办学的五种形式：境外课程、境外资历;双方课程、境外资历;双方课程、双方资历;境外课程、境外资历、国内证明书;双方课程、国内资历、境外证明书,并对每一种形式的教学方式进行举例说明;最后,作综合讨论了三个相关主题：优质的境外远程教育学历课程进入内地市场的挑战;未来教育的新模式对中外合作办学形式的挑战;香港高校与内地合作办学的独特优势。     

The Perfectionists and the School

Abstract

The statistical correlation between entry qualifications (A‐level scores) and degree results (proportion of first and upper second (i.e. good) degrees) is described for eight subjects over a 21‐year period. A significant positive correlation is demonstrated, but the strength of the relationship varies between subjects. More detailed analysis reveals that the nature of the relationship also varies at institutional (effectively departmental) level with some displaying consistently counter‐intuitive combinations of above‐average entry qualifications and below‐average proportions of good degrees (and vice versa). These results have implications for current policy debates regarding value‐added and standards in UK universities. They also identify some directions for future research.     

NCVQ and the 16‐19 Curriculum

Abstract

In Britain, the National Council for Vocational Qualifications is developing a new framework of vocational qualifications (NVQs). This paper presents an analysis of the place of this emerging system within the broader context of 16‐19 education. The needs for reform of 16‐19 provision are identified, and the potential contribution of NVQs to meeting them is analysed. Firstly, it is suggested that NVQs may not contribute sufficiently to student/trainee empowerment, because of employer control of the system. Secondly, while NVQs may be well suited to the training of adults in employment, it is suggested that they are ill‐equipped to meet the needs of students on full‐time vocational courses. Based on this analysis, three alternative future scenarios for 16‐19 education and training are presented, and the strengths and weaknesses of each identified. It is concluded that the extensive use of NVQs within 16‐19 education and training is likely to be counterproductive.     

Home-Based Child Care Provider Education and Specialized Training: Associations With Caregiving Quality and Toddler Social-Emotional and Cognitive Outcomes

ABSTRACT

Research Findings: Although there has been considerable research on the associations between the qualifications of teachers in center-based settings and preschool-age children’s developmental outcomes, very little is known about the relationships between home provider qualifications and the developmental outcomes of toddlers who attend licensed family child care settings or unregulated family, friend, and neighbor care settings. Analyzing a sample of toddlers and their providers drawn from the Early Childhood Longitudinal Study, Birth Cohort, we found positive relationships between home-based quality and higher education degree (defined as an associate’s degree or at least a bachelor’s degree), field of major, and coursework in early childhood education or a related field. However, provider qualifications were unrelated to children’s cognitive outcomes and related to a limited number of social-emotional outcomes. Practice or Policy: Our results suggest that as states consider the spectrum of supports needed for strengthening home-based child care in ways that facilitate responsive and developmentally supportive caregiving, strategies should include opportunities for home-based care providers to pursue higher education. A more nuanced analysis of the content and comprehensiveness of providers’ formal education is needed to better understand relationships between toddlers’ social and cognitive development and providers’ formal education.     

MovementScapes as ecomotricity in ecopedagogy

ABSTRACT

This elaboration of an ecopedagogy in movementscapes aims to present an empirically informed account of the concept of ecomotricity as manifested in the living body interacting in/with nature (human-and-other-than-human). This interaction is ludic (where pleasure or joy/happiness gives meaning to the lived experience) and ecological (ecosomaesthetic-environmentally ethical-ecopolitical) and provides for revitalized and animated ecopedagogical practices (and research). Critical examples and insights are presented as praxical evidence of how the ecophenomenological and ludic essence of ecomotricity challenges individuals in particular movementscapes to question their ways of being-in-the-world as a form of ecobecoming potentiality. This step towards the (de)(re)construction of environmentally oriented outdoor experiential learning in moving body-time-space relationalities is, potentially, significant to overcoming some of the ontological limits to rational change too often uncritically presumed pedagogically regarding human-nature relations.     

The Technical and Vocational Education Initiative and economic performance in the United Kingdom: an initial assessment

Government support for the Technical and Vocational Education Initiative (TVEI) is based largely on the belief that it will contribute to the improvement of the UK's economic performance. This paper reviews evidence on the relationship between education and training, workforce skills and economic performance, and concludes that it would be helpful if TVEI's objectives were to be expressed in more detail, so as to provide a better basis for evaluation.     

The Stratification of Youth Training∗

Abstract

This article is based on surveys in 1988 among representative samples totalling 4,247 16‐19 year olds from four areas (Kirkcaldy, Liverpool, Sheffield and Swindon), of whom 1,562 had some experience of Youth Training (YT). The young people's likelihood of entering YT depended primarily on their educational qualifications and where they lived. Trainees’ chances of being kept on by the firms where they were trained depended mainly on their types of schemes and their performances during training, indicated by whether or not they earned any additional qualifications. The chances of those not retained moving immediately to jobs with other firms were more dependent on their school‐leaving qualifications and where they lived. The authors interpret their evidence to argue that employers’ uses of YT in the late‐1980s were creating increasingly clear divisions between sponsorship, contest and warehousing sectors.

     

Validating ‘optimizing’ concepts: the importance of personal resolve,effective functioning,and academic striving

Abstract

Capitalising on current research progress, we explored via means of structural equation modelling (SEM), the operational nature of three newly developed optimising concepts: personal resolve, effective functioning, and academic striving. We conceptualised self-efficacy as a source of origin, and effort expenditure and motivation towards learning as adaptive outcomes of the three mentioned concepts. Correlational responses from Taiwanese university students (N?=?1010) affirmed our hypothesis regarding the operational nature of personal resolve, effective functioning, and academic striving – for example – (i) self-efficacy as a source of origin of personal resolve, effective functioning, and academic striving, and (ii) the effects of personal resolve and academic striving on effort, and the effects of effective functioning and academic striving on motivation towards learning. This significant evidence, overall, contributes to the study of optimal achievement best.     

Othermotherwork: testimonio and the refusal of historical trauma

Abstract

This paper is an ofrenda (offering), a testimonio (testimony) of the healing power of reconstituting severed relationships and reconstructing agentic creation stories in the pathology of soul-wounds where pictures and cuentos serve to mend genealogical traumas. This paper is a refusal of neglecting traumas, it is othermotherwork as an invitation of kin towards the healing soul-wounds by engaging my ageing father (apá), who has been battling a debilitating disease for over 30 years. In choosing to erase his memory and refusing well-being, I engage my apá by remembering and restoring cuentos (stories) retrieved from pictures in my abuela’s archival orange box. I retell the cuentos as told by my apá of his mother, father, great-grandmother and great-grandfather, while he offers an unexpected remedy for both our soul-wounds. In this exchange of ancestral herstories, it is an important move towards an Indigenous Else, transits beyond borders and separation, towards the restorative act of forging unity, love and compassion amongst my father, my ancestors and myself.