Validity of admission decisions based on assessment of prior learning in higher education

Assessment of prior learning (APL) refers to the process of validating individuals’ learning in a variety of contexts, representing a relatively new practice in many countries. In higher education, APL is used to receive access and credits based on skills and knowledge acquired mainly outside of formal academic settings. This paper focuses on validity of admission decisions based on this type of assessment in higher education. The study examines decisions made by higher education institutions for approximately 600 applicants who have used APL in order to receive admission to, and credits in the vocational teacher education programme in Sweden. The results are analysed and presented in relation to a validity discussion. A conclusion is that the existing practice of APL needs improvements in order to obtain validity and trustworthiness in the decisions made in relation to APL.     

Threats to the valid use of Assessment of Prior Learning in higher education: claimants’ experiences of the assessment process

Assessment of Prior Learning (APL) refers to a process where adults’ prior learning, formal as well as informal, is assessed and acknowledged. In the first section of this paper, APL and current conceptions of validity in assessments and its evaluation are presented. It is argued that participants in the assessment are an important source of information for the validation of the assessment. In the following section participants’ experiences from a particular APL scheme are evaluated using a questionnaire developed for that purpose. The questionnaire provides data on individuals’ perceptions of the procedure and result of the APL scheme. The results are described, analysed and discussed from a validity perspective. Conclusions drawn from the results are that possible threats to validity can exist in the administration of APL procedures, as well as in consequences of APL.     

The quality of procedures to assess and credit prior learning: Implications for design

The ever-changing requirements of working life require individuals to develop their competencies throughout their life cycle. This lifelong learning paradigm requires a renewed vision concerning assessment in which, besides formal learning, informal and non-formal learning experiences are also recognized. To support this lifelong learning paradigm, procedures have been developed worldwide to assess and credit prior learning experiences (APL). While research on APL stresses the importance of a high-quality standard, so far the literature has applied only a psychometric quality framework. However, from the perspective of APL, where, besides prior knowledge and skills, competencies need to be measured, it is more appropriate to use a combination of the psychometric and edumetric quality criteria. This article will analyze and describe the relationship between quality criteria and the characteristics of APL. The results have revealed that quality criteria based on both are fundamental for APL, but that some criteria are more recognized than are others. Based on this analysis, design guidelines for APL have been formulated.     

Self‐assessment in university assessment of prior learning procedures

Competency‐based university education, in which lifelong learning and flexible learning are key elements, demands a renewed vision on assessment. Within this vision, Assessment of Prior Learning (APL), in which learners have to show their prior learning in order for their goals to be recognised, becomes an important element. This article focuses on a first step in APL, namely students’ self‐assessment of their prior learning before entering university education. The main aim of the presented study is to examine the suitability of the use of self‐assessment in APL. First, in an explorative study, the main sources for self‐assessment are derived and the relation between sources and domain of study is investigated. Second, in a pre‐test post‐test research design, the hypothesis that students’ self‐assessment of prior learning related to a course changes after studying a domain‐specific course is tested. Pre‐test results reveal that students indicate that they have prior knowledge related to the chosen university programme. In general, this prior learning is obtained from study experience, work experience, books, newspapers, magazines, the Internet, TV, radio, film or video. A relation is found between the type of source and the university programme. The hypothesis that students change their self‐assessment after a study period could not be confirmed. Based on these results, it is concluded that self‐assessment in APL might be a suitable tool. Implications for further research are discussed.     

Assessors’ approaches to portfolio assessment in Assessment of Prior Learning procedures

In an effort to gain better understanding of the assessment of prior informal and non‐formal learning, this article explores assessors’ approaches to portfolio assessment. Through this portfolio assessment, candidates had requested exemptions from specific courses within an educational programme or admission to the programme based on their prior learning. The assessors judged the portfolios according to set rating criteria, and subsequently discussed their approaches. Their decision‐making processes, perception of portfolio use in the Assessment of Prior Learning (APL), deciding factors in portfolio assessment and use of the rating criteria were key elements in this discussion. The results show that they do use the rating criteria as an indicator in decision‐making, but have mixed perceptions regarding the fairness of APL portfolio assessment. They perceive the portfolio evidence in combination with sound argumentation as the deciding elements in portfolio assessment.     

论教学观的认知转向

杜威从逻辑的角度对思维活动做出了"数据—归纳—演绎—证实"的解释。利用当代语言学中形式学习理论可以对教学或学习过程做出一种认知解释,经验探究的学习过程是主体(学习者)从外界获取信息(数据)、加工信息(做出假设并根据假设进行推理)、确认信息的认知过程。这种认知解释的逻辑基础是当代动态认知逻辑。根据认知解释,可以对教学展开语言学、心理学、博弈论、信息科学、哲学等多学科交叉研究。     

A General Framework for the Validation of Embedded Formative Assessment

In educational practice, test results are used for several purposes. However, validity research is especially focused on the validity of summative assessment. This article aimed to provide a general framework for validating formative assessment. The authors applied the argument‐based approach to validation to the context of formative assessment. This resulted in a proposed interpretation and use argument consisting of a score interpretation and a score use. The former involves inferences linking specific task performance to an interpretation of a student's general performance. The latter involves inferences regarding decisions about actions and educational consequences. The validity argument should focus on critical claims regarding score interpretation and score use, since both are critical to the effectiveness of formative assessment. The proposed framework is illustrated by an operational example including a presentation of evidence that can be collected on the basis of the framework.     

Formal theories clarify the complex: Generalizing a neural process account of the interaction of visual exploration and word learning in infancy

The interaction of visual exploration and auditory processing is central to early cognitive development, supporting object discrimination, categorization, and word learning. Research has shown visual–auditory interactions to be complex, created from multiple processes and changing over multiple timescales. To better understand these interactions, we generalize a formal neural process model of early word learning to two studies examining how words impact 9- to 22-month-olds' attention to novelty. These simulations clarify the origin and nature of previously demonstrated effects of labels on visual exploration and the basis of mutual exclusivity effects in word learning. We use our findings to discuss key questions for this special section: what makes a good theory and how should formal theories interface with empirical paradigms and findings?     

Validity issues in standard-setting studies

Standard-setting procedures are a key component within many large-scale educational assessment systems. They are consensual approaches in which committees of experts set cut-scores on continuous proficiency scales, which facilitate communication of proficiency distributions of students to a wide variety of stakeholders. This communicative function makes standard-setting studies a key gateway for validity concerns at the intersection of evidentiary and consequential aspects of score interpretations. This short review paper describes the conceptual and empirical basis of validity arguments for standard-setting procedures in light of recent research on validity theory. It specifically demonstrates how procedural and internal evidence for the validity of standard-setting procedures can be collected to form part of the consequential basis of validity evidence for test use.     

Evaluating Content-Related Validity Evidence Using a Text-Based Machine Learning Procedure

Validity evidence based on test content is critical to meaningful interpretation of test scores. Within high-stakes testing and accountability frameworks, content-related validity evidence is typically gathered via alignment studies, with panels of experts providing qualitative judgments on the degree to which test items align with the representative content standards. Various summary statistics are then calculated (e.g., categorical concurrence, balance of representation) to aid in decision-making. In this paper, we propose an alternative approach for gathering content-related validity evidence that capitalizes on the overlap in vocabulary used in test items and the corresponding content standards, which we define as textual congruence. We use a text-based, machine learning model, specifically topic modeling, to identify clusters of related content within the standards. This model then serves as the basis from which items are evaluated. We illustrate our method by building a model from the Next Generation Science Standards, with textual congruence evaluated against items within the Oregon statewide alternate assessment. We discuss the utility of this approach as a source of triangulating and diagnostic information and show how visualizations can be used to evaluate the overall coverage of the content standards across the test items.     

Peer assessment for learning from a social perspective: The influence of interpersonal variables and structural features

This paper reports a systematic literature review examining empirical studies on the effects of peer assessment for learning. Peer assessment is fundamentally a social process whose core activity is feedback given to and received from others, aimed at enhancing the performance of each individual group member and/or the group as a whole. This makes peer assessment an interpersonal and interactional process. Using this social perspective in order to study learning effects, we focus on the impact of the structural arrangement of peer assessment on learning, and the influence of interpersonal variables.The literature search, focusing on empirical studies measuring learning outcomes in a peer assessment setting, resulted in 15 studies conducted since 1990 dealing with effects (performance or perceived learning gains) of peer assessment. Our analysis reveals that, although peer assessment is a social process, interpersonal variables have hardly been studied; more specifically, they were measured in only 4 out of 15 studies. Moreover, they are not used to explain learning gains resulting from peer assessment. Finally, comparing the studies with respect to structural features reveals that, although the differences between the studies are significant, there seems to be no relation with the occurrence of learning benefits. The results of this review seem to indicate that research on peer assessment from a social perspective is still in its infancy and deserves more attention.     

大学生学习适应状况量表的编制

在有关当代大学生学习适应状况研究的基础上,探讨了大学生学习适应状况的主要因素,编制了一套适合当代大学生学习适应状况的量表,对9所院校900名大学生进行施测,对所得数据进行探索性因素分析。结果表明,大学生学习适应状况的主要因素包括:学习方法、学习热情、学习态度、专业兴趣、学习动力、学习环境等六个因素。验证性因素分析结果表明,所提取的6个因素与构想模型拟合较好,测验具有较好的信效度。     

Articulating the validity evidence for a science alternate assessment

Students with the most significant cognitive disabilities (SCD) are the 1% of the total student population who have a disability or multiple disabilities that significantly impact intellectual functioning and adaptive behaviors and who require individualized instruction and substantial supports. Historically, these students have received little instruction in science and the science assessments they have participated in have not included age‐appropriate science content. Guided by a theory of action for a new assessment system, an eight‐state consortium developed multidimensional alternate content standards and alternate assessments in science for students in three grade bands (3–5, 6–8, 9–12) that are linked to the Next Generation Science Standards (NGSS Lead States, 2013 ) and A Framework for K‐12 Science Education (Framework; National Research Council, 2012 ). The great variability within the population of students with SCD necessitates variability in the assessment content, which creates inherent challenges in establishing technical quality. To address this issue, a primary feature of this assessment system is the use of hypothetical cognitive models to provide a structure for variability in assessed content. System features and subsequent validity studies were guided by a theory of action that explains how the proposed claims about score interpretation and use depend on specific assumptions about the assessment, as well as precursors to the assessment. This paper describes evidence for the main claim that test scores represent what students know and can do. We present validity evidence for the assumptions about the assessment and its precursors, related to this main claim. The assessment was administered to over 21,000 students in eight states in 2015–2016. We present selected evidence from system components, procedural evidence, and validity studies. We evaluate the validity argument and demonstrate how it supports the claim about score interpretation and use.     

Recruitment to the secondary technical schools

Abstract

There has developed over the last few years a growing interest in the Accreditation of Prior Learning (APL) as a route to vocational qualifications. This is a system, supported by the major awarding bodies, whereby evidence of competence based on past experience and achievements is given credit towards qualifications. In a recent number of The Vocational Aspect of Education, Slusarchuk and Nicholl (1990) reviewed issues surrounding the creation of “an advocacy portfolio” designed to compile such evidence. The present article describes an approach to evaluating or assessing this evidence of prior learning with a view towards awarding National Vocational Qualification Units. A methodology is offered, through which the validity and reliability of APL assessment decisions may be enhanced.     

Mobile Assisted Chinese Learning as a Foreign Language: An Overview of Publications between 2007 and 2019

This paper presents a critical overview of studies on mobile assisted language learning (MALL) in teaching Chinese as a foreign language (CFL) during the period 2007-2019. In the review, keyword and reference searches were conducted to identify and select empirical studies during the review period. Thematic and frequency analyses were employed on the data. This identified methodological trends and research outcomes in the reviewed studies. As shown in the results, most of the reviewed studies used qualitative methods to examine the effect of mobile CFL learning on formal learning in higher education settings. These studies document the positive impact that mobile technology has on CFL learning. Their attention is primarily on the use of mobile learning in Chinese vocabulary acquisition, language skill development and mobile seamless learning. Suggestions are provided for further research to support continuous mobile assisted CFL teaching and learning.     

Helping Teachers Formulate Sound Test Accommodation Decisions for Students with Learning Disabilities

This paper introduces a data-based approach as an alternative way to help teachers formulate decisions about the validity of test accommodations for students with LD. Three rationales for the approach are provided: (a) an inadequate research base to guide decision-making; (b) the heterogeneity of the LD population; and (c) problems with teachers' use of subjective judgment. Well-controlled studies on test accommodations are too scarce to draw firm conclusions about effects for the group of students labeled learning disabled (LD). Moreover, in light of the heterogeneity of learning disabilities, the individual, rather than the LD label, may be the more appropriate unit for deciding which test accommodations preserve the validity of test scores for students with LD. In this paper, we provide a rationale for a data-based approach to help teachers formulate decisions about the validity of test accommodations for individual students with LD. Then we describe an objective assessment process teachers may use in determining valid test accommodations. We conclude with recommendations for practitioners.     

Using self- and peer-assessments for summative purposes: analysing the relative validity of the AASL (Authentic Assessment for Sustainable Learning) model

The purpose of this paper is to provide a proof of concept of a collaborative peer-, self- and lecturer assessment processes. The research presented here is part of an ongoing study on self- and peer assessments in higher education. The authentic assessment for sustainable learning (AASL) model is evaluated in terms of the correlations between sets of marks. The article provides an explanation of the assessment process, and analyses sets of marks as a means of justifying the validity of the process. The results suggest that students, even those with no prior experience in peer- or self-evaluation, in their first year of tertiary study, under the right conditions, are able to accurately judge their own work and make reasonably accurate judgements of the work of their peers. While previous studies have expounded the benefits of self- and peer assessments in tertiary study, undertaking a prescribed process, such as AASL, has a further implication in allowing others to replicate the process with reasonable assuredness of the validity of the process across various fields of study.     

Students’ Interpretation of Formative Assessment Feedback: Three Claims for Why We Know So Little About Something So Important

If K‐12 students are to be fully integrated as active participants in their own learning, understanding how they interpret formative assessment feedback is needed. The objective of this article is to advance three claims about why teachers and assessment scholars/specialists may have little understanding of students’ interpretation of formative assessment feedback. The three claims are as follows. First, there is little systematic research of K‐12 students’ interpretations of feedback. Systematic research requires gathering substantive evidence of students’ cognitive and emotional processes using psychological methods and tools. Second, there is an overemphasis on the external assessment process at the expense of uncovering learners’ internal reasoning and emotional processes. This overemphasis may be due to vestiges of behavioral approaches and lack of training in social cognitive methods. Third, there are psychological tools such as the clinical interview, pioneered by Piaget and used by psychologists to “enter the child's mind,” which may be helpful in uncovering students’ interpretation of feedback and associated behavioral responses. If the purpose of formative assessment is to change student learning, and feedback is delivered as a conduit to help with this long‐term change, understanding students’ interpretation of feedback plays a central role in the validity of the process.     

Expectancy-value and cognitive process outcomes in mathematics learning: a structural equation analysis

Existing research has yielded evidence to indicate that the expectancy-value theoretical model predicts students' learning in various achievement contexts. Achievement values and self-efficacy expectations, for example, have been found to exert positive effects on cognitive process and academic achievement outcomes. We tested a conceptual model that depicted the interrelations between the non-cognitive (task value, self-efficacy) and cognitive (deep-learning approach, reflective-thinking) processes of learning, and academic achievement outcomes in mathematics. University students (n = 289) were administered a number of Likert-scale inventories and LISREL 8.80 was used to test various a priori and a posteriori models. Structural equation modeling yielded some important findings: (1) the positive temporally displaced effects of prior academic achievement, self-efficacy expectations and task value on achievement in mathematics, (2) the positive relations between self-efficacy expectations and task values and cognitive process outcomes and (3) the possible mediating role of self-efficacy expectations and task value between prior academic achievement and deep learning, reflective-thinking practice and academic achievement. Overall, our research investigation has provided empirical groundings for further advancement into this area of students' learning.     

The assessment of teaching in higher education: A critical retrospect and a proposal

Evaluation of college and university teaching is considered in the context of growing demands for accountability of educational institutions and particular roles of faculty in achieving goals for which these institutions are believed to be accountable. In the first of two papers,^* three contemporary aprroaches to evaluation of teaching in higher education are critically reviewed. All three of these approaches-assessment of learning outcomes, assessment of teacher characteristics and analysis of pedagogical behaviors-are found to be defective on logical, theoretical and empirical grounds. A common thread of deficiency is the absence of a coherent theoretical framework for analysis of teaching and phenomena associated with teaching. Specific defecs are analyzed in each of the three approaches as well as in the ubiquitous methodology of rating of teaching performance. In analysis of evaluation by assessment of learning outcomes, teaching is shown to be neither necessary nor sufficient to subsequent learning outcomes. Assessment of teacher characteristics fails to identify those characteristics peculiarly indigenous to teaching as a generic activity. Analysis of pedagogical behaviors fails to distinguish critical teaching acts from more general teacher characteristics interpretable in terms of teacher personality. Furthermore, no adequate basis is provided for normative interpretation of data pertaining to pedagogical behaviors. The use of rating scales to provide data about teacher characteristics or pedagogical behavior rests on assumed rather than demonstrated validity of results. Evidence of validity or meaningfulness is replaced by evidence of consistency which is often spurious. The first paper concludes with an outline of requirements for a more constructive approach to the task of teacher evaluation. In a second paper, an outline of a theory of teaching is sketched which conforms to these requirements. Realization of this theoretical structure in teaching assessment report forms is described. Tentative conclusions from trial use of forms-in-development and recommendations for additional data sources are discussed in terms of their potential contribution to improvement in teaching.