A General Framework for the Validation of Embedded Formative Assessment

In educational practice, test results are used for several purposes. However, validity research is especially focused on the validity of summative assessment. This article aimed to provide a general framework for validating formative assessment. The authors applied the argument‐based approach to validation to the context of formative assessment. This resulted in a proposed interpretation and use argument consisting of a score interpretation and a score use. The former involves inferences linking specific task performance to an interpretation of a student's general performance. The latter involves inferences regarding decisions about actions and educational consequences. The validity argument should focus on critical claims regarding score interpretation and score use, since both are critical to the effectiveness of formative assessment. The proposed framework is illustrated by an operational example including a presentation of evidence that can be collected on the basis of the framework.     

Utility of the International Classification of Functioning,Disability and Health (ICF) for educational psychologists’ work

Despite embracing a bio-psycho-social perspective, the World Health Organization’s International Classification of Functioning, Disability and Health (ICF) assessment framework has had limited application to date with children who have special educational needs (SEN). This study examines its utility for educational psychologists’ work with children who have Autism Spectrum Disorders (ASD). Mothers of 40 children with ASD aged eight to 12 years were interviewed using a structured protocol based on the ICF framework. The Diagnostic Interview for Social and Communication Disorder (DISCO) was completed with a subset of 19 mothers. Internal consistency and inter-rater reliability of the interview assessments were found to be acceptable and there was evidence for concurrent and discriminant validity. Despite some limitations, initial support for the utility of the ICF model suggests its potential value across educational, health and care fields. Further consideration of its relevance to educational psychologists in new areas of multi-agency working is warranted.     

Using technology for teaching and learning in higher education: a critical review of the role of evidence in informing practice

The use of technology for teaching and learning is now widespread, but its educational effectiveness is still open to question. This mixed-method study explores educational practices with technology in higher education. It examines what forms of evidence (if any) have influenced teachers' practices. It comprises a literature review, a questionnaire and interviews. A framework was used to analyse a wide range of literature. The questionnaires were analysed using content analysis and the interviews were analysed using inductive thematic analysis. Findings suggest that evidence has partial influence upon practice with practitioners preferring to consult colleagues and academic developers. The study underscored the difficulty in defining and evaluating evidence, highlighting ontological and epistemological issues. The academic developer's role appears to be key in mediating evidence for practitioners.     

Interpretive validity theory: mapping a methodology for validating educational assessments

Background:?Validity theory has evolved significantly over the past 30 years in response to the increased use of assessments across scientific, social and educational settings. The overarching trajectory of this evolution reflects a shift from a purely quantitative, positivistic approach to a conception of validity reliant on the interpretation of multiple evidence sources integrated into validity arguments. Moreover, within contemporary validity, interpretation has been emphasised as a central process; however, despite this emphasis, there have been few explicit articulations of specific interpretive methodologies applicable to the practice of validation.

Purpose:?To link contemporary theoretical foundations in validity to practical methods and structures to help guide the collection and analysis of interpretive validity evidence. By building upon existing validity theory, this paper aims to provide greater clarity on the practice of validation and contribute toward the larger developing framework for the validation of educational assessments.

Source of evidence:?An interdisciplinary, integrative review of over 60 research articles and sources related to the theory and practice of educational validation and interpretive inquiry approaches. Sources include literature from the fields of educational assessment and more broadly social scientific research.

Main argument:?As assessments in education increasingly aim to measure complex constructs that are value-laden and socially dependant, validity theory must keep pace and evolve in ways that address the inherent complexities associated with contemporary educational assessment. Through this paper, I assert that a greater understanding of interpretive methodologies represents one of the most promising areas for development of validation theory and practice. Specifically, I argue that dialectic, hermeneutic and transgressive forms of inquiry can be integrated within current argument-based structures for the collection, analysis and representation of validity evidence in several useful ways.

Conclusions:?Interpretive inquiry processes, namely dialectic, hermeneutic and transgressive forms of interpretation, serve to expand validation practice to include diverse evidences for the generation of multiple-perspective validity arguments. The paper concludes with specific implications for future research and practice within the field of interpretive validity theory.     

A systematic review on empathy measurement tools for care professionals

ABSTRACT

Empathic understanding of older adults is a critical attribute required for care professionals to provide quality care and relationship-based practice. To assess care professionals’ empathy, identifying an appropriate empathy measurement tool with good psychometric properties is needed. This systematic review aimed to identify empathy measures that can be used for care professionals and evaluate the rigor of empathy measures. Eligible studies published between 1950 and 2018 were extracted from five databases. A five-criteria appraisal framework was used for quality appraisal of empathy measures. A total of 11 empathy measurement tools were included. Based on the scores of the appraisal framework, CARE, JSE and TES were the highest quality empathy measures, and the lowest quality measure was EQ. None of the measures was specifically developed to measure empathy of geriatric care professionals. This review addresses the limitations of the existing empathy measures and suggests future directions for research.     

A review and evaluation of the internal structure and consistency of the Approaches to Teaching Inventory

ABSTRACT

This study presents a review from 39 studies that provide evidence for the structural validity and internal consistency of the Approaches to Teaching Inventory (ATI). In addition to this review, we evaluate many alternative factor structures on a sample of 267 first- and second-year chemistry faculty members participating in a professional development, a sample of instructors for which the ATI was originally designed. A total of 26 unique factor structures were evaluated. Through robust checking of assumptions, compilations of existing evidence, and new exploratory and confirmatory analyses, we found that there is greater evidence for the structural validity and internal consistency for the 22-item ATI than the 16-item ATI. Additionally, evidence supporting the original two-factor and four-factor structures proposed by the ATI authors (focusing on information transmission and conceptual change) were not reproducible and while alternative models were empirically viable, more theoretical justification is warranted. Recommendations for ATI use and general comments regarding best practices of reporting psychometrics in educational research contexts are discussed.     

基础教育质量指数构建的模式与途径

基础教育质量指数的构建可以为教育质量提供科学、有效的评价方法。本文首先回顾了对教育指标体系和教育指数的相关探索,总结了基础教育质量指数构建的模式,为构建我国基础教育质量指数提供参考。结合我国提升基础教育质量的目标和新时代教育评价机制改革的趋势对基础教育质量指数的构建提出的具体要求,本文提出了基础教育质量指数构建的可能途径:以CIPP框架为出发点初步构建并不断完善指标体系;基于效度证据选择投入、过程和结果指标;建立多方参与的综合评价系统;基于质量标准科学地使用和解释指数;在评价质量水平时兼顾公平。同时,为落实我国基础教育质量指数的构建,我们认为,还需要一系列教育实证研究的支撑,未来研究应关注数据链接和填补、质量指标的本土化效度验证、不同水平指数的效度验证、“互联网+”时代的指数构建、综合指数合成方法的比较、结合质量标准的指数使用案例,以及融合质量水平与公平的指数构建等研究主题。     

面向智慧课堂的灵活深度学习设计框架研制

深度学习是智慧教育的核心支柱,但目前尚缺少智慧课堂专属的深度学习设计方案。如何针对深度学习的灵活性诉求,研制一种智慧课堂赋能的灵活深度学习设计框架,是推进深度学习实践落地的关键所在。基于深度学习架构理念、采用教育设计研究范式研制的面向智慧课堂的灵活深度学习设计框架,遵循逆向设计逻辑,包括课堂环境分析、明确目标、确定评估、学生分析、任务设计、编列制定、绘制分布和决策预设等8个步骤,既体现了智慧课堂精准把脉、互动支持、适性推送、即时反馈等四大特色功能,也为智慧课堂中的学习任务、学习活动、学习进程和教学决策等四个方面的灵活性设计提供了详细的方案支持,并且还能够可视化学生深度学习的个性化进程。该框架经过高质量评估标准锚定下的专家校验与迭代修订,已达到可交付实施的质量要求。这一框架在智慧课堂中的应用有助于促使学生深度参与学习,引导学生采用高级学习方略,促进高阶知能的发展,加深概念理解及其迁移应用。     

Systemic action research: a strategy for whole system change

The aim of this action research project was to identify criteria that would best represent competent teaching and incorporate these within an effective teaching appraisal process. An initial literature review provided broad criteria of the process of teaching which had been shown by research to have fostered student learning. These criteria were structured within an initial proposal for an appraisal framework and were shared with staff at. a seminar in the Faculty of Health Studies at the Auckland Institute of Technology. Twelve staff representing six of the departments subsequently volunteered to participate in a research group to further develop this appraisal framework and the appraisal processes. The researcher used the action research processes of collaboration, power sharing and critical reflection to maximise the quality of and the commitment to the new appraisal procedures. This appraisal model features four dimensions of the teaching process which have been differentiated from the wider duties of a teacher's role. It was implemented in the Faculty promotion round, evaluated, and further upgraded for re‐implementation in the following year. The project has resolved several theoretical problems related to evaluating teaching. It has also produced a framework of valid appraisal criteria which can be used to effectively evaluate the quality of teaching. This appraisal process is considered to be based upon generalisable knowledge which could be utilised by other teaching organisations.     

Assessing students’ experiences of teaching–learning environments and approaches to learning: Validation of a questionnaire in different countries and varying contexts

The study examined the use of the modified Experiences of Teaching and Learning Questionnaire (ETLQ) in the Finnish context by focusing on its factor structures and comparing them with those for British data. A total of 2,509 Finnish and 2,710 British students completed the questionnaire. The comparison of the factor structures were conducted using exploratory structural equation modelling (ESEM) and a transformation analysis. Although the differences between the factor structures prevented a combined analysis, the structures were highly similar in the two contexts. The ETLQ appears to be a sufficiently robust and reliable instrument for use across countries and, in addition, at either the level of the degree subject or the single course module.     

Validity in quantitative content analysis

Over the past 15 years, educational technologists have been dabbling with a research technique known as quantitative content analysis (QCA). Although it is characterized as a systematic and objective procedure for describing communication, readers find insufficient evidence of either quality in published reports. In this paper, it is argued that QCA should be conceived of as a form of testing and measurement. If this argument is successful, it becomes possible to frame many of the problems associated with QCA studies under the well-articulated rubric of test validity. Two sets of procedures for developing the validity of a QCA coding protocol are provided, (a) one for developing a protocol that is theoretically valid and (b) one for establishing its validity empirically. The paper is concerned specifically with the use of QCA to study educational applications of computer-mediated communication.     

The role of social capital for teacher professional learning and student achievement: A systematic literature review

This systematic review synthesises research on social capital in relation to teachers and teacher professional learning between the years 2004–2019. The study was guided by the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) Statement and the Weight of Evidence framework for quality and relevance appraisal. After applying eligibility criteria, 66 empirical items were included in the final review. The review finds that social capital among teachers has been associated with five categories of outcomes: 1) teacher professional development, 2) the implementation of change, 3) the introduction of new and beginning teachers, 4) teacher retention and job satisfaction, and 5) improved student achievement. These have, in turn, been associated with the implicit outcome of promoting educational equity. A synthesis of enablers and barriers to building social capital among teachers identifies the pervasive role of organisational structures for moderating the relationship between social capital and these outcomes. Findings indicate that different organisational structures may foster different social capital dimensions, such as bonding, bridging, and linking. More research is needed on the relationship between these dimensions and schools' organisational structure to promote the desired outcomes of teacher social capital identified in this review.     

Nonparametric Evidence of Validity,Reliability, and Fairness for Rater‐Mediated Assessments: An Illustration Using Mokken Scale Analysis

Numerous researchers have proposed methods for evaluating the quality of rater‐mediated assessments using nonparametric methods (e.g., kappa coefficients) and parametric methods (e.g., the many‐facet Rasch model). Generally speaking, popular nonparametric methods for evaluating rating quality are not based on a particular measurement theory. On the other hand, popular parametric methods for evaluating rating quality are often based on measurement theories such as invariant measurement. However, these methods are based on assumptions and transformations that may not be appropriate for ordinal ratings. In this study, I show how researchers can use Mokken scale analysis (MSA), which is a nonparametric approach to item response theory, to evaluate rating quality within the framework of invariant measurement without the use of potentially inappropriate parametric techniques. I use an illustrative analysis of data from a rater‐mediated writing assessment to demonstrate how one can use numeric and graphical indicators from MSA to gather evidence of validity, reliability, and fairness. The results from the analyses suggest that MSA provides a useful framework within which to evaluate rater‐mediated assessments for evidence of validity, reliability, and fairness that can supplement existing popular methods for evaluating ratings.     

Design Research as a Mechanism for Consultants to Facilitate and Evaluate Educational Innovations

Schools across the nation are implementing innovative practices; however, questions remain regarding how to facilitate quality implementation. Research designs that emphasize high degrees of control over independent variables result in findings with internal validity, but that may not generalize to complex, dynamic educational systems. The purpose of this article is to propose a design research framework as a mechanism for consultants to facilitate and evaluate innovation implementation. Information on design research principles and processes is provided, and issues to consider when applying the framework are discussed. An illustration of how a design research framework was applied in a large-scale initiative to implement and evaluate Response to Intervention (RtI) implementation is also provided. Finally, issues and questions to consider relative to consultants' use of design research principles are explored.     

Screening systems and decision making at the preschool level: Application of a comprehensive validity framework

Universal screening is designed to be an efficient method for identifying preschool students with mental health problems, but prior to use, screening systems must be evaluated to determine their appropriateness within a specific setting. In this article, an evidence‐based validity framework is applied to four screening systems for identifying preschool students with mental health problems. The framework is influenced by the most recent standards for educational and psychological testing, research on test accessibility, and considerations for evaluating screening systems. Suggestions are provided for evaluating the accessibility (Step 1), reliability (Step 2), construct validity (Step 3), and consequential validity (Step 4) of an instrument. Other factors for consideration (i.e., developmental stage, incremental validity, and generalizability) are also identified. Special attention is given to conditional probability indices, which are highly relevant to evaluation of screening systems, given the dichotomous nature of decision making in preschool mental health. The authors suggest that this framework be used, along with specification of the construct of interest and characteristics of the environment, to identify the appropriate method to be used for each preschool screening decision. © 2011 Wiley Periodicals, Inc.     

The reliability and validity of the cultural congruity and university environment scales with Chicana/o community college students

Following the calls for increased research on the educational experiences of Chicana/o community college students, and the development of culturally applicable measures for communities of color, this study examined the utility and the applicability of the Cultural Congruity Scale (CCS) and University Environment Scale (UES) for use with Chicana/o community college students. Applying a psychosociocultural framework, the reliability, construct, and criterion-related validity of the scales for use with a sample of 110 Chicana/o community college students was examined. Results demonstrated adequate reliability and construct validity, with indication of applicability of these scales for the study’s sample. Overall, the study challenges normative practices in educational research that students—despite their race/ethnicity, backgrounds, and histories—face similar educational experiences. Implications are discussed.     

The Appraisal of Educational Psychologists: A very private affair

This paper presents the results of a survey into the use of appraisal in educational psychology services within Local Education Authorities in England and Wales. Based on the findings of the study, the current range of appraisal schemes in use within the educational psychology service are described. The way in which such schemes link with appraisal in the wider local authority context is considered, and an attempt is made to identify whether appraisal schemes used with educational psychologists have distinctive features and, if so, what those features are.     

Using psychometric technology in educational assessment: The case of a schema-based isomorphic approach to the automatic generation of quantitative reasoning items

This article deals with the investigation of the psychometric quality and constructs validity of algebra word problems generated by means of a schema-based version of the automatic min–max approach. Based on review of the research literature in algebra word problem solving and automatic item generation this new approach is introduced as a theory-based top–down method of automatic item generation featuring a quality control framework aimed to minimize the construct unrelated variance in the item parameters. The first study deals with the evaluation of an initial set of items. The results are replicated in the second study using a larger item set which also allows the investigation of the construct representation of the generated item. Since construct unrelated variance components (e.g. reading comprehension) have been controlled for in the item generation phase the results revealed some interesting insights into the cognitive processes of the actual mathematization phase of algebra word problem solving. The third study investigated the nomothetic span is using hierarchical confirmatory factor analysis. The results argue for the convergent and discriminant validity of the automatically generated items. Taken together, the results indicate that the automatic generation of construct valid algebra word problems at a high psychometric level is viable. The discussion is thus concerned with the implications of this new approach to item generation for theory development and evaluation as well as practical benefits for educational assessment and the development of intelligent tutoring systems.     

A Critical Analysis of Assessment Quality in Genomics and Bioinformatics Education Research

The growing importance of genomics and bioinformatics methods and paradigms in biology has been accompanied by an explosion of new curricula and pedagogies. An important question to ask about these educational innovations is whether they are having a meaningful impact on students’ knowledge, attitudes, or skills. Although assessments are necessary tools for answering this question, their outputs are dependent on their quality. Our study 1) reviews the central importance of reliability and construct validity evidence in the development and evaluation of science assessments and 2) examines the extent to which published assessments in genomics and bioinformatics education (GBE) have been developed using such evidence. We identified 95 GBE articles (out of 226) that contained claims of knowledge increases, affective changes, or skill acquisition. We found that 1) the purpose of most of these studies was to assess summative learning gains associated with curricular change at the undergraduate level, and 2) a minority (<10%) of studies provided any reliability or validity evidence, and only one study out of the 95 sampled mentioned both validity and reliability. Our findings raise concerns about the quality of evidence derived from these instruments. We end with recommendations for improving assessment quality in GBE.     

Facebook: a potentially valuable educational tool?

This paper is concerned with the educational value of Facebook and specifically how it can be used in formal educational settings. As such, it provides a review of existing literature of how Facebook is used in higher education paying emphasis on the scope of its use and the outcomes achieved. As evident in existing literature, Facebook has been used mainly for social networking purposes through the establishment and collaboration of social groups in educational settings. However, a set of recent studies has exemplified how Facebook can provide an empowering means for achieving educational goals and supporting students develop crucial skills (e.g., writing, networking, collaborating) by serving as members in various learning communities. Concluding, we argue that Facebook can provide a valuable pedagogical tool that enhances student learning. Hence, future research towards further exploring Facebook’s use in educational settings is warranted for the purpose of producing scientific evidence about the ways in which Facebook could be utilized to enhance learning.