Evaluating the potential effect of the increased importance of the impact component in the Research Excellence Framework of the UK

The UK's Research Excellence Framework (REF) is a system that is intended to evaluate the quality of the research produced by higher education institutes (HEIs) in the UK in three areas: quality of research outputs; impact of this research beyond academia; and research environment. For the next REF, the funding bodies have reviewed the importance of the three assessment elements and decided to increase the weight of ‘impact’ to 25% (from 20% in REF2014) and decrease the weight of ‘outputs’ to 60% (from 65% in REF2014). This article first examines the relevance of some factors for the quality of impact submissions in REF2014 and finds that larger submissions and institutes with higher external research income received better impact scores in the REF. The article then examines the units of assessment (UoAs) and HEIs that benefitted from the inclusion of the impact agenda as part of REF2014 by examining the distribution of the quality-related research (QR) funding in the 2017–2018 period and finds that the QR funding gap among different UoAs tends to decrease but the gap among HEIs in most of the UoAs increased. With the increased importance of the impact agenda as a criterion for funding bodies, it is expected that research income will be concentrated in fewer universities in the future, with the increased importance of non-academic impact. This article also discusses some of the gaming strategies and long-term investment priorities that HEIs may engage in based on the new submission rules of the next REF.     

Reconciling rigour and impact by collaborative research design: study of teacher agency

This paper illustrates a new way of working collaboratively on the development of a methodology for studying teacher agency for social justice. Increasing emphasis of impact on change as a purpose of social research raises questions about appropriate research designs. Large-scale quantitative research framed within externally set parameters has often been criticised for its limited potential for capturing the contexts and impacting change, while smaller, locally embedded, mostly qualitative inquiries have been questioned on the grounds of their limited generalizability and sometimes compromising research rigour. New ways of working collaboratively are increasingly explored as a way of reconciling research rigour and impact. The paper presents the procedures for designing a study that is both methodologically rigorous and potentially impactful. Twelve researchers, practitioners and policy-makers in Scotland were extensively involved in designing a mixed-method study of teacher agency for social justice. The Critical Communicative Methodology was employed to establish egalitarian dialogue between researchers and practitioners. The procedures and the resulting research tools can be used in future studies, including large-scale quantitative analysis. The paper discusses the challenges of ownership, choice of methods, and knowledge transfer, that need to be addressed in these ways of working.     

浅谈学术期刊的非学术因素影响

学术期刊所承载的自然是学术的东西,张扬的应该是学术精神,如果受非学术性的影响太大,承载的功利性因素太多,势必会影响学术质量,妨碍期刊功能的发挥。学术期刊非学术性影响主要是:评职称的影响;学术、科研管理部门对学术成果考核量化管理的影响;研究生管理的影响等。在呼吁营造学术期刊的良好社会氛围的同时,必须提高编辑素质,自觉抵制非学术性因素对学术期刊的影响。     

The competitive environment of academic productivity and the academic research enterprise in the case of Japan

This article deals with the various problems related to the given title from four sections. Section “Social function of the academic research enterprise (ARE)” focuses on three problems: Framework of the research on the ARE; Functions of the graduate school in the ARE; and Centers of learning and Japanese ARE. Section “Structure of ARE” discusses two problems: Structure of social stratification and the structure of ARE in the policy framework for the current ARE. Section “Social condition of ARE” argues over three problems: Social condition of the ARE; The effects of change of national policy on the ARE; and National government policy for the ARE in the postwar period. Section “The impacts of the national government policies on ARE” treats three problems: Trend of the recent national government policy on the ARE; Some unintended effects of the policies; and Discussion of proposals for reform. These discussions lead to concluding remarks.     

A proposed framework and tool for non-economic research impact measurement

ABSTRACT

Research impact features heavily in debates about ‘the measured university’ and is now formally assessed by governments in the UK and Australia. Yet clear guidance on how impact can be measured in non-monetary ways is often lacking because of confused thinking and the context-specific nature of outcomes. To help resolve this, we first propose a general impact model of inputs, processes, outputs and outcomes framework. Then, using a survey approach of research users, we measure outcomes at different levels of abstraction using usefulness as the central construct and impact categories from the European (EU) and Association to Advance Collegiate Schools of Business (AACSB). The survey measures are simple, comparable between different impact cases, cost-effective, externally verifiable, and easily administered by those for whom impact measurement is new and puzzling. They can also be combined to form an impact index to address criticism that there is little standardisation in impact measurement. To improve the standardisation of context-specific measures, we suggest a common methodology for deriving these (SROI). The article discusses limitations of using surveys including administration, self-report data, and impact timescale problems, and suggests ways to reduce these. Implications for researchers, research managers, and assessors interested in measuring impact, such as who should do it and pay for it, are discussed.     

Conceptualizing impact in academic development: finding a way through

This paper explores the notion of impact in the context of academic development programs and considers how it can be described and understood. We argue that impact has a range of meanings and academic development programs such as graduate certificates have a broad group of stakeholders and hence the impact is different for each group depending on how the program aims and objectives are defined and understood. In finding a way through the difficulties of evaluating impact in academic development we point to the importance of clearly conceptualizing the notion of impact, a careful identification of the assumptions underpinning any program and an understanding of who academic development will benefit and how. We suggest that impact in academic development cannot be understood without taking account of the range of possible impacts and the difficulty of attributing simple cause and effect to a complex environment.     

Institutional hierarchies and research impact: new academic currencies,capital and position-taking in UK higher education

Abstract

Globally, performance-based research funding aims to support the most deserving academic institutions and researchers. However, overcoming entrenched assumptions about quality is a persistent challenge for higher education research policies worldwide; traditionally powerful institutions tend to maintain dominance. Research impact as a performance criterion presents an opportunity for position-taking through success according to non-academic criteria. Could impact-oriented research funding challenge institutional hierarchies? The UK university system presents an instructive case study for exploring this question. However, exposing the effects of such performance-based funding on institutional stratification requires focusing on the interface between institutions and disciplines. A Bourdieusian analysis of 53 cases of research-based impact on higher education policy/practice revealed the differential capital that researchers from more and less ‘prestigious’ universities mobilise when generating research impact. By uncovering how impact reinforces disparities in research power between UK institutions, the study contributes to understanding of sectoral reproduction through discipline-level mediation of research policy.     

Evaluating the impact of an academic teacher development program: practical realities of an evidence-based study

This study aimed to assess the impact of an entire academic teacher development programme at a Midwestern masters comprehensive university in the United States over a period of five years by examining changes in teaching and student outcomes of nine randomly selected programme participants. Researchers analysed syllabi, course evaluations, grade distributions, programme activity reports, interviews and vita, and concluded that the centre’s impact was greatest on tenure-track faculty who struggled with their teaching. Far more useful to future improvement, however, were the insights gained from reflection on study methods, the researchers’ emotional responses to their findings, their struggle to make sense of the data and grappling with the definition of ‘impact’. In this paper, the researchers share useful lessons learned and illustrate the difficulties inherent in evaluating academic development programmes, with a view to qualifying recommendations currently prevailing in the literature.     

Evaluating the impact of the graduate certificate in academic practice (GCAP) programme

Graduate teaching assistants (GTAs) and postgraduates in the UK are taking on greater responsibility for teaching, and therefore it has become increasingly necessary to explore the teacher training that supports them in this endeavour. This paper outlines an impact evaluation carried out on a graduate certificate programme primarily aimed at GTAs and support staff at a research-intensive institution. The concept of impact evaluation is explored. This qualitative research establishes impact on the approaches participants took and their understanding of teaching following on from the programme. Interviews with nine participants, analyses of their critical reflective portfolios, and analyses of their teaching observation commentary are reported on. Findings reveal participants felt they had gained particular skills of teaching, were more reflective, and adopted a more student-centred approach to their teaching. It is recommended that training on specific teaching skills be incorporated into the provision available for GTAs and that disciplinary differences accommodated.     

Student-staff partnerships in academic development: the course design studio as a model for sustainable course-wide impact

ABSTRACT

Sector-wide research has shown the benefits of student-staff partnerships in course design whilst highlighting the complexity of this work, and the difficulties in gaining traction to create sustained ways of working. Reflections by academic developers on the roles and partnerships in course design at one Australian university reveal the critical three-way partnership among academic developers, students, and academics. The benefits brought by a Course Design Studio model in providing a liminal space to sustain positive three-way collaborations are outlined. This paper offers a conceptual argument that addresses areas of concern often encountered in traditional course design student-staff partnership relationships.     

The white cube in the black box: assessing artistic research quality in multidisciplinary academic panels

Despite its imperfections, academic peer review has been accepted as a satisfactory process by which assessment panels comprised of different disciplinary representatives arrive at agreement through a system of shared rules and language that respects disciplinary plurality. Artistic researchers, whose output is required to meet both scholarly research criteria and the aesthetic standards expected by their peers in the art world, express concern that university peer review processes disadvantage their disciplines due to differences between artistic research and more science-informed research expectations. As research created with the artistic ‘white cube’ in mind intersects with this ‘black box’ of scholarly peer review, do the shared protocols adopted by peer-review panels extend to research expressed not in scholarly text but in artistic artefact? Are there features inherent in artistic research that impinge upon equitable consideration within these settings? This paper draws upon face-to-face interviews with visual and performing artists in three Australian universities to explore how these researchers experience university peer review. It contributes a distinctive disciplinary perspective to understanding university assessment and evaluation processes for the benefit of both academic staff and postgraduate students.     

Evaluating the impact of an environmental education programme: an empirical study in Mexico

This study draws on information from 11 in‐depth interviews, two focus groups and 72 written questionnaires to evaluate an extra‐curricular environmental education programme on forestry designed for preparatory school students from a small rural community in Mexico. Specifically, the study assessed the impact of the programme on the ecological knowledge of 72 students. Qualitative feedback suggests that students learnt about forestry, acquired greater awareness of the importance of conservation for the local environment and enjoyed the participatory teaching methods used in the programme. Quantitative results show a positive and significant association between the number of times a student participated in the programme and the student’s ecological knowledge. Students who participated in the programme once had a 16.3% higher knowledge on ecological concepts and knew, on average, 1.5 more local forest plants than students who never attended it (p<.001). Findings suggest that the inclusion of participatory environmental education programmes in preparatory schools would improve the acquisition of ecological knowledge. Further research could consider the consistency of the findings by replicating participatory methods presented here and by using an experimental research design.     

基于灰色系统理论的绿色设计影响评估算法研究

生命周期评估(LCA)是绿色产品设计的一项重要内容, 影响评估是LCA的重要阶段. 本文将各种排放物影响因子进行分类后, 应用灰色系统理论, 提出了影响评估灰类划分及其白化权函数, 对各影响类别进行了灰色聚类分析, 然后根据分析结果给出了分类影响指数的计算方法. 根据各影响类别的作用范围, 将影响类别按全球影响、区域影响、本地影响进行了分组, 并给出了分组影响指数以及总体影响指数的计算方法. 最后, 以设计中的3种材料钢、铝、工程塑料为例, 说明了以上方法在一类产品比较选择中的应用.     

Government research evaluations and academic freedom: a UK and Australian comparison

Performance management systems have been an inevitable consequence of the development of government research evaluations (GREs) of university research, and have also inevitably affected the working life of academics. The aim of this paper is to track the development of GREs over the past 25 years, by critically evaluating their adoption in the UK and Australian higher education sector and their contribution to the commodification of academic labour, and to highlight the resultant tensions between GREs and academic freedom. The paper employs a literature-based analysis, relying on publicly available policy documents and academic studies over the period 1985–2010. GREs are a global phenomenon emanating from new public management reforms and while assessments of university research have been welcomed, they have attracted critique based on their design, the manner in which they have been applied, and the unintended consequences of their implementation on academic freedom in particular. Consistent with international research on the impact of GREs, Australian research assessments appear to be undoing the academic freedom that is central to successful research. Further empirical research on the impact of GREs on academics is urgently needed.     

英国RAE大学科研评估制度及其对大学科研拨款的影响

随着英国高等教育规模的不断扩张以及政府财政压力的日益增长,为了加强科研拨款的选择性和透明度,提高政府科研拨款的使用效率,英国自1986年开始推行RAE大学科研评估制度。该制度对促进英国大学的科研水平、学科的调整等,起到了积极的作用。我国可考虑建立权威的半官方性质的中介评估机构,加强对科研拨款制度的研究,寻找评估与拨款结合的生长点。     

Assessment rubrics: towards clearer and more replicable design,research and practice

‘Rubric’ is a term with a variety of meanings. As the use of rubrics has increased both in research and practice, the term has come to represent divergent practices. These range from secret scoring sheets held by teachers to holistic student-developed articulations of quality. Rubrics are evaluated, mandated, embraced and resisted based on often imprecise and inconsistent understandings of the term. This paper provides a synthesis of the diversity of rubrics, and a framework for researchers and practitioners to be clearer about what they mean when they say ‘rubric’. Fourteen design elements or decision points are identified that make one rubric different from another. This framework subsumes previous attempts to categorise rubrics, and should provide more precision to rubric discussions and debate, as well as supporting more replicable research and practice.     

百年清华 继往开来——清华大学学术研究回顾与展望

清华大学的百年发展史,是一部为祖国兴旺的百年学术奋斗史。本文将清华大学的百年历程分为七个发展时期,梳理每一时期的学术研究发展情况,呈现清华学术研究的整体发展脉络。继而历数了不同时期、各科类的学术精英,展现清华百年人才辈出之画卷。最后指出,清华要着眼于未来新的百年,人文日新,科教兴国,为了人类的光明未来创新不止,奋斗不息。     

冲击的可靠性设计

用能量法求冲击应力和冲击变形．并将应力强度干涉模型运用于冲击问题,讨论了冲击的可靠性设计。     

The invisible impact of educational research

Although there are policy calls for educational research to discover ‘what works’ and thereby inform decision making directly, the research literature argues instead for research to have a ‘conceptual’ impact on practice. Empirical studies also suggest that, when teachers use research, their use is conceptual; research influences the content and the process of their thinking, changing attitudes and perceptions and making educational decision making more intelligent. This study investigates the ways in which educational research has achieved impact on practice from the perspective of the researchers. A sample of highly-rated impact case studies in the UK’s research assessment exercise (REF2014) were subject to content analysis, using qualitative coding techniques. Analysis shows that most research is ‘invisible’ to education practitioners because it is embedded in educational policies, technologies, and services. This ‘invisible use’ is unlikely to realise the conceptual benefits claimed for research utilisation. If educational research is to make educational decision making more intelligent at its point of use, it will be necessary to re-think current notions of quality in research impact.     

Engaging the wider academic community in a postgraduate certificate in academic practice: the issue of standards

This paper critically reflects on the challenges associated with academic standards in a postgraduate certificate in academic practice, which involved the wider academic community of the institution. It is underpinned by a socio-cultural constructivist view that suggests standards do not exist independently of assessors, but are co-constructed by participation in communities of practice through the process of making assessment judgements. Following an outline of the programme design, the discussion focuses on the uncertainties around standards arising from the fragility and fragmentation of a nascent community of practice comprised of a multiplicity of personal standards frameworks and disciplinary perspectives.