Understanding research strategies to improve ERA performance in Australian universities: circumventing secrecy to achieve success

ABSTRACT

Many Australian universities have prioritised improving discipline performance on the national research assessment – Excellence for Research in Australia. However, a culture of secrecy pervades Excellence in Research for Australia (ERA). There are no specified criteria for the assignment of ratings on a 5-point scale ranging from ‘well above world standard’ (5) to ‘well below world standard’ (1). No rationale is provided to institutions for their discipline ratings and university staff on the ERA panels sign confidentiality agreements. However, what is available to universities are the research strategies that each university documents to improve its ERA performance in its Mission-based Compact, a government funding agreement. Thus, the purpose of this paper is to investigate the similarities and differences in the research strategies that universities with different performance profiles employ. Following an analysis of the strategies, substantial commonality was identified in strategy use. However, what was different was how universities employed these strategies and the associated contexts.     

Education research Australia: a changing ecology of knowledge and practice

Processes of national research assessment, such as Excellence in Research for Australia (ERA) are a type of audit technology that confronts and steers established institutional identities and traditions. This nexus between policy and practice drives boundary work that diffracts prevailing policy logics, organisational practices, and habits of mind. We use this notion of ‘boundary work’ as an analytical lens for understanding the nature and effects of ERA in the Australian educational research space. This paper explains the methodology that informed the AARE–ACDE research reported in Strategic Capacity Building for Australian Educational Research. It documents the policy logic of ERA and the way it cuts across the established ecology of educational research, revealing social and symbolic work that is remaking the boundaries of educational research. We report on the historical trajectory of Australian educational research, the way ERA codes research outputs, and how educational researchers are repositioning in this shifting research space. We argue that there are specific loci of boundary work where capacity building in Australian educational research can make a difference to future educational knowledge building.     

Benchmarking Academic Research Output in Australia

In higher education the uptake of benchmarking has been slow, despite its recognition elsewhere as an important tool in the pursuit of continuous quality improvement. The need for national benchmarks of research productivity and quality will be felt intensely in Australia in 1995 as the Committee for Quality Assurance in Higher Education focuses on research evaluation. This paper presents benchmarks for publications productivity for Australian academic researchers based on an empirical study of all reported publications output from the Australian university system in 1991. ‘Publications productivity’ in this paper is defined as the number of publications per 10 equivalent academic staff members. Benchmarks for aggregate publications output per staff member, and for each of 13 types of publications, are reported for the highest performing institution, and highest performing organisational units within any institution. Applicability within Australian higher education of citation impact indices as an indicator of research quality is discussed.     

Living in a 2.2 world: from mapping to strategic capacity building for Australian educational research

The results of Australia’s first national research assessment, excellence for research in Australia, provoked considerable discussion about the state of educational research in Australia. Understanding the nature of this field of research became the focus for empirical research and analysis that was intended to inform strategic planning. This paper provides an overview of the articles in this special issue, bringing together research on Australian educational research and insights from international researchers who research the effects of research assessment in educational research in their own countries. These sources inform a commentary on Australian educational research and suggest an agenda for further action on strategic capacity building for Australian educational research.     

Australia’s Engagement with Asia in the National Curriculum

The 2014 national curriculum of Australia is a significant initiative that the Australian government has taken in proposing a curriculum that stresses Australia’s engagement with Asia. In practice, this means that Asian cultures, beliefs, environments and the connections between Australia and Asia are embedded in the learning processes of Australian schools. This article provides an analysis of Australia’s engagement with Asia, which is a cross-curriculum priority in the Australian curriculum. In particular, using the example of China, the article examines the strengths, weaknesses, opportunities and threats (SWOT) analysis involved in Australia’s engagement with Asia from an educational perspective, especially during the global pandemic with the spread of COVID-19 affecting the world so significantly. The intent of this analysis is to map out the educational factors involved and consider what might happen to Australian students and schools as a result of their engagement with the educational cultures, beliefs and practices of China. This is connected to the two countries’ economic engagements and people-to-people ties. By considering some of the current discourses that shape the Australia-China relationships, possibilities are opening up to rethink educational positions.     

Education research in Australia: where is it conducted?

Research assessment exercises aim to identify research quantity and quality and provide insights into research capacity building strategies for the future. Yet with limited knowledge of the ecology of Australian educational research, there is little chance of understanding what research audits might contribute towards a capacity building agenda for such a complex field. This paper draws on secondary data analysis of research outputs submitted by 13 Australian higher education institutions to the Excellence in Research for Australia (ERA) 2010 and 2012 national research assessment exercises, to show where Australian educational research is conducted. Findings offer a profile of education researchers by location in academic organisational units within universities. By analyzing data not accessible through reported ERA data we were also able to present information about appointment profiles, specifically levels and type of appointment within universities, as well as data on institutional and geographic region, and patterns associated with type of outputs (books, book chapters, journal articles, conference papers and other outputs) and field of research. Analysis of the data reveals definitive shifts in the nature of the published outputs and in employment profiles of researchers and their location across university and regional groupings. Research audits are administrative processes that reshape institutional and disciplinary governance structures, policies, individual outputs, work practices and careers, but they are not the sum total of the field per se.     

The white cube in the black box: assessing artistic research quality in multidisciplinary academic panels

Despite its imperfections, academic peer review has been accepted as a satisfactory process by which assessment panels comprised of different disciplinary representatives arrive at agreement through a system of shared rules and language that respects disciplinary plurality. Artistic researchers, whose output is required to meet both scholarly research criteria and the aesthetic standards expected by their peers in the art world, express concern that university peer review processes disadvantage their disciplines due to differences between artistic research and more science-informed research expectations. As research created with the artistic ‘white cube’ in mind intersects with this ‘black box’ of scholarly peer review, do the shared protocols adopted by peer-review panels extend to research expressed not in scholarly text but in artistic artefact? Are there features inherent in artistic research that impinge upon equitable consideration within these settings? This paper draws upon face-to-face interviews with visual and performing artists in three Australian universities to explore how these researchers experience university peer review. It contributes a distinctive disciplinary perspective to understanding university assessment and evaluation processes for the benefit of both academic staff and postgraduate students.     

澳大利亚科研经费管理和促进研究产业化对我国高校的启示——以澳大利亚国立大学为例

科学管理并合理使用科研经费以及科研成果产业化是高校面临的重要课题。澳大利亚促进科研投入的立法和措施完备,科研管理及绩效评估成熟;澳大利亚国立大学是世界一流的研究型大学,科研管理工作严谨有序,科研成果产业化有效开展。我国高校对其进行借鉴,通过完善制度体系、开展从业教育、健全机构设置等,加强对科研经费的管理并促进科研成果产业化。     

Changing boundaries—Shifting identities: Strategic interventions to enhance the future of educational research in Australia

This paper reflects on the geography of Australian educational research in the context of the ERA 2010 and 2012 assessments results. These results reflect significant changes to the nature of educational research over the past decades, where this research is conducted and by whom. We recap the historical changes to the formation of educational institutions and their impact on research outputs to demonstrate that interdisciplinary work is growing in a context where there has been a shift in research outputs away from the traditional area of school education. The ERA results demonstrate a high level of research activity in the intersections between previously distinct discipline areas, particularly in the scholarship of teaching and learning. The future of ERA itself is addressed in order to propose interventions that might make a difference to an ecology that is anchored in traditions and tends towards inertia. Finally, we argue that efforts by universities to build research capacity are likely to continue to be competitive, to focus on the individual rather than on departments and schools, and to be subject to an increasingly pervasive culture of accountability. Against this discourse of accountability, and an accompanying loss in autonomy and creative ‘think-time’, we propose that academics in education actively engage in a community of research. We conclude with interventions designed to build a high-quality, analytical and theoretically intensive research culture to underscore educational research in Australia.     

澳大利亚高校本科教学绩效评估之管窥

澳大利亚联邦教育、科学与培训部及相关教育机构对本科教学质量进行评估的重要方式之一是针对全体毕业生进行绩效评估。本文介绍了澳大利亚全国性的绩效评估工具,着重描述了联邦教育、科学与培训部结果绩效指标的数据来源、处理和应用,并就这种评估所存在的问题进行了阐述。     

A fair slice of the pie? Problematising the dispersal of government funds to Australian universities

A common theme in higher education research is the factors that affect university funding. Studies frequently examine how universities cope with funding cuts and the changes that have stemmed from operating in a neoliberal age, a period that now sees institutions commonly functioning on a cost/benefit basis. This paper offers an original contribution by using the Australian Federal Government Department of Education and Training’s document, Finance 2015: Financial Reports of Higher Education Providers, to problematise the financial assistance individual Australian public universities received from the federal government. The paper acknowledges the factors that can account for some institutions receiving more funding than others but nonetheless highlights the substantial discrepancies concerning government financial contributions that exist between Australia’s highest and lowest funded universities. At a time when a performative culture has increasingly become part of higher education, and an institution’s research performance is calculable and rankable, it is crucial to Australia’s long-term research success that all its universities remain competitive in the race for research and government funding.     

Reframing quality and impact: The place of theory in education research

In March 2004, Stephen Ball and others presented a symposium at the conference of the British Educational Research Association (BERA) on the necessity of theory in educational research. Like Ball, I have observed that theory, not just social theory, is a difficult space and one that divides researchers (those comfortable with theory and those less so), within educational research. It is an aspect of educational research training that rarely receives the attention essential for ‘quality’ educational research. In the context of the contemporary research assessment exercises, it is worth reflecting on the relationship between research informed by social theory and expectations of quality and impact. In this paper I revisit the argument made by Ball and others for the necessity of theory, and discuss its role in framing research questions, informing analysis, and promoting reflexivity on the significance and relevance of research. I illustrate this process by discussing the ways theory can assist in the generation of research agendas and questions. I conclude the paper with an example of how a team of educational researchers from Australia, UK and New Zealand have made use of social theory to inform an Australian Research Council (ARC) funded project investigating the recontextualisation of health knowledge in schools.     

Who is conducting educational research in Australia and how can their work be supported?

Educational research has long been the subject of lively and agitated debate, not least because of its diversity. Ranging in scope from academic development and broad-scale policy research through to student engagement and discipline-specific research, it includes methods of traditional academic inquiry and investigations and also less traditional modes of research. However, the topography of Australian educational research and the characteristics of the people who undertake this complex body of work are currently unclear. This paper explores some of the complexities of the Australian research community, drawing on the findings of a national online survey of academics who identified as researching in the field of education from within and outside education schools and faculties. The survey attracted 504 responses from 38 of Australia’s 39 universities, and just over two-thirds of respondents were located in a school or faculty of education. We draw on the results to answer the questions of who is undertaking educational research and who how they might be supported. We utilise a conceptual model that ‘segments’ the educational research workforce represented by the survey respondents, and we conclude by indicating strategies that might be utilised to address research barriers indicated by educational researchers.     

Benchmarking citation measures among the Australian education professoriate

Individual researchers and the organisations for which they work are interested in comparative measures of research performance for a variety of purposes. Such comparisons are facilitated by quantifiable measures that are easily obtained and offer convenience and a sense of objectivity. One popular measure is the journal impact factor, based on citation rates, but it is a measure intended for journals rather than individuals. Moreover, educational research publications are not well represented in the databases most widely used for calculation of citation measures, leading to doubts about the usefulness of such measures in education. Newer measures and data sources offer alternatives that provide wider representation of education research. However, research has shown that citation rates vary according to discipline and that valid comparisons depend upon the availability of discipline-specific benchmarks. This study sought to provide such benchmarks for Australian educational researchers based on analysis of citation measures obtained for the Australian education professoriate.     

The role of achievement,gender, SES,location and policy in explaining the Indigenous gap in high-school completion

Internationally, there is a gap in high-school completion rates for Indigenous and non-Indigenous students. In Australia, gap estimates are commonly based on lag indicators, precluding examination of underlying mechanisms. Using two longitudinal representative samples of Australian youth, we explored differences in high-school completion between Australian Indigenous and non-Indigenous rates, and whether the gap varies for students of similar academic ability. Using an intersectional approach, we show the Indigenous gap is significant, is mostly a function of differences in academic achievement, but varies by socioeconomic status (SES) and location. Specifically, high SES and living in urban settings are protective factors for non-Indigenous students, but not for Indigenous students. Conversely, rural and poor non-Indigenous students appeared to have dropout rates as large or even larger than similarly poor and rural Indigenous youth. Overall, the results suggest the need for a more nuanced perspective on ‘Indigenous gaps’ in educational attainment.     

The faculty perspective on holistic and systems thinking in American and Australian mechanical engineering programmes

This research effort examined current mechanical engineering educational programmes in America and Australia to determine the degree of holistic, systems thinking of each programme. Faculty from ten American universities and ten Australian universities participated in online surveys and interviews. Resulting data analysis and interpretation suggest that holistic, systems thinking is present in both American and Australian engineering educational programmes, although it is more prevalent in Australian programmes. Specific examples of educational opportunities (courses, projects, extracurricular activities, research experiences) that integrate complex systems study are described in the full paper. This study is currently limited a small sample size within two countries, but it would be very useful to expand the study to a larger population and to include European sites to gain a fuller picture of the state of holistic, systems thinking in higher education.     

A comparison of three major instruments used for the assessment of university student experience: Toward a comprehensive and distributed approach

Higher education's rapid expansion is paired with growing social expectations of its benefits and concern on its teaching quality. In response to these, institutional/national surveys based on an array of theories are widely used in universities for quality assurance, enhancement, and benchmarking. This paper reviews three major types of instruments used for such purposes, including two distinct schools of theory that have guided the development of such assessment in the United States, Australia and the United Kingdom and then spread to the other parts of the world. The theories shaping the development of the two instruments, the dimensions assessed, and the challenges and criticisms involved when using such instruments for quality assurance are each discussed. This review concludes with a call for comparisons of different lines of research in this area, discussions on student learning experience that include more diverse characterizations of student experience across different educational contexts, development of tools to enable distributed leadership among teachers, and encouragement of students as partners for quality enhancement in higher education.     

‘Measuring Up’? Assessment and students with disabilities in the modern university

In this article, I ask how university students with disabilities negotiate with staff arrangements for alternative assessment practices. I draw on three case studies using a personal pronoun perspective to challenge the conventional view that educational policy and teaching practice are forms of rational action. I demonstrate how the lives of students and staff are typically characterised by unexpected events, disorder, emotion and prejudice. The analytic perspective offered here establishes how meanings, intentions and different viewpoints and alliances emerge as social actors work to create specific faculty and institution cultures. The case studies also reveal what does and what does not work – some of the obstacles – and what needs to be done if we are serious about equity and inclusive education. They include practical assistance in recognising the specific requirements of students with disabilities and how to design alternative assessment for students with specific ‘conditions’. I argue that professional development and specific techniques in curriculum design are needed. Some staff also require help in recognising their policy and legal obligations. A cultural change which identifies and challenges prejudice is a larger task if universities are to become places in which equal opportunity principles and inclusive education are present and actively practised.     

The impact of size and specialisation on universities’ department performance: A DEA analysis applied to Austrian universities

This paper explores the performance efficiency of natural and technical science departments at Austrian universities using Data Envelopment Analysis (DEA). We present DEA as an alternative tool for benchmarking and ranking the assignment of decision-making units (organisations and organisational units). The method applies a multiple input and output variables approach, which is a clear advantage to other approaches using simple performance ratios. To deliver reasonable results, suitable input and output variables have been determined in a previous step using correlation analyses and OLS regression. The results validate the methods applied, and reveal performance differences and scale effects. The use of multiple output variables enables the revealing of detailed improvement or reduction amounts of each input and output of the evaluated units and furthermore for identifying the specialisation of teaching, research, and industrial cooperation. We find significant evidence that the size of a department influences its overall and specialisation performance; both small and large departments perform above average, which proves that simple linear scale effects do not exist.