Research planning and management in non‐traditional research discipline areas

Abstract

In its drive for higher efficiency and effectiveness in higher education, the Australian Commonwealth Government released its policy on higher education, Higher Education: A Policy Statement during the late 1980s which contributed to the introduction of performance indicators to manage and assess the performance of the higher education system. The research component of annual Commonwealth funding to universities, called the Research Quantum, is now distributed by a set of performance indicators: external research income, publications count and higher degree research completions. This paper analysed the impact of these indicators on the research activities of Australian university academics based on Leibenstein's model of X‐efficiency. Although the impact of performance indicators on university academics was found to follow the pressure‐effort relationship, for some respondents, the increase in effort as a result of heightened pressure had involved strategic behaviour. Two conditions that are identified for the minimisation of an institution's X‐inefficiency despite high staff effort are a high ratio of maximising to non‐maximising decisions for both the staff and institution, and for all the institution's essential objectives/activities, and not just those which are measured by the performance indicators. The implications of these findings for the development and application of performance indicators are discussed.     

Why are some universities better? An evaluation in terms of organizational culture and academic performance*

ABSTRACT

The purpose of this study is to investigate the relationships between organizational culture and academic performance among higher education institutions. The sample was composed of 647 faculty members working in the colleges of education, health, engineering, law, economics, and administrative sciences in four central research universities in Turkey. The Organizational Culture Assessment Instrument (OCAI) was used to describe organizational culture, whereas University Ranking by Academic Performance (URAP) data were used to investigate academic performance. A multilevel analysis in which one unit of analysis (i.e., faculty member) was nested within another (i.e., college) was performed. The original contribution made to the literature by this research is its examination of the relationships between organizational culture and tangible performance indicators (e.g., the number of academic publications and citations). Results show that three different culture types were observed in the universities: Innovative team culture, competitive culture, and hierarchic culture. Although significant relationships were found between academic performance and both competitive culture and innovative team culture, the sole significant predictor of academic performance was competitive culture. One particularly interesting finding was that whereas competitive culture was effective on quantitative indicators (i.e., number of publications and number of citations), only innovative team culture affected the indicators related to effectiveness (i.e., article impact level and citation impact level).     

Do performance‐based research funding systems affect research production and impact?

We study the potential impact of introducing performance‐based funding systems (PBFS) on national research systems, using information on the number of publications and their scientific impact (citations or publications in top‐ranked journals) for 31 countries over the period 1996–2016. The analysis is performed both at the aggregate level and looking separately at each of the six main scientific areas identified according to the Organisation for Economic Co‐operation and Development (OECD) classification. On average, PBFS are found to increase the number of publications, though the effect is only temporary and fades after a few years. Looking at the scientific impact, PBFS are found to have a negligible effect on excellence as measured by the share of articles published in the top journals, irrespective of the type of assessment adopted. On the contrary, PBFS have some influence on average research quality, as measured by the number of citations per paper normalised with respect to the field.     

The Impact of Performance Indicators on the Work of University Academics: Evidence from Australian Universities

This paper examines the impact of the application of performance indicators on the teaching and research activities of 152 university academics from four representative Australian universities. Most of the participants reported an increase in pressure to focus on the activities measured by the teaching and research performance indicators. A majority of the participants also indicated that their approach to teaching and research have changed. In research, this essentially meant focusing on external research grants applications and publications, and maximising the quantity produced from these activities. Some staff did use various strategies to increase their scores such as writing shorter papers in order to increase the quantity of publications. Moreover, a sizeable proportion of the participants indicated a shift in their work priorities towards research at the expense of teaching. One possible reason could be the lack of special incentives to increase their emphasis on teaching.     

Efficiency by Performance Indicators? Evidence from Australian Higher Education

In its drive for higher efficiency and effectiveness in higher education, the Australian Commonwealth Government released its policy on higher education, Higher Education: A Policy Statement during the late 1980s which contributed to the introduction of performance indicators to manage and assess the performance of the higher education system. The research component of annual Commonwealth funding to universities, called the Research Quantum, is now distributed by a set of performance indicators: external research income, publications count and higher degree research completions. This paper analysed the impact of these indicators on the research activities of Australian university academics based on Leibenstein's model of X-efficiency. Although the impact of performance indicators on university academics was found to follow the pressure-effort relationship, for some respondents, the increase in effort as a result of heightened pressure had involved strategic behaviour. Two conditions that are identified for the minimisation of an institution's X-inefficiency despite high staff effort are a high ratio of maximising to non-maximising decisions for both the staff and institution, and for all the institution's essential objectives/activities, and not just those which are measured by the performance indicators. The implications of these findings for the development and application of performance indicators are discussed. This revised version was published online in August 2006 with corrections to the Cover Date.     

Predicting institutional research performance from published indicators: A test of a classification of Australian university types

In 1998, the Australian federal government published comprehensive performance indicator data on universities. This article makes use of several indicators to examine correlates of research productivity in Australian universities, and to test the explanatory power of Marginson's (1997) typology of universities. Research productivity is measured using the quantity known as the 'research quantum' (RQ), which is a performance-related component of operating grant funding. Results of the analysis reveal that a large proportion of the variability in research performance can be attributed to two latent constructs. One relates together favourable student-staff ratios, a high proportion of graduating students continuing into further study, and a high proportion of research qualified staff. The other connects graduates' experiences of low quality courses, good student progress rates and low attrition with high graduate employability. These factors also predict membership of the four Marginson types. The results confirm that Australian universities can be separated into four main reputational groups based chiefly on research performance. Some implications for the use of performance indicators in a more market-driven system are discussed.     

Phelan’s bibliometric analysis of the impact of australian educational research

Phelan (2000) has produced a complex bibliometric analysis of the international contribution of Australian educational research based upon publications and citations reported in the journals indexed by the Institute for Scientific Information — the Standard & Poors of the academic world. This paper examines Phelan’s analysis, showing its strengths and weaknesses, as well as examining his proposal for the establishment of an Australian database along the lines of the ISI’s index.     

Assessing the performance of educational research in Australian universities: an alternative perspective

This study uses bibliometric data to assess the performance of educational research in Australian universities. It provides an alternative perspective to the Australian government’s Excellence in Research for Australia (ERA) assessment. ERA results suggest that the performance of educational research is substantially less compared to other disciplines, with only a handful of universities performing ‘above world standard’. I use a range of indicators that capture citation impact and journal quality from Elsevier’s SciVal research benchmarking tool to provide an alternative perspective that is authentic, transparent, comprehensive and multi-dimensional. The findings show that most Australian universities are performing above the world average in educational research. Australian universities perform especially well on the citation indicators, with more than 75% of universities performing above the world average on all three indicators. Performance on ERA is only moderately correlated with research performance in my analysis but is strongly related to one indicator, publication volume. This finding suggests that ERA assessments favour large entities and disadvantage smaller ones. The findings also suggest that the current ERA peer-review process may not accurately or fully reflect the performance of educational research in Australia. I anticipate that this alternative, bibliometric-based assessment will be a useful benchmarking tool for a range of purposes. I conclude with recommendations for future research.     

The Role of Department Head in Australian Universities: changes and challenges

The role of the academic departmental head in Australian universities has been little explored. This article reports a study of the role by comparing key demographic data with a similar American‐based study, and examining the role in terms of personal, professional, and organisational variables, and their relationships with the generic measures of role‐conflict and role‐ambiguity. Results indicate differences in the role classified by gender, type of initial appointment, age and willingness to serve (among other variables), and on such dimensions as job satisfaction, work stress, role‐conflict and ambiguity, and publications record. Findings indicate a job where stress appears manageable, satisfaction is in short supply and maintaining an appropriate publications record is an ongoing pressure.     

Associations between research and teaching in Australian higher education

This article describes results of an empirical investigation of the relationship between research and undergraduate teaching in Australian higher education. Two research indexes (weighted number of publications, and number of research activities) were used. Scores on a Likert-type scale of reported commitment to teaching undergraduate students formed the main criterion of teaching effectiveness. This was supplemented by student ratings in one of the aggregate-level analyses. The results revealed typically no relation or a negative relation between teaching and research at the level of the individual and at the level of the department, across all subject areas. The only exceptions concerned one group of former colleges of education. Further analysis by staff self-rating of academic quality showed that there existed one group of staff, mainly in the universities, who were committed to teaching and highly active researchers. However, the data did not support a causal interpretation of the association. It is concluded that there is no evidence in these results to indicate the existence of a simple functional association between high research output and the effectiveness of undergraduate teaching. Some implications for policy and student course choice are discussed.     

Academics and their work under dawkins: A study of five NSW universities

This paper reports views of academic staff from five Australian institutions of higher education on a number of issues relating to teaching and research and changes affecting their work since the introduction of the unified national system. While most academics are satisfied with their jobs, staff morale is given a low rating. Academics cite a number of limitations experienced in carrying out their research and teaching. The academic work week is just under 50 hours, but there are clear differences between the established and newer universities in the apportioning of time to research and teaching related activities. This pattern is repeated in the publication record, grant success and perceived pressure to upgrade professional qualifications. The data suggest that the newer universities feel under pressure to emulate the structure and organisation of the traditional “elite” universities rather than seeking a legitimate niche for their particular type of operations within the unified national system. This means that for some years they will find considerable difficulty competing with traditional universities using the standard measures of research performance.     

Teaching,research and scholarship in different disciplines

Teaching and research are the primary functions of academics in all academic disciplines in all Australian universities. Scholarship is expected of all academics whether in the university or college sector. Under a new policy Australian higher education institutions have to develop educational profiles that will describe their strengths in teaching and research. The federal government, and indeed, institutions, are developing and using performance indicators to distribute resources. Some of these, e.g. number of publications, number of research grants, and number of Ph.D. graduates are disadvantaging the Humanities. This paper addresses differences in four disciplines, Chemistry, Engineering, English and Law as they are described by other researchers and emerge from a questionnaire study at an Australian university.     

Moving towards professionalism: The strategic management of international education activities at Australian universities and their Faculties of Business

Universities around the world are increasingly focusing on entrepreneurial activities (Slaughter and Lesley 1997; Clark 1998). In Australia, the growth of international entrepreneurial activities has resulted in the creation of a significant export-oriented sector. These activities include the recruitment of international students to Australian campuses, the development of Australian university campuses in offshore locations, and the delivery of Australian degree programs at both onshore and offshore locations in partnership with universities, professional associations and private corporations. In 2000, Australian universities received A$770 million from international student fees (Maslen 2000a, p. 10). Australian universities currently receive on average around eight percent of revenues from these sources, with some depending on international entrepreneurialism for as much as one-third of revenues (DETYA 1998a). Managing these activities in an efficient, effective and sustainable manner has thus become critically important to virtually every institution in the Australian higher education sector. As is the case when businesses become international, universities are faced with the need to manage the complexities, risks and challenges associated with international operations. To date, little empirical work has been undertaken which explores and examines how Australian universities are managing their international entrepreneurial business operations. The aim of this study is thus to respond to this research gap by exploring how Australian universities, particularly in terms of their Faculties of Business, organise and manage international entrepreneurial activities. The research examines management approaches, practices and processes at five Australian universities. A Strategic Advantage Model of Internationalisation is presented representing a theoretical and conceptual synthesis of the findings.     

The determinants of research performance: A study of Australian university economists

In this article, the relationship between individual productivity in research, as measured by an index of publications produced, and their preferences and perceptions about research-related issues is explored. A sample of 134 Australian university economists were classified as low, average and high in respect of their publication performance using cluster analysis. Discriminant analysis was then used to see whether membership of these groups was associated with items representing individuals' preferred research approach, their involvement in a range of research-related activities, the things which they felt constrained their research and their perceptions of the benefits of their position which might be conducive to research.Our results show that there was a relationship between these items and group membership, and therefore productivity. Highly productive researchers made deliberate choices about the type of research they undertook in order to enhance their career advancement; they were heavily involved in a number of areas of research activity; they felt relatively few constraints on their research by comparison with low producers; and they enjoyed the freedom and challenge of their positions. These results imply that research performance is more a function of individual motivation than resource support.The authors would like to acknowledge the helpful comments of two anonymous referees.     

Benchmarking citation measures among the Australian education professoriate

Individual researchers and the organisations for which they work are interested in comparative measures of research performance for a variety of purposes. Such comparisons are facilitated by quantifiable measures that are easily obtained and offer convenience and a sense of objectivity. One popular measure is the journal impact factor, based on citation rates, but it is a measure intended for journals rather than individuals. Moreover, educational research publications are not well represented in the databases most widely used for calculation of citation measures, leading to doubts about the usefulness of such measures in education. Newer measures and data sources offer alternatives that provide wider representation of education research. However, research has shown that citation rates vary according to discipline and that valid comparisons depend upon the availability of discipline-specific benchmarks. This study sought to provide such benchmarks for Australian educational researchers based on analysis of citation measures obtained for the Australian education professoriate.     

The delicate dance: the assessment implications of awarding students vocational qualifications within university degrees

This paper explores the implications for assessment of embedding qualifications from the vocational education and training (VET) sector within university qualifications. As VET qualifications are now all competency based, assessment in the two sectors is quite different, since universities have generally eschewed competency‐based training and assessment. A general discussion of the issues is followed by the results of a small‐scale research study carried out in the VET discipline area itself, with participants drawn from the AVTEC list of VET teacher‐educators. Telephone interviews were conducted with nine Australian university academics involved with such qualifications; and data were extracted from a broader study of students who had undertaken the Certificate IV in Assessment and Workplace Training embedded within degrees and graduate diplomas in VET at one of these universities. It is clear that there are many assessment challenges involved with the practice of embedding, which have not yet been systematically addressed.     

Intervention research productivity from 2005 to 2014: Faculty and university representation in school psychology journals

The purpose of this study was to identify authors and training programs making the most frequent contributions to intervention research published in six school psychology journals (School Psychology Review, School Psychology Quarterly, Journal of School Psychology, Psychology in the Schools, School Psychology International, and Journal of Applied School Psychology) over the 10‐year period from 2005 to 2014. A total of 310 articles were identified as intervention articles; 919 unique authors, representing 289 universities or organizations, contributed to these publications. Top‐ranked authors and universities were identified based on authorship rating and total publications, respectively. This evaluation highlights key contributors to the primary intervention research in school psychology, as well as themes of the publications of top‐ranked individuals and programs.     

Understanding the evolution of the entrepreneurial university. The case of English Higher Education institutions

There has been strong policy interest in universities becoming more entrepreneurial and engaging in knowledge exchange activities as part of an expanding third mission agenda. However, our understanding of the evolution and diversity of such activities is limited. Using longitudinal data from the Higher Education Business Community Interaction (HEBCI) Survey, this study examines the evolving configuration of universities' knowledge exchange activities and stakeholders by analysing distinctive clusters of English universities. We find an increasingly diverse profile of third mission activities across different types of universities: within old, more established universities, Russell Group universities increasingly focus on research‐oriented activities typically in partnership with large firms and non‐commercial organisations; while another group engages in a broad range of knowledge exchange activities with low specialisation over time. Newer, less research intensive, universities increasingly rely on activities such as consultancy and formation of spin‐offs. A decreased engagement with small and medium enterprises and a lower share of knowledge exchange activities at the regional level are observed across the time studied for all universities.     

University Admissions and the Prediction of Degree Performance: an Analysis in the Light of Changes to the English Schools’ Examination System

The United Kingdom's Department for Education has recently changed the nature of the AS‐level examinations normally taken by students aspiring to enter higher education degree courses one year into their post‐compulsory education. In the face of protests from universities and other institutions that this would both harm students’ progression towards the A‐level qualifications, on which entry to English universities is normally based, and make it difficult for universities determining which of their applicants were best placed to benefit from their degree courses, the Department conducted research which it claimed showed that degree outcome could be predicted as well from the results of GCSE examinations taken one year before AS‐levels as from AS‐levels themselves. This paper critiques those analyses and their conclusions showing, through a re‐analysis and extended interpretation of the Department's data, that AS‐levels provide a more reliable predictor of degree performance in 2011 than GCSEs and that many students who performed better at AS‐level than at GCSE gained a place, and performed well, at a university with high entrance standards.     

ePortfolio use by university students in Australia: a review of the Australian ePortfolio Project

In October 2008, the Australian Learning and Teaching Council released the final report for the commissioned project ‘ePortfolio use by university students in Australia: Informing excellence in policy and practice’. The Australian ePortfolio Project represented the first attempt to examine the breadth and depth of ePortfolio practice in the Australian higher education sector. The research activities included: surveys of stakeholder groups in learning and teaching, academic management and human resource management, with respondents representing all Australian universities; a series of focus groups and semi‐structured interviews that sought to explore key issues in greater depth; and surveys designed to capture students’ pre‐course expectations and their post‐course experiences of ePortfolio learning. Further qualitative data was collected through interviews with ‘mature users’ of ePortfolios. Project findings revealed that, while there was a high level of interest in the use of ePortfolios in terms of the potential to help students become reflective learners who were conscious of their personal and professional strengths and weaknesses, the state of play in Australian universities was very fragmented. The project investigation identified four individual, yet interrelated, contexts where strategies may be employed to support and foster effective ePortfolio practice in higher education: government policy; technical standards; academic policy; and learning and teaching. Four scenarios for the future were also presented with the goal of stimulating discussion about opportunities for stakeholder engagement. It is argued that the effective use of ePortfolios requires open dialogue and collaboration between the different stakeholders across this range of contexts.