Outcomes Assessment in Higher Education: Challenges and Future Research in the Context of Voluntary System of Accountability

Higher education institutions are becoming increasingly accountable to stakeholders at many levels as the federal government, accrediting organizations, and the public place a stronger emphasis on student learning outcomes. Evidence of student learning that is comparable across institutions is of urgent demand. Various accountability initiatives such as the Voluntary System of Accountability (VSA) mark a milestone in the evaluation of instructional effectiveness of higher education using standardized outcomes assessment. Although the current accountability initiatives hold great potential in evaluating institutional effectiveness, some design and methodological issues need to be addressed before outcomes assessment can be best utilized. In this paper, I (a) review the need for outcomes assessment in higher education, (b) discuss the VSA initiative, (c) summarize the most prominent challenges facing outcomes assessment for accountability purposes, and (d) discuss future directions of research in advancing accountability assessment in higher education.     

Measuring learning outcomes

The growing interest for measurement of learning outcomes relates to long lines of development in higher education, the request for accountability, intensified through international reforms and movements such as the development and implementation of qualifications frameworks. In this article, we discuss relevant literature on different approaches to measurement and how learning outcomes are measured, what kinds of learning outcomes are measured, and why learning outcomes are measured. Three dimensions are used to structure the literature: Whether the approaches emphasise generic or disciplinary skills and competence, self‐assessment or more objective test based measures (including grades), and how the issue of the contribution from the education program or institution (the value‐added) are discussed. It is pointed out that large scales initiatives that compare institutions and even nations seem to fall short because of the implicit and explicit differences in context, whilst small‐scale approaches suffer from a lack of relevance outside local contexts. In addition, competence (actual level of performance) is often confused with learning (gain and development) in many approaches, laying the ground for false assumptions about institutional process‐quality in higher education.     

Managing Change—Engaging Faculty in Assessment Opportunities

Regional accrediting bodies require evidence that higher education institutions are meeting their stated goals. Institutions have answered this call for accountability by assessing student learning. Managing change in order to implement assessment practices is a challenge, however, particularly when autonomy, academic freedom, and shared governance are involved. Leadership theories offer practical strategies for administrators instituting assessment-related change. Using these theories as a guide, this article provides suggestions for leading assessment initiatives, garnering faculty support, and establishing a culture of assessment. The suggestions are organized around a four-frame model based on the premise that leaders must consider multiple perspectives to be successful.     

Neoliberal Ideologies,Governmentality and the Academy: An examination of accountability through assessment and transparency

Colleges and universities exist within a political arena where external demands for accountability materialize within a market-driven environment. As a result, government agencies pressure colleges and universities to rely on assessment and transparent reporting to become more market-driven assuming that the competition within the market, led by public choice and institutional selection, will drive improvements in learning and will also self-govern the institutions. This article explores how Foucault informs our conception of neoliberal governmentality through political rationality and technologies of self-governance in order to inform our understanding of accountability in higher education and to challenge institutions to develop a counter dialogue which meets institutional obligations to the public and market.     

Accountability, Assessment and Doctoral Education: recommendations for moving forward

This article opens with an overview of the pressures that have led to heightened calls for accountability in higher education, both in the US and Europe. Noting that doctoral programmes have, to date, drawn minimal attention in the accountability debates, the authors note that these programmes cannot stay forever on the sidelines. Drawing on their knowledge of the American context in particular, the authors go on to outline current efforts to assess the quality of doctoral programmes in the US with attention to how these efforts respond to accountability demands. They urge the higher education community to re-shape the accountability discussion, shifting its centre from government to higher education and the institutions training doctoral students who will become faculty. Crucially, they argue that graduate programmes should train doctoral students to undertake and use discipline-appropriate assessment measurement and scholarship as part of their future work as college and university instructors. If doctoral students emerge as faculty with an understanding of how to conduct assessments for the purpose of improving student learning, they will advance not only their fields, but higher education more generally.     

Inclusive discourses in early childhood education?

This article explores the discursive formation of inclusion in early childhood education and after‐school (recreation) centres in a Danish municipality. While inclusion has been a central educational issue in research and practice for well over quarter of a century, with continuing emphasis worldwide on ‘initiatives by governments’, this interest has centred on the school environment and institutions of higher education. Thus, despite increasing recognition of the significance of preschool and after‐school‐care, inclusion in these environments remains peripheral to the main debate.     

The accountability era in US teacher education: looking back,looking forward

Abstract

In keeping with the theme of the 40th anniversary issue of EJTE, this article looks back and forward at US teacher education accountability. It argues that “holding teacher education accountable” has been the major approach to reforming teacher education in the US for the last two decades, assuming that enhanced teacher education quality depends on vigilant public evaluation and monitoring of outcomes related to teacher education institutions, programs, and teacher candidates. This article looks back at the “era of accountability” by examining five policy, political, and professional developments that contributed to its emergence and strong hold on US teacher education. Looking forward to the future of teacher education accountability in the US, the article argues that we need a new approach – democratic accountability in teacher education – which is based on intelligent professional responsibility for students’ learning including democratic knowledge and skills, strong equity, and genuine collaboration with multiple stakeholders.     

Implementing Outcomes Assessment in an Academic Affairs Support Unit

While outcomes assessment has become a focus for institutions of higher education, assessment in academic support units remain sparse. Traditionally, assessment has been comprised of one-time student satisfaction surveys and/or supervisor evaluations tied specifically to job criteria. Although informative, these methods fail to account for or measure specific student learning outcomes completed by students via their interaction with faculty and staff. This article presents the development and implementation of a program of outcomes assessment at a 4-year public university in the northeast. Results indicate that students come to college with varying expectations of the role of academic advisors and range on a continuum of academic and social preparation to meet the demands of higher education. A majority of students are meeting the stated learning outcomes. Feedback from this assessment model is being used to improve the quality of services provided to successive cohorts of students at this institution.     

Towards Responsible Massification: Some Pointers for Supporting Lecturers

Teaching large classes poses many challenges to lecturers where massification is a reality in higher education. There are implications for both teaching and effective learning in this context. The need for accountability to learners in education provision served as motivation for a study of large classes in the largest faculty of one university where enrolment figures had recently increased. Semistructured interviews were conducted with 14 lecturers who teach on average over 500 students in one class. Lecturers reported challenges and experiences related to the higher education environment, resources and support, and teaching and learning. The article provides pointers for supporting lecturers who are teaching large classes based on their identified tensions due to challenges they face. Taking cognisance of lecturers’ experiences could guide institutions towards relevant support for academics in the large-class setting and ultimately contribute to accountability and responsible massification of higher education.     

Enhanced student learning in accounting utilising web-based technology,peer-review feedback and reflective practices: a learning community approach to assessment

Higher education is becoming a major driver of economic competitiveness in an increasingly knowledge-driven global economy. Maintaining the competitive edge has seen an increase in public accountability of higher education institutions through the mechanism of ranking universities based on the quality of their teaching and learning outcomes. As a result, assessment processes are under scrutiny, creating tensions between standardisation and measurability and the development of creative and reflective learners. These tensions are further highlighted in the context of large undergraduate subjects, learner diversity and time-poor academics and students. Research suggests that high level and complex learning is best developed when assessment, combined with effective feedback practices, involves students as partners in these processes. This article reports on a four-phase, cross-institution and cross-discipline project designed to embed peer-review processes as part of the assessment in two large, under-graduate accounting classes. Using a social constructivist view of learning, which emphasises the role of both teacher and learner in the development of complex cognitive understandings, we undertook an iterative process of peer review. Successive phases built upon students’ feedback and achievements and input from language/learning and curriculum experts to improve the teaching and learning outcomes.     

Vocational lifelong learners?

The notion of lifelong learning has become a mantra within educational policies. However these have been strongly critiqued for reflecting an understanding of learning that privileges the economic benefits of participation in formal education. In UK contexts, the importance attached to widening participation in higher education is one manifestation of these policy discourses, which can be interrogated as a form of governmentality. This paper draws upon a recent small‐scale mixed‐method study of different vocational learners’ transition from Level 3 courses to consider how these policy discourses are being mediated by ‘learners’ who were qualified to enter higher education, but decided instead on alternative life courses. The analysis suggests that policy constructions of participation in higher education sit at a disjuncture with respondents’ longer‐term experiences of institutionalised education processes. In other ways, lifelong learning seemed to be willingly embraced in respondents’ different commitments to learning and self‐development, although higher education institutions were not often seen as a source of this learning. The article aims all the same to allow this interpretation of respondents’ voices to speak back and disrupt policy mantras.     

The assessment of learning skills in nursing student selection: A scoping review

Abstract

Higher education student selection has significant societal, institutional and individual impacts. Thousands of applicants apply only for nursing, one of the major higher education disciplines. As the nursing profession is characterised by cognitive requirements, higher education institutions assess the learning skills of nursing applicants. However, there has been no comprehensive analysis of learning skills assessment for nursing student selection. The purpose of this scoping review was to describe the assessment of learning skills in undergraduate nursing student selection. Five databases were systematically searched, and 24 studies published between 2006 and 2016 were included. Learning skills were most commonly assessed using standardised tests in the areas of language and communication, reasoning, mathematics and natural sciences. Overall scores of onsite selection methods were found to best predict future academic performance. The results indicate that higher education institutions may benefit from comprehensive assessment of learning skills in their selection processes. This assessment should focus on a wider range of cognitive aptitudes, including reasoning skills. This review focussed on nursing education, but the results may benefit other higher education disciplines due to the generic nature of learning skills and similar cognitive requirements of higher education studies. The results support the development of more comprehensive and valid methods for assessing learning skills.     

Faculty and Students Conceptions of Assessment in Higher Education

Assessment in higher education serves multiple purposes such as providing information about student learning, student progress, teaching quality, and program and institutional accountability. Yet, little is known about faculty and students’ attitudes regarding different aspects of assessment that have wide-ranging implications for policy and practice in tertiary institutions. To investigate these views, parallel surveys of conceptions of assessment were administered to faculty and undergraduate students across four tertiary institutions including universities, an indigenous tertiary institution, and an institute of technology. A mean and covariance structures approach was used to test for measurement invariance and latent means differences between faculty and students regarding their conceptions of assessment. Results revealed differences in the latent means across the two groups. Faculty were likely to view assessment as a trustworthy process aiding teaching and learning, whereas students viewed assessment as focussed primarily on accountability and perceived assessment as irrelevant or even ignored in the teaching and learning process. These findings highlight the importance of ensuring that assessment policy and practices are fit for purposes, and are being carried out with integrity in ways that are transparent to and understood by both staff and students. While these results show how staff and students view assessment practices, one should keep in mind that while the sample was large and did incorporate different types of tertiary institutions, the inclusion of a broader range of disciplines would make the conclusions more generalizable.     

Value-added assessment in higher education: a comparison of two methods

Evaluation of the effectiveness of higher education has received unprecedented attention from stakeholders at many levels. The Voluntary System of Accountability (VSA) is one of the initiatives to evaluate institutional core educational outcomes (e.g., critical thinking, written communication) using standardized tests. As promising as the VSA method is for calculating a valueadded score and allowing results to be comparable across institutions, it has a few potential methodological limitations. This study proposed an alternative way of value-added computation which takes advantage of multilevel models and considers important institution-level variables. The institutional value-added ranking was significantly different for some of the institutions (i.e., from being ranked at the bottom to performing better than 50% of the institutions) between these two methods, which may lead to substantially different consequences for those institutions, should the results be considered for accountability purposes.     

Assessment of research quality

The assessment of research quality and accountability for research funding are major issues in higher education. This paper describes the British experience of nationwide research quality assessment exercises, and newly introduced measures intended to improve accountability. The consequences are examined, including those for the higher education system as a whole and for individual institutions, using the University of Glasgow as an example. Comprehensive peer review of research quality has demonstrated the usefulness of certain performance indicators. Their relevance for U.S. institutional researchers is discussed, and similarities with the measures used by the National Academy of Sciences in its assessment of researchdoctorate programs are identified. The requirements of the U.S. Government Performance and Results Act (GPRA) are noted.     

Quality Assurance in UK Higher Education: Issues of Trust, Control, Professional Autonomy and Accountability

This article explores the issues of trust, control, professional autonomy and accountability in higher education quality assurance in the UK. The main part of this article is conceptual, but it includes results from semi-structured interviews with academic staff that were conducted at two “new university” business schools. Both institutions are broadly similar in their key characteristics and have experienced a transformation to university status in the early 1990s. The article argues that there has been a change from informal “light-touch” quality control systems based on local practices and a significant amount of trust and professional autonomy in the early 1990s to a highly prescribed process of audit-based quality control today. The article argues that accountability and transparency are important principles that academics should wholeheartedly embrace, but that the audit format adopted in the UK introduces a one-way accountability and provides “rituals of verification” (Power 1997) instead of fostering trust, has high opportunity costs and may well be detrimental to innovative teaching and learning.     

The Effects of Motivational Instruction on College Students' Performance on Low-Stakes Assessment

Assessments of student learning outcomes (SLO) have been widely used in higher education for accreditation, accountability, and strategic planning purposes. Although important to institutions, the assessment results typically bear no consequence for individual students. It is important to clarify the relationship between motivation and test performance and identify practical strategies to boost students' motivation in test taking. This study designed an experiment to examine the effectiveness of a motivational instruction. The instruction increased examinees' self-reported test-taking motivation by .89 standard deviations (SDs) and test scores by .63 SDs. Students receiving the instruction spent an average of 14 more seconds on an item than students in the control group. Score difference between experimental and control groups narrowed to .23 SDs after unmotivated students identified by low response time were removed from the analyses. The findings provide important implications for higher education institutions which administer SLO assessments in a low-stakes setting.     

Breaking Fresh Ground: Regional Co‐operation in Higher Education

One of the most recent European developments in the internationalization of higher education is the stimulation of regional co‐operation among institutions. In the so‐called border countries policy, Flanders, The Netherlands, and the German Länder of North‐Rhine Westphalia, Bremen, and Lower Saxony are collaborating in different ways to establish an “open higher education space”. Within the framework of this border countries policy, various initiatives have been taken in order to support co‐operation among higher education institutions from the countries involved. Amongst other things, research was undertaken in order to gain insight into the differences and similarities of the programmes offered in the five countries. In addition, the Ministries of Education of the participating countries have begun to allocate funds to institutions that have taken co‐operation initiatives that coincide with the policies of the countries concerned. These two aspects of the border countries policy will be discussed in this article.