Assessing the unassessable: making learning visible in undergraduates’ experiences of scientific research

We suggest that academics involved in the provision of research experiences to undergraduate science students may benefit by reconceptualising these experiences as work-based learning. In particular, drawing on the widespread use of reflective practice in work-based learning allows for a more effective focus on process-related learning. We describe an empirical study involving the implementation of reflective blogs in which science students responded to prompt questions designed to focus their attention on how their project was progressing, the process of research and their own learning. The blogs reveal examples of students engaging in independent thinking, creative thinking and developing a sense of themselves as scientists, all qualities valued by supervisors. They also show how these combine as ‘wicked competences’ that allow students to address the complex problems and uncertainties of research. We identify characteristic patterns of reflection that indicate the exercise of such competences and suggest that these could be used as explicit evidence by academics making judgements for assessment purposes.     

Employability and higher education: the follies of the ‘Productivity Challenge’ in the Teaching Excellence Framework

This article considers questions of ‘employability’, a notion foregrounded in the Green and White Papers on the Teaching Excellence Framework (TEF). The paper first questions government imperatives concerning employability and suggests a series of mismatches that are evident in the rhetorics in this area. This summary opens up elements of what I am calling the first ‘folly’ in the field. The second section of the paper considers recent research with individual academics engaged in employability activity. This research suggests another series of mismatches in the aims and outcomes of ‘employability initiatives’ and opens up a further series of ‘follies’ in the day-to-day practices of academics and students’ responses to them. The third section of the paper turns to academics’ reports of student behaviour in relation to the outcomes of their degree. This section develops an argument that relates to the final ‘folly’ associated with the current focus on employability. I argue that students’ focus on outcomes (which at face value suggests they have internalized the importance of employment) is contributing to the production of graduates who do not have the dispositions that employers – when interviewed – say that they want. The highly performative culture of higher education, encouraged by the same metrics that will be extended through the TEF, is implicated then in not preparing students for the workplace.     

Social moderation,assessment and assuring standards for accounting graduates

Evidencing student achievement of standards is a growing imperative worldwide. Key stakeholders (including current and prospective students, government, regulators and employers) want confidence that threshold learning standards in an accounting degree have been assured. Australia’s new higher education regulatory environment requires that student achievements are benchmarked against intended programme learning outcomes, guided by published disciplinary standards and a national qualifications framework, and against other higher education providers. Here, we report on a process involving academics from 10 universities, aided by professional practitioners, to establish and equip assessors to reliably assure threshold learning standards in accounting that are nationally comparable. Importantly, we are learning more about how standards are interpreted. Based on the premise that meaning is constructed from tacit experiences, social interactions and intentional reflection on explicit information, we report outcomes of three multi-part calibration interventions, situated around judgements of the quality of the written communication skills exhibited in student work and their related assessment tasks. Qualitative data from 30 participants in the calibration process suggest that they perceive that the process both assists them both in developing a shared understanding of the accounting threshold learning standards and in the redesign of assessment tasks to more validly assess the threshold learning standards.     

This is the way to teach: insights from academics and students about assessment that supports learning

In this study, assessment and learning is reconceptualised as an integrated and dialogic process. Positioned in a sociocultural framework, the experiences of academics and students at an Australian university were examined to understand how they think about and participate in formative assessment to support learning. Semi-structured interviews and document analysis were used to investigate the lived experience, individual meanings and context of participants. The sociocultural issues that participants emphasised were highlighted through a power, risk and reconceptualisation analytical framework. Findings showed that academics and students give high value to dialogue as a device of trust and power for building learner capacity through assessment, and that academics’ familiarity with students’ individual learning reinforces student trust in the integrity and reliability of assessment. This study contributes to understandings of learning and assessment by offering their reconceptualisation as an integrated and dialogic whole when considered from sociocultural perspectives of the lived experience and context. Importantly, the study proposes that the design of intended learning experiences in assessment can also facilitate development of specific dispositions for thinking and being in students as learners and future citizens.     

Qualitatively different ways of experiencing student self‐assessment

In the literature, student self‐assessment as a practice and as a goal in higher education is generally emphasized for academics. This paper reports a study that investigates what academics emphasize in their experiences of student self‐assessment. The investigation focused on the different ways academics described their understanding and practise of self‐assessment. A phenomenographic approach was used to research and identify a set of progressive variations of academics’ ways of experiencing student self‐assessment. Altogether, 16 academics from a variety of disciplines and programs of study in three Australian universities participated in the investigation. The consequent research findings describe five qualitatively different conceptions that depict how academics understand and use student self‐assessment. These findings are subsequently discussed in terms of the potential for academics to understand and use student self‐assessment to enhance students’ self‐assessment ability, to further students’ lifelong learning and to empower, rather than discipline, students.     

Literacy,achievement and success in a Māori tourism certificate programme: reading the world in order to read the word

ABSTRACT

This article critiques international assessment of adult literacy using research findings from students completing a Māori tourism certificate who achieved significant gains in assessment. It is argued that the focus of literacy assessments potentially forces educators to narrow their teaching and learning approaches, manoeuvring them into teaching toward singular or convergent literacy. This leads to utilising teaching and learning strategies drawn from the cultural and social capital of the dominant culture, which is problematic for students without abundance of such capital.

Blending Kaupapa Māori research theory with appreciative inquiry, research revealed that students made significant gains in assessment scores because their educators acknowledged and utilised ways in which they made sense and meaning of their world. Educators drew upon the social and cultural capital of students and engaged them as partners in culturally based teaching and learning processes. Relating to students’ multiple literacies, and the contexts with which they bring meaning to their world in this way, is referred to as drawing from divergent literacies.

This study proposes a need to develop and research alternative ways for improving convergent literacy outcomes. It is suggested that multiple interventions using multi-modal and 360° approaches need to occur simultaneously for best outcomes.     

Constructing academic identity in the changing Russian higher education context: preliminary perspectives

ABSTRACT

Currently, Russian academics are facing significant demands because of a new, urgent requirement to pursue their research in accordance with international standards. Until recently, these academics were used to working within a familiar community and communicating their research via the Russian language, but now, they are expected to move beyond that and function on a global level, which can challenge their academic identities. A qualitative study was carried out in a research-intensive Russian university. The aim was to explore academics’ experiences and perceptions of the dynamic changes in their academic work with a particular focus on the limitations and pressures they meet in their career path towards global research. The data were collected by means of reflective journals during a course on English for academic purposes. Thematic analysis was employed for data analysis. The results showed that global trends turned out to be unachievable for Russian academics due to the lack of specific knowledge, attributes and skills required to do research globally. This might be the first study from Russia contributing to the literature on academic identity construction. The article provides insights into the experiences and perceptions of Russian academics and offers a research agenda for further investigation to bridge identity shifts in academia. In terms of practical purposes, the results will be used as a source of preliminary data within a broader research project aimed at supporting Russian academics in their research in an international English-speaking academic context.     

Academics’ virtual identities

The focus of this article is on the experiences of staff members involved in a student support programme. The experiential, social, and student-centred approaches incorporated in this programme provided not only students, but also academics with pathways to lifelong learning. Functioning in a community of practice (CoP) (with students and also with like-minded colleagues) created an enabling environment for the development of effective teaching and learning approaches, as well as practical skills. Additional positive results relate to the fact that academics are aware of the importance of and more equipped with the competence to scaffold students towards becoming more effective in the learning process.     

Exploring learning goals and assessment approaches for Indigenous health education: a qualitative study in Australia and New Zealand

In higher education, assessment is key to student learning. Assessments which promote critical thinking necessary for sustained learning beyond university are highly valued. However, the design of assessment tasks to achieve these types of thinking skills and dispositions to act in professional practice has received little attention. This research examines how academics design assessment to achieve these learning goals in Indigenous health education. Indigenous health education is an important area of learning for health practitioners to help address worldwide patterns of health inequities that exist for Indigenous people. We used a constructivist qualitative methodology to (i) explore learning goals and assessment strategies used in Indigenous health tertiary education and (ii) examine how they relate to higher education assessment ideals. Forty-one academics (from nine health disciplines) involved in teaching Indigenous health content participated in a semi-structured interview. Thematic analysis revealed learning goals to transform students’ perspectives and capacities to think critically and creatively about their role in Indigenous health. In contrast, assessment tasks encouraged more narrowly bounded thinking to analyse information about historical and socio-cultural factors contributing to Indigenous health. To transform students to be critical health practitioners capable of working and collaborating with Indigenous people to advance their health and well-being, the findings suggest that assessment may need to be nested across many aspects of the curriculum using a programmatic approach, and with a focus on learning to think and act for future practice. These findings accord with more recent calls for transformation of learning and assessment in health education.     

e is for exploration: Assessing hard-to-measure learning outcomes

The focus of this paper is on the use of e‐technologies to enable higher education to better assess aspects of learning that have proved difficult to assess using more conventional means. Higher education describes the knowledge and abilities it intends its graduates to have acquired before graduation, and it has a wide range of approaches to assess these. Higher education also seeks affective outcomes in the form of values, attitudes, behaviours and related attributes or dispositions, and these have consistently proved more difficult to assess by examination or assignment. After graduation, however, graduates are often assessed within the professions via portfolios, interview and peer or expert review. Assessment may focus on teamwork and networking skills, productivity, creativity and values fit to the profession. How can e‐technologies help with these forms of assessment? This paper reviews some of the e‐based approaches and explorations that have supported or could support assessment of affective attributes. At each stage, the paper seeks to establish the common elements of assessment in the different regimes and how e‐assessment contributes, or may contribute in the future. The paper concludes that many of the strengths of e‐assessment lend themselves to an evaluation paradigm rather than to conventional assessment for intended learning outcomes.     

A quantitative and qualitative study of changes in the use of learning outcomes and distractions by students and tutors during a biology poster assessment

Learning outcomes are statements of intended learning within a module. In practice, students may consider various options when undertaking assessments. They may feel they can meet the stated learning outcomes by demonstrating other aspects of study which are termed distractions. Thirty-three undergraduate students' constructed scientific posters and completed questionnaires prior to and on the day of poster submission. Additionally, ten of these students were interviewed. Analysis of questionnaire data using principle components analysis and interview data by clustering units of relevant meaning revealed that students did not differentiate between learning outcomes and distractions, both of which steered their learning during the assessment. These results are discussed within the context of assessment tasks and implications for practice.     

Unpacking Participatory Research in Education

The authors of this paper are members of Ottawa Participatory Researchers in Education (OPRE) a group of teachers, university academics, and students attempting to develop research initiatives using democratic and collaborative research approaches. Building on the experiences of the group, the paper examines and problematizes three elements typically ascribed to non-traditional research approaches: a social action focus, a transformative objective, and a participatory process. Particular emphasis is placed on the questions of who participates, in what ways, and under what conditions. As a result of OPRE's experience and self reflection, seven traps of "pseudo-democracy" are identified and illustrated as potential problems in research purporting to embody equal collaboration between university based academics and community members, such as teachers and students. It is argued that in order to further a democratic research process, equality of opportunity to participate cannot be assumed at any stage of the research process. Attention is paid to the following stages: setting the research agenda, the research epistemology, chronology of the project, representativeness of project participants, funding processes, knowledge cultures, and dissemination choices. Conclusions call for raising awareness of the possibility of teachers and students collaborating to identify research issues of mutual concern and developing a controlling role in research where academics are included as consultants and equals, but not as the possessors of superior and controlling research expertise.     

A conceptual approach for blended leadership for tertiary education institutions

Over the last 20 years, the tertiary education sector has adopted new administrative management approaches, with the aim of improving accountable and strategic focus. Over the same period, the question of how to build leadership capacity to improve learning and teaching and research outcomes has led to discussion on what constitutes academic leadership. While both these advances are needed, what is missing is integration between the two such that academics and professional staff are engaged collaboratively to achieve learning and teaching and research outcomes that are accountable within a strategic focus. This article builds on research into the use of distributed leadership to build leadership for learning and teaching that resulted in the design of a conceptual framework for distributed leadership. Based on this conceptual framework, it proposes a conceptual blended leadership approach to engage academics and professional staff working in collaboration.     

Aboriginal and Torres Strait Islander university students' conceptions of formal learning and experiences of informal learning

This paper describes an investigationof conceptions of learning held by 22 Aboriginal andTorres Strait Islander students from threeuniversities in Queensland, Australia. Other areasinvestigated were students' experiences of informallearning, their reasons for studying and thestrategies they used to learn. Research intoconceptions of learning is gaining impetus and currentbeliefs include the premise that approaches tolearning adopted by university students, and hencelearning outcomes, are closely related to theirconceptions of learning. There is substantial researchfocused on Aboriginal learning styles in earlychildhood and primary school which indicates thatAboriginal children prefer to learn in a practical wayas well as through observation and imitation and trialand error. Very little research has focusedspecifically on Aboriginal university students'conceptions of learning. Results of this study foundthat these students view and approach formaluniversity learning in much the same way as otheruniversity students and most hold quantitativeconceptions of learning. The most interesting resultwas the difference between students' conceptions offormal learning and their experiences of informallearning. Many students' experiences of informallearning were grounded in practical activities orexhibited a cultural focus, however, most formallearning is not dependent upon practical or culturalknowledge. It is proposed that formal learning forIndigenous students recognise and include anIndigenous perspective such as integrating, whereappropriate, practical strategies for learning. Wealso suggest that Indigenous students be helped todevelop conceptions that will enable them to learnformal, theoretical material successfully.     

The use of modern pedagogical techniques when introducing information technology students to entrepreneurship

This paper describes the design of teaching and learning activities (TLAs) in an entrepreneurship subject offered to Information Technology (IT) students. We describe the challenges that we have encountered. Within one teaching semester, the students are expected to achieve a high level of applied knowledge in an area where they have little theoretical knowledge. We introduce TLAs that are not common in an IT curriculum. These TLAs use experience as the basis of knowledge, collaboration as the focus of learning and a reflective approach to structuring knowledge. We identify our rationale for using these new TLAs to achieve the intended learning outcomes for the subject based on the existing literature. To evaluate the effectiveness of our design, we conducted an annual survey. The results of the survey support our design. Student feedback surveys and end of course assessment results indicate that the new TLAs have enhanced the students’ learning.     

Student perceptions of assessment and student self-efficacy in competence-based education

The purpose of this study was to provide insight into the interplay between student perceptions of competence-based assessment and student self-efficacy, and how this influences student learning outcomes. Results reveal that student perceptions of the form authenticity aspect and the quality feedback aspect of assessment do predict student self-efficacy, confirming the role of mastery experiences and social persuasions in enhancing student self-efficacy as stated by social cognitive theory. Findings do not confirm mastery experiences as being a stronger source of self-efficacy information than social persuasions. Study results confirm the predictive role of students’ self-efficacy on their competence outcomes. Mediation analysis results indicate that student’s perceptions of assessment have an indirect effect on student’s competence evaluation outcomes through student’s self-efficacy. Study findings highlight which assessment characteristics, positively influencing students’ learning, contribute to the effectiveness of competence-based education. Limitations of the study and directions for future research are indicated.     

When academics integrate research skill development in the curriculum

This study considered outcomes when 27 academics explicitly developed and assessed student research skills in 28 regular (non-research methods) semester-length courses. These courses ranged from small (n?=?17) to medium-large (n = 222) and included those from first year to masters in business, engineering, health science, humanities and science, across five universities in three Australian cities. The two-year study used three data sets to determine the outcomes of development and assessment initiatives: student pre- (n?=?779) and post-questionnaires (n?=?601), interviews with students (n?=?46) one year after completing a course that developed research skills and interviews with academics (n?=?17) involved in developing and assessing student research skills. These multiple sources provided evidence that students developed a variety of discipline-specific research skills and that these skills were useful for subsequent studies and especially for employment. Academics indicated that the process of making explicit the development of student research skills led to enhancement of their teaching, helping the academics to clarify major course purposes as well as enabling them to provide more substantial feedback to students than in the past. Academics also indicated that this teaching process changed their understanding of disciplinary research and, for some, even suggested new directions in their research.     

Qualitative variation in constructive alignment in curriculum design

Constructive alignment has emerged as a powerful curriculum design idea, but little is known of the extent to which the effectiveness of this idea is a function of qualitative variation. This article introduces a model of qualitative variation in constructive alignment, and uses the results from known alignment studies to test the model. The research reviewed reveals that university teachers have at least two qualitatively different experiences of the core elements of constructive alignment. The teachers who describe their approaches to teaching as involving conceptual change/development intentions with student-focused strategies were found to be likely to see the intended learning outcomes for students in more holistic terms, and assessment as an integral part of that teaching approach. When teachers had more of an intention to transfer information using teacher-focused strategies, they saw the object of study more in terms of parts (concepts, definitions, formulae) with assessment focused on those parts. The implications of this qualitative variation in constructive alignment for curriculum design and for the quality of the outcomes of student learning is discussed.     

Academics’ perceptions of the challenges and barriers to implementing research-based experiences for undergraduates

How can universities ensure that strategic aims to integrate research and teaching through engaging students in research-based experiences be effectively realised within institutions? This paper reports on the findings of a qualitative study exploring academics’ perceptions of the challenges and barriers to implementing undergraduate research. Academics were asked about perceived constraints and enablements, how they defined undergraduate research, the forms of undergraduate research used, and they were encouraged to provide examples. Perceived constraints included particular institutional policies and structures, academics’ mindsets and lack of skills and questions of time and money. It was found that different definitions of undergraduate research lead to different practices and varying opportunities for further development. This paper presents different forms of engagement in undergraduate research allied to these different definitions and it draws on interviewees’ ideas about what has been enabling in their context to suggest possible strategies for institutions to move forward.