Transforming evaluation thinking and behaviour: programmes develop,teachers learn and student learning outcomes improve

This article examines evaluation intervention influence on programme performance through surveys, qualitative evaluation, questionnaires, interviews and test scores. The study uses quantitative (SPANOVA and MANOVA) and qualitative (the constant comparative method) data analysis, together with examining practical significance through effect size and power analysis. Major findings show positive contributions of evaluation intervention to quality management, teacher pedagogical skills and student learning outcomes (SLOs). Quantitative and qualitative findings, however, diverge regarding evaluation impact on programme research. A positive transfer of evaluation ownership discourse from decision-maker to implementer stakeholders has been observed. The study recommends evaluation intervention for programme, institutional and individual development alongside quality improvement.     

Creating high challenge/high support academic environments through constructive alignment: student outcomes

Higher education needs to provide challenging yet supportive learning environments catering for students with diverse academic needs. There is also an emphasis on using student-driven outcome measures to determine teaching effectiveness. How can these measures be used to reflect upon and evaluate teaching initiatives? Using an undergraduate occupational therapy programme as the site for exploration, this article reports on an application of constructive alignment principles and describes how available empirical data were used to explore student outcomes. A comparison was made between student evaluations and academic grades prior to, and after the implementation of the initiative. Results provide evidence of improvement in student satisfaction and academic grades as a result of implementing constructive alignment. Whilst it is acknowledged that changes in academic grades and student evaluations can be attributed to a number of factors, findings of this study support a view that constructive alignment facilitates students' learning and experiences.     

Strengthening internal quality assurance processes: facilitating student evaluation committees to contribute

Student evaluation committees play a crucial role in internal quality assurance processes as representatives of the student body. However, the students on these committees sometimes experience difficulty in providing constructive and structured feedback to faculty in an environment characterised by a strong power differential between student and staff. This study aimed to evaluate a leadership and quality assurance training implemented for students involved in internal quality assurance. Furthermore, we explored how students give shape to their internal quality assurance role. Students from three health sciences programmes participated in a mixed methods study to evaluate the training and reflect on their internal quality assurance role. Overall, the students were very enthusiastic about the training. Qualitative data analysis indicated that in their internal quality assurance role, students: (1) harnessed the power of the entire student population; (2) tried to create focus and take charge; (3) searched for a common ground with staff, and (4) they explained how they dealt with the power differential. Providing training for students with internal quality assurance roles is a valuable endeavour. Future research needs to investigate further ways to help students to contribute to internal quality assurance processes in higher education.     

Graduate learning outcomes in science: variation in perceptions of single- and dual-degree students

The development of transferrable skillsets, articulated in statements of graduate learning outcomes, is emphasised in undergraduate science degree programmes. Science students enrolled in dual (double) degrees comprise a significant minority of Australian science undergraduates. Research comparing perceptions of single and dual degree students on their science learning outcomes has rarely been explored. The Science Students Skills Inventory was used to compare the perceptions of single (n = 640) and dual (n = 266) degree undergraduate science students. The instrument explored science graduate learning outcomes across six indicators: importance; the extent to which outcomes were included; the extent to which they were assessed; improvement; confidence; and likely future use. Analysis of findings, employing the planned–enacted–experienced curricula framework, offers insight into potential avenues towards coherence of the experienced curriculum by arguing the need for shared perceptions of graduate learning outcomes for single and dual degree science students. The key contribution of this study is a shift towards progressive curriculum development that draws on both single and dual degree student perspectives to achieve graduate learning outcomes. Recommendations include: whole-of-programmes curricular pathways premised on progressive development of learning outcomes that are inclusive of dual degree students, explicit interdisciplinary learning opportunities, and adoption of dual/single status as a demographic variable reported in future research.     

Quality of learning outcomes in an online video-based learning community: potential and challenges for student teachers

This study investigates the learning outcomes of 25 student teachers in an online video-based learning community (VBLC). Data were drawn from the student teachers' written comments and feedback recorded in the VBLC and the post-course interviews. Based on Biggs and Collis's Structure of Observed Learning Outcomes (SOLO) taxonomy, the majority of comments and feedback were classified as uni-structural, but more sophisticated responses could also be found. The interviews revealed that the student teachers benefited from the opportunities of peer interaction and self-reflection. The study thus makes suggestions for further improvement of the operation of the VBLC.     

Strengthening internal quality assurance as a lever for enhancing student learning experiences and academic success: lessons from Namibia

Higher education plays an important role in the social, economic and political development of any given nation. In the Namibian context, quality assurance systems have been put in place at both national and institutional levels through the establishment of national quality assurance agencies and units responsible for quality in institutions. A critical literature analysis of quality assurance systems in higher education shows that although they claim to strike a balance between improvement and accountability, external quality assurance mechanisms tend to have an accountability and compliance focus and to promote managerialisation and bureaucratisation of academia. This article argues that internal quality assurance systems, built on a strong quality culture, tend to be more effective in enhancing the quality of student learning experience and academic success than externally driven quality assurance. Thus, to promote genuine quality, more energy must be invested in external quality assurance capacitating and strengthening internal quality assurance systems.     

Block-based versus text-based programming environments on novice student learning outcomes: a meta-analysis study

ABSTRACT

Background and Context: The use of block-based programming environments is purported to be a good way to gently introduce novice computer programmers to computer programming. A small, but growing body of research examines the differences between block-based and text-based programming environments.

Objective: Thus, the purpose of this study was to examine the overall effect of block-based versus text-based programming environments on both cognitive and affective student learning outcomes.

Method: Five academic databases were searched to identify literature meeting our inclusion criteria and resulted in 13 publications with 52 effect size comparisons on both cognitive and affective outcomes.

Findings: We found small effect size (g = 0.245; p = .137; with a 95% confidence interval of ?0.078 to 0.567) in favor of block-based programming environments on cognitive outcomes, and a trivial effect size (g = 0.195, p = .429; with a 95% confidence interval of ?0.289 to 0.678) on affective outcomes. Both effect size calculations were statistically insignificant using random effects models. The effect sizes were examined for moderating effects by education level, learning environment, and study duration. Some evidence of publication bias was detected in these data.

Implications: More research is needed to examine the utility and efficacy of block-based programming environments for novice programmers. Future studies should account for hybrid programming environments using novel research methods.     

Evoked prior experiences in first-year university student learning

In this article, we focus on three aspects of students' prior experiences of learning: evoked conceptions of learning, evoked motivation and evoked self-efficacy. We show how, for a first-year undergraduate population, these three aspects of evoked prior experience relate to students' approaches to learning and their perceptions of the learning environment as well as to their previous schooling, their gender and the broad discipline area in which they are studying. In doing so, we confirm that evoked prior experiences are distinct and measurable and can be used to better understand the ways in which students experience learning in higher education.     

市场竞争中的高校评估及其范式的更新

在市场力量的推动作用下,当今的高校评估不再局限于学校内部,而是依据高校自身设定的使命完成评估的循环过程,以期同步实现问责制。作者提出了一个新的三级评估范式,以对公众需求的信息做出回应。新评估范式的三个等级需要展示学生学习的成果、显示高等教育投资回报率和确保高校的社会效益。在这个三级评估范式中,评估活动根据信息接收端的需求展开,并且上一级的评估总是建立在下一级评估所提供的信息基础之上。     

Alternative perspectives on learning outcomes: challenges for assessment

In discussing the relationship between curriculum and assessment it is commonly argued that assessment should be aligned to curriculum or, alternatively, that they should be congruent with each other. This article explores that relationship in five educational contexts in the UK and in Europe, ranging across school education, workplace learning, vocational education and higher education. Four main themes are highlighted: construct definition, progression, assessment procedures, and system-level accountability. What emerges from the five case studies under review is a multi-layered process of knowledge being constructed in diverse ways at different levels in each context. The article concludes that, rather than thinking in terms of either alignment or congruence, these relationships are better understood in terms of non-linear systems embracing curriculum, pedagogy and assessment.     

Evaluating student perceptions of learning processes and intended learning outcomes under inquiry approaches

Inquiry-based learning (IBL) is promoted as a teaching approach that can enhance student learning outcomes. IBL can be categorised according to scale (e.g. tasks, course/module/paper, degree), mode (structured, guided, open) and framing (information or discovery-oriented). Our research used a survey instrument to determine how student perceptions of learning processes and intended learning outcomes (ILOs) varied in response to courses with different modes and framing of IBL. The survey was completed by 940 students in 15 IBL courses across a range of disciplines and levels. All types of IBL courses were well rated by students for encouraging learning processes and ILOs congruent with a well-designed inquiry experience. Regarding modes of IBL, there was a clear hierarchy from open (most highly rated) to guided and then structured inquiry. For the framing of IBL, courses using discovery-oriented IBL were more highly rated than information-oriented IBL. Overall, the most highly rated course design was open, discovery-oriented IBL, but other types of IBL remained important in terms of developing research and inquiry skills. To determine whether all types of IBL courses are promoting enhanced ILOs compared to more traditionally taught courses, a comparative study should be undertaken.     

Intelligent design: student perceptions of teaching and learning in large social work classes

Research into the effects of large classes demonstrates that students are disadvantaged in terms of higher order learning because interactions between teachers and students occur at lower cognitive levels. This has significance for social work education, with its emphasis on the development of critical thinking and problem solving, both higher order cognitive skills. This paper reports on quantitative and qualitative research that explored social work students’ perceptions of different teaching and learning strategies in a large mental health course designed with reference to principles of student‐centred learning and constructive alignment. Findings revealed that well‐integrated design, relevance to the real world and teacher enthusiasm were seen as most useful by students, rather than particular learning strategies per se. Higher satisfaction ratings and grades were also associated with this student‐centred course compared with an earlier traditional lecture‐style course. The paper concludes that design based on the interplay between diverse learning activities, including lecture input, strengthened the student‐centred orientation of learning and recommends further research that compares learning outcomes associated with these contrasting approaches to professional education.     

A conceptual framework for analysing the impact of influences on student engagement and learning

The notion that ‘customer satisfaction’ should be the ultimate measure of quality provision of any service organization is often accepted in the higher education context. However, measuring the quality of an educational institution based on students’ satisfaction is insufficient as it diverts the focus from student development, advancement and growth to an affective evaluation of the service. More appropriate measures of quality of higher education institutions are student engagement and learning. This study describes the likely impact of perceived overall quality of higher education institutions on student engagement. The contribution of this study is twofold. First, it draws attention to the importance of higher education institution quality as the main institutional-level determinant of student engagement and, second, it suggests a comprehensive multilevel conceptual framework for its empirical testing.     

Evidencing learning outcomes: a multi-level,multi-dimensional course alignment model

This conceptual framework proposes a multi-level, multi-dimensional course alignment model to implement a contextualised constructive alignment of rubric design that authentically evidences and assesses learning outcomes. By embedding quality control mechanisms at each level for each dimension, this model facilitates the development of an aligned curriculum. The use of rubrics is a crucial theme for many higher education institutions owing to the binding requirement by universities to provide evidence to quality assurance agencies. The success of evidencing learning outcomes through rubrics, however, is only one piece of the puzzle. The other is the contextualised constructive alignment of intertwined factors. Despite the significance of embedding these factors, there has been little, if any, systematic framework in this area. The two key instrumental forces underpinning the conception of this model are: seeking external accreditation and the implementation of programme enhancement thus realising the strategic agenda for an Australian university.     

Curriculum Mapping: A Method to Assess and Refine Undergraduate Degree Programs

Over the past several decades, there has been increasing interest in program‐ and university‐level assessment and aligning learning outcomes to program content. Curriculum mapping is a tool that creates a visual map of all courses in the curriculum and how they relate to curriculum learning outcomes. Assessment tools/activities are often included in the curriculum map. The School of Food Science, a joint program between the Washington State Univ. and the Univ. of Idaho, recently completed a curriculum mapping and assessment exercise. The goal of this exercise was to determine the alignment of the undergraduate curriculum with program learning outcomes and the degree of student mastery of these outcomes. Curriculum learning outcomes were revised from existing learning outcomes to better align with university learning outcomes and the IFT Core Competencies. All undergraduate courses were mapped to the IFT Core Competencies, and several competencies were assessed to determine student mastery. The mapping and assessment activities revealed potential gaps and redundancies in course content, as well as student ability to meet standards for competency. The results of the mapping and assessment exercises will be used to refine the content delivered to undergraduate students in the School of Food Science, with the overall goal of increasing the quality of the education provided to these students and helping them to be more prepared for a successful career in food science.     

Evaluating the quality of work-integrated learning curricula: a comprehensive framework

There are many different forms that work-integrated learning (WIL) takes and variants go by a range of different names. Based on current literature, key dimensions, shared by the various and disparate forms of WIL curricula, were identified and operationalised in a measurement model. The key dimensions identified were: authenticity, integrated learning supports (both at university and the workplace), alignment (of teaching and learning activities and assessments with integrative learning outcomes), supervisor access and induction/preparation processes. It is suggested that variations in the way that WIL courses or subjects are designed within these dimensions are the basis for different expressions of the quality of such courses. A latent construct measurement model was developed and validated with a sample of Australian and UK students. This paper presents the model and discusses the results of the validation study. It is proposed that the measures validated in this study will be useful for evaluating a wide variety of WIL curricula.     

Researching learning outcomes: building confidence in our conclusions

The ESRC Teaching and Learning Research Programme is expected to produce knowledge of high social scientific quality and practical pedagogic value. It faces, in a particularly acute form, the issue of how confidence in its conclusions can be built. This article begins with some conceptual clarification of the notion of a ‘warrant’ for research findings and distinguishes between warrants in social research and warrants for educational policy and practice. It then, in the specific context of research on learning outcomes, proposes an integrated model that recognizes the importance of, and relationships among, criteria associated with research design and empirical work, theoretical coherence, relationship with other research, and user affirmation. It concludes by emphasizing the need for continuing debate about, and development of, the roles and relationships of ‘researchers’ and ‘users’.     

Collaborative learning practices: teacher and student perceived obstacles to effective student collaboration

While the educational literature mentions several obstacles affecting the effectiveness of collaborative learning (CL), they have often been investigated through the perceptions of only one actor, either teachers or students. Therefore, some sources of obstacles that teachers and students encounter may not have been revealed. In this study, 19 teachers and 23 students in different disciplines at a pre-service teacher education faculty at a university in Vietnam were interviewed. In total, 47% of the teachers taught science subjects and 53% taught social subjects; 35% of the students majored in science subjects, 57% in social subjects, and 8% in primary education. With regard to study cohorts, 22% of the students were in the first and second year while 78% were in the third and fourth year of their four-year bachelor’s programme. These programmes produce qualified teachers for primary and secondary schools nationwide. Based on grounded theory analysis, four common obstacles to collaboration were identified: students’ lack of collaborative skills, free-riding, competence status, and friendship. Furthermore, the results showed three interrelated antecedents that contribute to these obstacles. Central to the antecedents is the strong focus of the teachers on the cognitive aspects of CL, which led the participating teachers to neglect the collaborative aspects of CL. These antecedents were demonstrated in the ways teachers set CL goals, provided instruction, and assessed student collaboration. This study may be useful for educators, designers, and researchers to foster the quality of student collaboration.     

自主学习质量控制要素探析

在现代远程教育中，影响自主学习质量的要素包括学习观念、学生的知识基础和社会背景、办学条件、质量评价和质量控制体系以及考试与考核等。提高自主学习质量的措施与途径包括转变观念、加强师资队伍建设、完善现代化教学设备和教学支持服务体系、建立自主学习质量体系和加强自主学习质量监控以及全面深化教学全过程的改革等。     

The place of content and pedagogy in shaping sustainability learning outcomes in higher education

This research investigates the ways in which undergraduate courses dealing with the environment address sustainable development (SD), and contribute to the development of sustainability learning outcomes (SLO). The participants in the study were 13 instructors, and 360 students who were enrolled in 13 courses that addressed the environment in a technical university. Mixed methods were used for data collection and analysis. We found that courses that address the environment are varied in the way that they address sustainable development, and in the pedagogy they use. We also found links between teaching methods and the pattern of reported SLOs: skills development was promoted mainly in courses with a high degree of participatory learning; changes in awareness and attitudes were reported mainly in courses that addressed sustainable development, and motivation to promote sustainable development was reported mainly in courses that addressed sustainable development and also provided opportunities for active and participatory learning. The implications of our research are discussed.