Comparison of standardized assessment methods: logistics,costs, incentives and use of data

Critical thinking, problem solving and communication are fundamental elements of undergraduate education, but methods for assessing these skills across an institution are susceptible to logistical, motivational and financial issues. Queen’s University conducted two research studies investigating the use of standardised tests to assess cognitive skill development across the institution. Synthesis of results from implementing the Collegiate Learning Assessment (CLA+), the Critical Thinking Assessment Test (CAT), and the HEIghten^TM test found that student test effort was a significant factor, and effort level correlated with performance at r?=?.33. Test incentives were also a significant factor; effort levels for the $25 financial incentive group were one standard deviation higher than effort for the in-class test group. A dedicated computer lab was the preferred option for computer-based testing. A paper-based test was found to be much simpler to administer, but test results were not available for a long time, therefore limiting the usefulness of data. The true cost of tests was greater than the price of the instrument; recruitment, training, proctoring and marking costs need to be included in the calculation. Generally speaking, alignment of test objectives with student or course objectives, and timeliness of data, were key for participation and motivation.     

Fear appeals in the primary classroom: Effects on test anxiety and test grade

The present study investigated whether fear appeals used prior to a test increased self-reported test anxiety and had a detrimental effect on test scores. Forty primary school pupils were instructed for one week under a low threat condition under which no fear appeals were made and another week under a high threat condition in which fear appeals were made salient. An end-of-week test was given in both conditions. Pupils reported an increase in test anxiety related worrisome thoughts and autonomic reactions under the high threat condition, but not in off-task behaviours. Test scores were lower under the high threat condition, but were not attributable to the increases in test anxiety related thoughts and autonomic reactions. This study adds weight to the argument that fear appeals are a damaging classroom strategy, but the mechanism by which fear appeals are reducing test scores is not yet clear.     

The instrumentation of test-based accountability in the autonomous dutch system

ABSTRACT

Test-based accountability or ‘TBA,’ as a core element of the pervasive Global Education Reform Movement (GERM), has become a central characteristic of education systems around the world. TBA often comes in conjunction with greater school autonomy, enabling governments to assess ‘school quality’ (i.e. test results) from a distance. Often, quality improvement is further encouraged through the publication of these results. Research has investigated this phenomenon and its effects, much of it focusing on Anglo-Saxon cases. This paper, drawing on expert interviews and key policy documents, couples a policy borrowing with a policy instruments approach to critically examine how and why TBA has developed in the highly autonomous Dutch system. It finds that TBA evolved incrementally, advancing towards higher stakes for schools and boards. Further, it argues that school autonomy has been central to the development of TBA in two ways. Firstly, following a period of decentralisation that increased school(board) autonomy, the Dutch government saw a need to strengthen accountability to ensure education quality. This was influenced by international discourse and accelerated by a (politically exploited) national ‘quality crisis’ in education. Secondly, the traditionally autonomous Dutch system, shaped by ‘Freedom of Education’, has at times conflicted with TBA, and has played a significant role in (re)shaping global policy and in mitigating the GERM.     

Preparing beginning teachers for technology integration in education: ready for take-off?

The overall aims of this study are to explore (1) how beginning teachers integrate technology in their practice and (2) the connections between teachers’ technology uses and their pre-service education programmes. Data of this follow-up study were collected through in-depth interviews with beginning teachers. The results reveal that all beginning teachers used a wide range of technological applications, mainly for structured learning approaches, while few created opportunities for student-centred technology use. Further, pre-service learning experiences that impact graduate teachers’ technology use are identified. While teacher educators modelling technology use are an important motivator for beginning teachers to use technology in their own teaching, field experiences seem to be the most critical factor influencing their current practice. Based on the results of this study, recommendations about how to prepare and support pre-service and beginning teachers for technology integration are discussed.     

Ethnicity and engagement in first-year New Zealand law programmes

This paper investigated the extent to which the engagement levels of a self-selected cohort of students enrolled in first-year law programmes at three New Zealand universities varied according to ethnicity. When viewed in the light of factors identified within the international literature as having a bearing on student engagement and, in consequence, academic success and retention, no significant differences were identified in the ways students of Pākehā (European), Māori, Pasifika, Chinese and Indian descent interacted with the law school at which they were enrolled. This was despite some identified differences in students’ backgrounds and motivations for study and in the external factors having an impact on their study. Nevertheless, the identified differences between the ethnicities in relation to personal and external factors can be utilised to enhance and/or improve the engagement of particular groups of students. Overall, the findings indicate a need for law schools to focus on student engagement in first-year programmes with a view to improving the engagement levels of students across all ethnicities.     

Technological pedagogical content knowledge in teacher education: in search of a new curriculum

The aim of this study was to explore the ways in which teacher education institutions (TEI) prepare pre-service teachers for integrating information and communication technology (ICT) in their classroom practise. Specifically, a multiple case study was conducted to examine the ways in which the development of technological pedagogical content knowledge (TPACK) was promoted in the existing curriculum of three TEI in Flanders. In the three cases, data were drawn from semi-structured interviews with the heads of the department and the ICT coordinators. Focus group discussions collected the perspectives of pre-service teachers and teacher educators. The results indicate that (1) the three institutions are moving from ICT as a ‘stand-alone’ course towards embedding ICT across the curriculum and (2) three approaches were adopted for developing pre-service teachers’ TPACK, each representing different ways of understanding the place of ICT in the curriculum. The discussion will focus on the challenges and opportunities inherent in understanding how to develop pre-service teachers’ TPACK in the curriculum of TEI.     

The Relationship between Students' Connections to Out‐of‐School Experiences and Factors Associated with Science Learning

This study examined the relationship between students' out‐of‐school experiences and various factors associated with science learning. Participants were 1,014 students from two urban high schools (secondary schools). They completed a survey questionnaire and science assessment describing their science learning experiences across contexts and science understanding. Using multilevel statistical modelling, accounting for the multilevel structure of the data with students (Level 1) assigned to teachers (Level 2), the results indicated that controlling for student and classroom factors, students' ability to make connections between in‐school and out‐of‐school science experiences was associated with positive learning outcomes such as achievement, interest in science, careers in science, self‐efficacy, perseverance, and effort in learning science. Teacher practice connecting to students' out‐of‐school experiences was negatively associated with student achievement but has no association with other outcome measures. The mixed results found in this study alert us to issues and opportunities concerning the integration of students' out‐of‐school experiences to classroom instruction, and ultimately improving our understanding of science learning across contexts.     

A comparative analysis of international frameworks for 21st century competences: Implications for national curriculum policies

National curricula need to change drastically to comply with the competences needed for the 21^st century. In this paper eight frameworks describing 21^st century competences were analysed. A comprehensive search for information about 21^st century competences was conducted across the official websites of the selected frameworks, resulting in 32 documents that were analysed in detail. Travers and Westbury’s framework of curriculum representations was used to determine horizontal and vertical consistency between the frameworks. The frameworks were compared on their underlying rationales and goals, their definition of 21^st century competences, and the recommended strategies for the implementation and assessment of these skills in educational practice. In addition three international studies were examined to analyse how various countries (EU member states, OECD countries) and schools (SITES studies) deal (or not) with 21^st century competences. The findings indicate a large extent of alignment between the frameworks about what 21^st century competences are and why they are important (horizontal consistency), but intentions and practice seemed still far apart, indicating lack of vertical consistency. The implications of the implementation of 21^st century competences in national curriculum policies are discussed and recommendations are provided.     

Biological Inquiry: A New Course and Assessment Plan in Response to the Call to Transform Undergraduate Biology

We transformed our first-year curriculum in biology with a new course, Biological Inquiry, in which >50% of all incoming, first-year students enroll. The course replaced a traditional, content-driven course that relied on outdated approaches to teaching and learning. We diversified pedagogical practices by adopting guided inquiry in class and in labs, which are devoted to building authentic research skills through open-ended experiments. Students develop core biological knowledge, from the ecosystem to molecular level, and core skills through regular practice in hypothesis testing, reading primary literature, analyzing data, interpreting results, writing in disciplinary style, and working in teams. Assignments and exams require higher-order cognitive processes, and students build new knowledge and skills through investigation of real-world problems (e.g., malaria), which engages students’ interest. Evidence from direct and indirect assessment has guided continuous course revision and has revealed that compared with the course it replaced, Biological Inquiry produces significant learning gains in all targeted areas. It also retains 94% of students (both BA and BS track) compared with 79% in the majors-only course it replaced. The project has had broad impact across the entire college and reflects the input of numerous constituencies and close collaboration among biology professors and students.