Sources of difficulty in assessment: example of PISA science items

ABSTRACT

The understanding of what makes a question difficult is a crucial concern in assessment. To study the difficulty of test questions, we focus on the case of PISA, which assesses to what degree 15-year-old students have acquired knowledge and skills essential for full participation in society. Our research question is to identify PISA science item characteristics that could influence the item’s proficiency level. It is based on an a-priori item analysis and a statistical analysis. Results show that only the cognitive complexity and the format out of the different characteristics of PISA science items determined in our a-priori analysis have an explanatory power on an item’s proficiency levels. The proficiency level cannot be explained by the dependence/independence of the information provided in the unit and/or item introduction and the competence. We conclude that in PISA, it appears possible to anticipate a high proficiency level, that is, students’ low scores for items displaying a high cognitive complexity. In the case of a middle or low cognitive complexity level item, the cognitive complexity level is not sufficient to predict item difficulty. Other characteristics play a crucial role in item difficulty. We discuss anticipating the difficulties in assessment in a broader perspective.     

Rites of Passage: Does adaptation to university mean severing connections?

Many factors go into the decision of selecting a university, ranging from the overall costs and the programme of study to the institute’s reputation; however, accommodation while at university and proximity of the university to one’s permanent abode also play important roles in this decision-making process. Thus, first-year students experience different stages of separation from their prior networks depending on their individual situations. The results of this Ontario (Canada) 2012 study add to prior research by indicating that separation of students’ living arrangements and distance from home impact on transition and incorporation variables, which influenced students’ adaptation to university.     

Educating Nontraditional Students Through Interdisciplinary Collaboration

For over 30 years, researchers and practitioners have identified challenges unique to small, rural community colleges. The purpose of this study was to examine the distinctive problems facing rural community colleges today and the challenges those institutions must address to fulfill their mission in rural America. There are 5 current challenges commonly identified by 10 rural community college administrators in Kansas: technology use, grant writing, an evolving student population and corresponding changes to the mission, funding inequalities between institutions, and hiring qualified people. Identified challenges were compared with problems described in the literature from the last 30 years.     

Experiments of two-level training in Hungarian higher education

An experiment designed to train engineering students to two levels of terminal achievement within single institutions is reported. The operative engineers (formerly high level technicians) qualify after three years whilst research and development engineers (fully qualified engineers) take five years. Organisational and curricular problems necessarily created by such integrated schemes are described and discussed. Social attitudes in society in general and in the student body in particular make the task of directing students towards the shorter form of training complex and difficult. From the experience gained so far it can confidently be asserted that the experiments in two-level training have been successful.Note by the Editor: This article is an abbreviated version prepared by the Editor of a 90-page UNESCO (copyright) report with the same title, Document ED. 75/WS/64, dated October 1975. We are grateful to the authors and to UNESCO for permission to print this summary of the report.     

The Role of Anatomy Computer-Assisted Learning on Spatial Abilities of Medical Students

Currently, medical education context poses different challenges to anatomy, contributing to the introduction of new pedagogical approaches, such as computer-assisted learning (CAL). This approach provides insight into students' learning profiles and skills that enhance anatomy knowledge acquisition. To understand the influence of anatomy CAL on spatial abilities, a study was conducted. A total of 671 medical students attending Musculoskeletal (MA) and Cardiovascular Anatomy (CA) courses, were allocated to one of three groups (MA Group, CA Group, MA + CA Group). Students' pre-training and post-training spatial abilities were assessed through Mental Rotations Test (MRT), with scores ranging between 0-24. After CAL training sessions, students' spatial abilities performance improved (9.72 ± 4.79 vs. 17.05 ± 4.57, P < 0.001). Although male students in both MA Group and CA Group show better baseline spatial abilities, no sex differences were found after CAL training. The improvement in spatial abilities score between sessions (Delta MRT) was correlated with Musculoskeletal Anatomy training sessions in MA Group (r = 0.333, P < 0.001) and MA + CA Group (r = 0.342, P < 0.001), and with Cardiovascular Anatomy training sessions in CA Group (r = 0.461, P = 0.001) and MA + CA Group (r = 0.324, P = 0.001). Multiple linear regression models were used, considering the Delta MRT as dependent variable. An association of Delta MRT to the amount of CAL training and the baseline spatial abilities was observed. The results suggest that CAL training in anatomy has positive dose-dependent effect on spatial abilities.