Sources of difficulty in assessment: example of PISA science items

ABSTRACT

The understanding of what makes a question difficult is a crucial concern in assessment. To study the difficulty of test questions, we focus on the case of PISA, which assesses to what degree 15-year-old students have acquired knowledge and skills essential for full participation in society. Our research question is to identify PISA science item characteristics that could influence the item’s proficiency level. It is based on an a-priori item analysis and a statistical analysis. Results show that only the cognitive complexity and the format out of the different characteristics of PISA science items determined in our a-priori analysis have an explanatory power on an item’s proficiency levels. The proficiency level cannot be explained by the dependence/independence of the information provided in the unit and/or item introduction and the competence. We conclude that in PISA, it appears possible to anticipate a high proficiency level, that is, students’ low scores for items displaying a high cognitive complexity. In the case of a middle or low cognitive complexity level item, the cognitive complexity level is not sufficient to predict item difficulty. Other characteristics play a crucial role in item difficulty. We discuss anticipating the difficulties in assessment in a broader perspective.     

School Religious-Cultural Attributes and School Principals’ Leadership Styles in Israel

Schools in Western cultures, including Israel, have been transformed from closed to open systems. This change is especially important to the principal’s role and leadership style. This article seeks to investigate the cultural-religious differences in Israeli Jewish principals’ leadership styles. Data were collected from 1,859 teachers, 34% from ultra-Orthodox schools, 50% state-religious schools, and 16% from state-secular schools. Regression results support the study's main hypotheses: cultural-religious orientation explains participative and transactional leadership styles but not transformational leadership. We discuss the implications of these findings for both theory and practice.     

Measurement error in short-term power testing in young people

The aim of this study was to examine the consistency or reproducibility of measuring cycling peak power in children and adults. Twenty-seven pre-pubertal girls and boys and 27 female and male physical education students (age 9.8 +/- 0.5 and 24.4 +/- 4.3 years, respectively; mean +/- s) participated in the study. All participants performed five tests over 15 days and underwent a habituation session before the study. Each test included four sprints against four different braking forces. We found that braking forces of 7.5% of body weight in children and 10% of body weight in adults were too high for most of the participants to elicit maximal cycling power. Unlike the children, the physical education students improved their performance between session 1 and session 2 (1025 +/- 219 vs 1069 +/- 243 W; P < 0.001). Therefore, to obtain reproducible measures of cycling peak power, a habituation session including a complete test protocol (i.e. warm-up plus three sprints) is highly recommended. When the protocol included three sprints in children and at least two sprints in adults, measurement of cycling peak power was found to be highly reliable (test-retest coefficient of variation approximately 3%). Finally, to avoid performance fluctuations, especially over several consecutive evaluations (e.g. longitudinal studies), it is necessary to maintain high motivation in children.     

The educational aims of the OECD in its TALIS insight and lesson reports: exploring societal orientations

ABSTRACT

The purpose of this study was to explore the societal orientations (individualist vs. collectivist) of educational aims, in constructions of teacher professionalism framed by the Organisation for Economic Co-operation and Development (OECD) Teaching and Learning International Survey (TALIS) insight and lesson reports. The critical discourse analysis of OECD TALIS documents suggests that the OECD discourse on teacher professionalism attested to a dual orientation: individualist and anti-collectivist. Our results contribute to the theoretical understanding of the educational aims of the OECD, which lead the global discourse in education, and of the cultural orientation that is part of its conceptualization of new professional teaching. The article discusses the implications of its findings.     

Does contextual change determine long-term forgetting?

A series of experiments were run to test the hypothesis that “spontaneous forgetting” could result from subtle contextual changes. The first experiment demonstrated that when Sprague-Dawley male rats are trained in a runway alley with a food reinforcer, retention performance is dramatically affected by a change in the pattern of the walls of the training apparatus when testing takes place 1, 3, or 5 days following training and not after 1 week. Experiment 2 demonstrated that this performance deficit cannot be alleviated by any of the three selected cuing treatments, whereas “spontaneous forgetting” (resulting from a training-to-test interval of 14 days) can be. These data indicate that the detrimental effect of contextual change reduces over time, and that such an effect cannot be interpreted in terms of retrieval failure. These results lend strong support to the argument that the disruptive contextual change effect is basically different from the disruptive effect that results from an extended training-to-test interval.     

Opening the Regulatory Black Box of Clinical Cancer Research: Transnational Expertise Networks and “Disruptive” Technologies

Building on previous work on “regulatory objectivity,” the paper examines recent translational research and cancer genomics to explore the bundle of scientific and regulatory activities that generate and manage the platforms at the core of clinical trials, the “gold standard” of clinical research and evidence-based medicine. In particular, the paper explores the activities of a chain of mediators within a seamless regulatory web characterized by the interaction of endogenous and hybrid regulatory activities that are neither hierarchical nor linear. We contend that a full understanding of the dynamics of regulation in the biomedical domain ought to consider this chain of mediations; that their analysis necessitates understanding the content of the practices they regulate; and that in addition to examining the interactions between different regulatory modalities, we need to pay attention to their development insofar as regulation, far from being mere routine, leads to the emergence of novelty by coproducing the entities it regulates. These activities include not only setting out the conditions that must be respected in order to produce reliable test results, but also the conditions that define the relations (within a clinical context) between the different components of diagnosis as well as the consequences of such relations on clinical judgment. This is why we cannot treat organizational practices as distinct from the content of bio-clinical activities.     

Coherence of retrieval cues,rather than additivity,determines prior cuing effectiveness in the rat

In previous studies, we have shown that adding a supplementary cue in a prior-cuing treatment may dramatically reduce its initial effectiveness. This series of experiments was aimed at testing the hypothesis that such a lack of additivity among retrieval cues could be due to the combined reactivation of memory attributes related to incompatible behavioral responses. For that purpose, rats were partially trained to reach the lighted arm (light condition; usual training condition) or the darkened arm (dark condition) of a Y-maze in order to avoid an electric foot-shock. The first experiment demonstrates that animals learned the two versions of the task similarly and that an exposure to the light similarly acts as an effective retrieval cue after a 24-h interval. In the second experiment, the effectiveness of exposure to the grid alone and to the light alone as well as to the combined light + grid cue was investigated after a 21-day training-to-test interval in the light condition, where light was related to safety and grid was associated with the shock, and in the dark condition, where both light and grid were related to the shock. The results indicate that whereas the compound light + grid cue was an ineffective retrieval cue in the light condition, it acted as an effective retrieval cue in the dark condition, supporting the hypothesis that coherence between retrieval cues is a necessary condition for effective cuing.     

Marketing teacher quality: critical discourse analysis of OECD documents on effective teaching and TALIS

ABSTRACT

This study uses critical discourse analysis (CDA) to examine Organisation for Economic Cooperation and Development (OECD) texts on teacher quality and the Teaching and Learning International Survey (TALIS) framework. Specifically, it explores the forewords of documents written by OECD leaders, which we believe are charged with meanings related to the Global Education Reform Movement (GERM) discourse. We suggest that CDA of the texts sheds light on the manner in which OECD leaders attempt to gain normative control in the teacher quality discourse. Based on Fairclough’s three-dimensional framework for studying discourse, our analysis shows that the OECD (a) uses a discourse of fear to market teacher quality in light of global changes, implicitly framing teachers as ‘bad teachers’; (b) advocates reliance on the organization as a protector and (c) promises a remedy by regulating teachers in the name of effectiveness and the knowledge economy. The study offers a nuanced insight into OECD efforts to promote normative control in the teacher quality discourse, using three dimensions of discourse (i.e. the textual micro linguistic dimension, the meso-interdiscursive dimension, and the macro sociocultural dimension) that help gain ideational powers.     

How Do the Different Components of Episodic Memory Develop? Role of Executive Functions and Short‐Term Feature‐Binding Abilities

This study investigated the development of all 3 components of episodic memory (EM), as defined by Tulving, namely, core factual content, spatial context, and temporal context. To this end, a novel, ecologically valid test was administered to 109 participants aged 4-16 years. Results showed that each EM component develops at a different rate. Ability to memorize factual content emerges early, whereas context retrieval abilities continue to improve until adolescence, due to persistent encoding difficulties (isolated by comparing results on free recall and recognition tasks). Exploration of links with other cognitive functions revealed that short-term feature-binding abilities contribute to all EM components, and executive functions to temporal and spatial context, although ability to memorize temporal context is predicted mainly by age.