Sources of difficulty in assessment: example of PISA science items

ABSTRACT

The understanding of what makes a question difficult is a crucial concern in assessment. To study the difficulty of test questions, we focus on the case of PISA, which assesses to what degree 15-year-old students have acquired knowledge and skills essential for full participation in society. Our research question is to identify PISA science item characteristics that could influence the item’s proficiency level. It is based on an a-priori item analysis and a statistical analysis. Results show that only the cognitive complexity and the format out of the different characteristics of PISA science items determined in our a-priori analysis have an explanatory power on an item’s proficiency levels. The proficiency level cannot be explained by the dependence/independence of the information provided in the unit and/or item introduction and the competence. We conclude that in PISA, it appears possible to anticipate a high proficiency level, that is, students’ low scores for items displaying a high cognitive complexity. In the case of a middle or low cognitive complexity level item, the cognitive complexity level is not sufficient to predict item difficulty. Other characteristics play a crucial role in item difficulty. We discuss anticipating the difficulties in assessment in a broader perspective.     

‘Does it answer the question or is it French fries?’: an exploration of language supports for scientific argumentation

ABSTRACT

Reform initiatives around the world are reconceptualising science education by stressing student engagement in science practices. Yet, science practices are language-intensive, requiring students to have strong receptive and productive language proficiencies. It is critical to address these rigorous language demands to ensure equitable learning opportunities for all students, including English language learners (ELLs). Little research has examined how to specifically support ELL students’ engagement in science practices, such as argumentation. Using case-study methodology, we examined one middle school science teacher's instructional strategies as she taught an argumentation-focused curriculum in a self-contained ELL classroom. Findings revealed that three trends characterized the teacher’s language supports for the structural and dialogic components of argumentation: (1) more language supports focused on argument structure, (2) dialogic interactions were most often facilitated by productive language supports, and (3) some language supports offered a rationale for argumentation. Findings suggest a need to identify and develop supports for the dialogic aspects of argumentation. Furthermore, engaging students in argumentation through productive language functions could be leveraged to support dialogic interactions. Lastly, our work points to the need for language supports that make the rationale for argumentation explicit since such transparency could further increase access for all students.     

What’s in a Name: Differential labelling of plant and animal photographs in two nationally syndicated elementary science textbook series

This study investigated plant and animal photographs in elementary science textbooks to discern whether there were disparities in the number of plant and animal photographs or in how those photographs were labelled. We examined the Life Science sections of two nationally syndicated (USA) textbook series. For each text, we identified the photographs with plant and/or animal content and evaluated them for two features: (1) the subject of the photograph, and (2) the specificity of the label (name) provided. We found that photographs with animal subjects were more numerous than those with plant subjects; they also represented a greater diversity of animals and had a higher instance of repetition than did plant photographs. We also found a significant naming disparity: animal photographs were three times more likely to be provided with a specific label (common name) than were plant photographs. Not only were plant photographs less likely to be provided with a specific name for the plant (e.g. orchid or dandelion), but also they were commonly identified only by the name for a plant part (such as flower or leaf) or life‐form (e.g. tree or shrub). To address the disparity revealed by this study, and to encourage student interest in and knowledge about plants, we recommend that educators go beyond textbooks to expose students to a diversity of named plants, and present plants as distinct organisms rather than as a collection of parts.     

What do Children Aged Four to Seven Know about the Digestive System and the Respiratory System of the Human Being and of Other Animals?

The object of this paper is to learn what little children know about the inside of their bodies before they have studied these particular aspects at school. The data for our project were collected by means of drawings made by 342 Spanish children aged four to seven. They were required to depict where the food, drink, and air which enter their bodies go. In addition to this, we intend to study how the ideas of children evolve during three consecutive years. For this purpose, a group of 32 subjects was monitored. Our findings show that the children recognise specific organs in their own bodies which they associate with the intake of food and air. Furthermore, they usually extrapolate those organs to other animals they are familiar with. Their ideas about the digestive system are more adequate than the ones about the respiratory system, though their ideas improve as they become older, above all those concerning the digestive system. Taking these findings as a basis, this paper suggests some points to be taken into consideration for teaching purposes.     

“A moving target”: a critical race analysis of Latina/o faculty experiences,perspectives, and reflections on the tenure and promotion process

This article examines how Latina/o professors perceive, experience, and reflect on the tenure and promotion process. Findings for this longitudinal study are drawn from a purposive sample of nine female and seven male, Latina/o tenure-track faculty participants. Using a Critical Race Theory, Latino Critical (LatCrit) Race Theory, and Chicana Feminist framework, this article documents fundamental inequities in the tenure and promotion policies and practices that affected the Latina/o faculty in this study. Using narrative data, educational biographies, in-depth semi-structured interviews, and unstructured ethnographic interviews, this study found four common themes: (1) tenure and promotion processes functioned as a “tool of fear,” (2) the tenure and promotion process was like a “moving target,” (3) tenure provided limited forms of respect but not full membership, and (4) Latina/o supervivencia enabled the professors to thrive despite unsupportive and sometimes hostile campus and departmental climates.     

The Use of Criteria in Argumentation and the Construction of Environmental Concepts: A university case study

We have analysed the processes of argumentation of three university student groups (A: six students, C: five students, and J: seven students) while making a decision about an environmental problem (selection of a heating system). The discussions took place in three 1½‐hour sessions that were audio‐taped and transcribed. For the analysis of the oral discussions, on the one hand, we have taken into account some of the dimensions characterising the quality of this decision‐making process, including the number and variety of criteria utilised, whether criteria, which did not favour the selected option, were considered and whether priorities were established among criteria, and, on the other hand, the use of environmental concepts such as renewable and sustainability as well as the meanings that were constructed for both concepts. We have determined that the students in this study proposed and utilised, both explicitly and implicitly, a high number and great variety of criteria to support their choices, although they were rarely able to consider contradictory evidence; that is, those that demonstrated disadvantages of the option selected. In terms of the construction of knowledge, we observed that in some groups the proposed task favoured the construction of a concept of sustainability that took the future into consideration and which was utilised as the most important justification in their selection. In terms of the concept of renewable, we found that they did not relate depletion of resources to economic consequences. We discuss the implications for the educational competence development.     

Investigating the emerging generic features of the blog writing task across three discrete learner groups at a higher education institution

Increasingly universities have supported several innovative flexible learning projects over the past few years, and there exists a small but enthusiastic group of staff who have demonstrated that something is possible. Our task is to suggest additional strategies that can realize some of the strategic goals set for flexible learning over the next few years. This paper is based upon the assumption that a strategic direction has already been chosen and suggests some strategies to ensure practical and realistic outcomes.

Création d'une communauté virtuelle de troisième cycle universitaire: apprentissage de conduites d'évaluation.

Durant plusieurs années un nombre croissant d'universités ont soutenu de nombreux projets innovants d'apprentissage flexible et il existe des groupes restreints mais enthousiastes d'enseignants qui ont démontré que l'on peut faire quelque chose. Notre tâche est de suggérer des stratégies complémentaires qui permettent d'atteindre certains des objectifs fixés pour les apprentissages flexibles dans les prochaines années. Cet article est fondé sur l'hypothèse qu'une direction stratégique a déjà été choisie et suggéré certaines stratégies pour obtenir des résultats pratiques et réalistes.

Erstellen einer virtuellen Gemeinschaft für Absolventen: Lernantrieb durch Einstufung

In zunehmendem Maße haben Universitäten in den letzten Jahren verschiedene innovative Projekte für Flexibles Lernen unterstützt. Es gibt eine kleine aber begeisterte Gruppe von Mitarbeitern, die gezeigt hat, daß auf diesem Gebiet durchaus etwas möglich ist. Unsere Aufgabe ist, zusätzliche Strategien vorzuschlagen, die einige der strategischen Ziele, die für Flexibles Lernen gesetzt sind, in den nächsten Jahren erreichen helfen. Dieses Papier geht von der Annahme aus, daß es schon eine strategische Richtung gibt und schlägt einige Taktiken vor, um praktikable und realistische Resultate sicherzustellen.     

Scaffolding young children’s understanding of symbolic objects

Symbolic objects are cognitive tools children must learn to master very early in life. A crucial factor in cognitive development is the instruction or informational support children receive in social contexts. While numerous studies have investigated the relation between instruction and development in various domains, like language or symbolic play, still little is known concerning the role of this factor in the comprehension of symbolic objects. The purpose of this article is to illustrate with an overview of the results of the authors’ research how instruction scaffolds the comprehension and use of symbolic objects. In this framework instruction is conceived as a process that mediates the skills that children develop and meanings they build. Some educational implications are discussed.     

Teacher professional development through collaborative action research: impact on foreign English-language teaching and learning

In 2011 we, a group of English-as-a-foreign-language teachers at a secondary school in Argentina, decided to investigate our teaching practices through collaborative action research so as to improve our students’ learning opportunities and thus revitalise English-language teaching in our context. We implemented and evaluated the integration of content and language learning in our classrooms through the development of our own materials. The experience revealed a growth in professional development and how our motivation and autonomy influenced our students’ motivation and language development. In our attempt to disseminate our experience as a group, this report particularly focuses on the evaluation facets of our collaborative action research project so as to encourage other teachers and teacher-researchers to adopt collaborative action research to improve their own practices.     

Correlation between variables subject to an order restriction,with application to scientometric indices

Variables subject to an order restriction, for instance Y ≤ X, have a bivariate distribution over a non-rectangular joint domain that entails a non-null and potentially large structural relation even if the variables show no association (in the sense that particular ranges of values of X do not co-occur with particular ranges of values of Y). Order restrictions affect a number of scientometric indices (including the h index and its variants) that are routinely subjected to correlational analyses to assess whether they provide redundant information, but these correlations are contaminated by the structural relation. This paper proposes an alternative definition of association between variables subject to an order restriction that eliminates their structural relation and reverts to the conventional definition when applied to variables that are not subject to order restrictions. This alternative definition is illustrated in a number of theoretical cases and it is also applied to empirical data involving scientometric indices subject to an order restriction. A test statistic is also derived which allows testing for the significance of an association between variables subject to an order restriction.