A discussion of some methodological issues in international assessments

Three topics addressed in the previous chapters are identified and discussed from a somewhat different perspective from that of the chapter authors. The topics are: the level of scoring in assessment studies, translation of test items, and sampling of curriculum content. Based on the analysis of these topics, five recommendations are offered. International assessments should be scored and reported at a more specific level that is currently the practice. There is a need of sound statistical checks on the quality of item translations. Rather than sampling the curriculum only once or twice, sampling could be in real time and on a permanent basis. Tests could be administered with open books and a well-chosen time limit per item. Finally, schools could be instructed to prepare their students for the assessment.     

Naturalistic case studies exploring social-cognitive mechanisms,and some methodological issues in research on problems of teachers

The argument is advanced that research could be more effective for use by teachers and simultaneously more scientific, in the sense that it would emulate the historical process of the development of quantitative science, if it were (1) of the form referred to as naturalistic case studies and (2) oriented toward discovering the cognitive and social interaction mechanisms that underlie the learning processes in classrooms. The limitations of individual case studies of children with difficulties learning in school are examined and some practical approaches to interventionist case studies done in classrooms are considered. It is proposed that the standard view of quantitative research in the social sciences derives from the physics curriculum rather than from the history of physics. This view also mistakenly places higher priority on finding predictive linear relationships among measurements rather than qualitative understanding of mechanisms, which could guide researchers toward useful quantitative research. The researcher who has not yet penetrated the world of the individuals being studies is in no firm position to begin developing predictions, explanations, and theories about that world.     

Evaluation of projects in the developing world: some cultural and methodological issues

This article stems from fieldwork for a series of short external summative evaluations of environmental education projects in developing countries, and distills from these experiences some cultural issues, constraints and differences that either confront, or are created by western evaluators in the developing world. The assumption is that the range of cultural issues, differences and constraints discussed here are either absent from, or are present to a markedly different degree, in western evaluation environments. The paper concludes by suggesting some revisions to evaluative methodology as a contribution to resolving some of the issues.     

Conducting Research into European Union Education and Training Policy: some theoretical and methodological considerations

This paper has several purposes. It introduces a new project funded by the European Commission within its Training and Mobility of Researchers (TMR) programme; it describes the research design of the Oxford part of that project, which is now beginning, and of a comparative study which was already being undertaken in Oxford on the implementation of the ‘European Dimension’ in education in England, Scotland and Wales; and it describes a number of theoretical and methodological problems we anticipate having to address as the studies proceed.     

Complexity and action research: exploring the theoretical and methodological connections

Complexity theory is essentially a formal attempt to question how coherent and purposive wholes emerge from the interactions of simple and sometimes non-purposive components. Explicit recognition of complexity can provide a fresh and enlightening perspective on action research. Through an expository discussion of the foundational postulates of complexity theory this article demonstrates the theoretical and methodological connections between complexity and action research, with particular emphasis on the relevance of complexity in educational and workplace contexts. Complexity is an emerging theoretical perspective, which presents possibilities for revolutionizing approaches to action research, as well as strengthening arguments promoting the value of action research in a wide range of contexts. Complexity, it is argued, can provide a valuable theoretical underpinning for action research. Furthermore, action research provides a valid methodological approach to the study of complexity. This article is primarily theoretical and attempts to demonstrate the application of complexity to a specific action research project will be left to future publication(s). Rather, this article explores the general applicability of complexity as both theory and metaphor in action research. The article begins with a brief exploration of the theory, particularly focusing on its application in the social sciences. The theoretical and methodological connections between complexity and action research are discussed through several of the foundational postulates of complexity, how these manifest in action research and how they add to our understanding of action research itself.     

Sharpening the focus: methodological issues in analysing on-line conferences

Abstract

As on-line discussion becomes an increasingly important and significant aspect of teachers' professional education, most particularly as part of distance learning environments, there is a need to understand and develop effective analytic techniques that provide insights into the processes at work within these systems. The computer conferencing environment at the centre of this study is a complex, unstructured discussion forum, of a type increasingly found as a component of continuing professional development courses. In these situations the evidence of knowledge building processes is a key concern. This study concentrates on a close examination of some existing analytic tools and considers their contribution to identifying the processes at work within these on-line environments, as teachers discuss aspects of education.     

Speaking back to the deficit discourses: a theoretical and methodological approach

The educational attainment of Aboriginal and Torres Strait Islander students is often presented within a deficit view. The need for Aboriginal and Torres Strait Islander researchers to challenge the societal norms is necessary to contribute to the struggle for self-determination. This paper presents a theoretical and methodological approach that has enabled one researcher to speak back to the deficit discourses. Exemplification of how Indigenous Critical Discourse Analysis (in: Hogarth, Addressing the rights of Indigenous peoples’ in education: A critical analysis of Indigenous education policy, Queensland University of Technology, Brisbane, 2016) identifies the power of language to maintain the inequitable positioning of Aboriginal and Torres Strait Islander peoples within Australian society is provided. Particular focus is placed on the Aboriginal and Torres Strait Islander Education Action Plan 2010–2014 (in: MCEECDYA, Aboriginal and Torres Strait Islander Education Action Plan (2010–2014), 2011) and how policy discourses ignore the historical, political, cultural and social factors that influence the engagement and participation of Indigenous peoples in education today. The paper argues for the need to personalise methodological approaches to present the standpoint of the researcher and, in turn, deepens their advocacy for addressing the phenomenon. In turn, the paper presents the need to build on existing Indigenous research frameworks to continue advocating for the position of Indigenous research methodologies within the Western institution.     

Argumentation in elementary science education: addressing methodological issues and conceptual understanding

In this review essay I respond to issues raised in Mijung Kim and Wolff-Michael Roth’s paper titled “Dialogical argumentation in elementary science classrooms”, which presents a study dealing with dialogical argumentation in early elementary school classrooms. Since there is very limited research on lower primary school students’ argumentation in school science, their paper makes a contribution to research on children’s argumentation skills. In this response, I focus on two main issues to extend the discussion in Kim and Roth’s paper: (a) methodological issues including conducting a quantitative study on children’s argumentation levels and focusing on children’s written argumentation in addition to their dialogical argumentation, and (b) investigating children’s conceptual understanding along with their argumentation levels. Kim and Roth emphasize the difficulty in determining the level of children’s argumentation through the Toulmin’s Argument Pattern and lack of high level arguments by children due to their difficulties in writing texts. Regarding these methodological issues, I suggest designing quantitative research on coding children’s argument levels because such research could potentially provide important findings on children’s argumentation. Furthermore, I discuss alternative written products including posters, figures, or pictures generated by children in order to trace children’s arguments, and finally articulating argumentation and conceptual understanding of children.     

Recurrent education: some economic issues

Despite increasing interest internationally concerning the concept of recurrent education, comparatively little work has been undertaken on its economic implications. This paper applies the principles of social cost-benefit analysis to the recurrent education model and contrasts it with the conventional apprenticeship or front-end model of provision. Consideration is given to the implications of redistributing education and training over the lifespan of the individual, and also of utilising alternatives to conventional full-time formal educational provision. It is suggested that, when a range of factors is taken into account, recurrent education may not be so obviously uneconomic as simple analyses suggest. On the other hand, the concept may pose rather starkly the possible conflict between efficiency and equity objectives in educational policy-making. These findings must be tentative, however, because the cost implications of different forms of provision geared to the specific characteristics and needs of adults have been little researched. The paper concludes by asking how far the cost-benefit framework itself is an appropriate tool for choosing between alternative educational strategies, when decisions are non-marginal and the criteria for educational policy-making may not be primarily economic.The author wishes to thank Professor Gareth Williams for helpful suggestions on an earlier draft.     

Social technologies for online learning: theoretical and contextual issues

Three exemplars are presented of social technologies deployed in educational contexts: wikis; a photo-sharing environment; and a social bookmarking tool. Students were found to engage with the technologies selectively, sometimes rejecting them, in the light of their prior conceptions of education. Some students (a minority in all the studies) were unsympathetic to the educational philosophy underpinning the technology’s adoption. The paper demonstrates, through an examination of in-context use, the importance of sociocultural factors in relation to education, and the non-deterministic nature of educational technology. The academic study of technology has increasingly called into question the deterministic views which are so pervasive in popular discourse and among policy-makers. Instead, sociocultural factors play a crucial role in shaping and defining technology and educational technology is no exception, as the examples in the paper show. The paper concludes by drawing out some implications of the examples for the use of social technologies in education.     

Qualifications Frameworks: some conceptual issues

The aim of this article is to contribute to realising the progressive and democratic opportunities that National Qualifications Frameworks (NQFs) can offer. In doing so it will be critical of many of the ways that NQFs have been interpreted to date and the claims that have been made for them. The article has six sections. Section 1 considers some of the reasons for the recent growth of interest in NQFs. Section 2 is concerned with the widely shared goals of NQFs and some of the contradictions and problems that they involve. Section 3 discusses the very different forms that NQFs can take. Section 4 examines a number of issues that underlie all approaches to qualifications but are made more explicit by qualification frameworks and concludes by questioning the much lauded claim that NQFs can promote and accredit informal learning. The concluding section considers the longer term implications for the future of NQFs of the earlier analysis.     

Emerging issues in the genetics of dyslexia: a methodological preview

A review of the classic and recent evidence on the genetics of reading disability (RD) shows encouraging progress, and accumulating evidence of genetic risk factors that operate within families and are separately localizable to more than one chromosomal region. The accelerating pace of these findings, however, suggests the need to consider some methodological issues about the design and interpretation of current and future studies. A major issue is the shape of the distribution of reading ability in the population, and we offer three tests of increasing rigor for determining whether those distributions are categorical, and hence not suitable for analyses that depend on the assumption of a continuous normal distribution. These tests are as follows: a nonnormal preponderance of cases with RD (i.e., the hump in the lower end of the distribution); a difference in the within-group variance-covariance matrices for typical readers compared to those with RD; and a correlation between a neurogenetically relevant criterion and a categorical reading variable that is larger than the correlation between the same criterion and a continuous version of the same reading variable. We emphasize also the importance of interactive relationships between multiple genetic loci, the variations in genotypic range as well as type of affectedness, the need to account for remediation variance, and the importance of lifespan changes in the phenotypes.     

Researching academic stress and anxiety in students: some methodological considerations

Despite a long history of interest in North American and Western European literature, researchers in the UK are only now beginning to turn attention to the issue of academic stress in schoolchildren and how it may affect emotional well‐being, health and performance on school assessments. Based on the author's experiences of designing an extensive research project, this article explores the conceptual and methodological difficulties encountered when designing and conducting research in this area. First, there is a lack of precision in terminology used. The terms ‘stress’, ‘anxiety’ and ‘worry’ are used interchangeably in the literature as if they referred to the same phenomenon, and the domains of ‘examination stress’ and ‘academic stress’ are not clearly defined. As a consequence, it is not clear exactly what phenomenon the literature is actually referring to. Second, it is not always clear in the literature what the term ‘stress’ is referring to. In some cases, it is being used to refer to the properties of a stimulus (e.g. an examination) and in other cases to the subjective experience of distress. Assuming a subjective experience of distress will necessarily follow from a particular stimulus is problematic as it fails to account for the interpretation of that stimulus to the student involved. The much ignored construct of test anxiety may offer some advantages to the researcher by having a clearly defined domain and referent. Third, there is an overwhelming bias in the research towards quantification and ways of ‘measuring’ stress and anxiety in students. The usefulness of this approach is considered along with the potential advantages of alternative approaches.