The multiple effects of international large-scale assessment on education policy and research

This paper introduces a serendipitous special issue of Discourse: Studies in the Cultural Politics of Education, focusing on the rise of large-scale international testing and performance data in school system accountability and the effects that such changes have produced in education policy and research. These developments are theorised in terms of the reworking of the State and networked modes of governance, including the increased involvement of edu-businesses in education policy-making and enactment, and the emergence of new topological spatialities and connectivities associated with globalisation. We contend that the prevalence of international large-scale assessments has greatly enhanced the mutual ‘visibility’ between participating national (and subnational) schooling systems within a commensurate space of measurement, which in turn makes possible new ways of acting in light of ‘evidence-informed’ policy-making. This analysing and theorising serves to both contextualise and introduce the papers included in this special issue. The paper closes by considering the implications of such developments on education policy and research, and how this necessitates the development for a new approach for researchers engaged in policy analysis, now and into the future.     

Global frameworks,local contingencies: policy translations and education development in India

Policies and programmes pursuing the universalisation of elementary education (UEE) in developing nations have been influenced by a set of complex forces in international, state, and local arenas. This paper explores how a large‐scale standardised assessment programme shaped by international and market‐oriented discourses has been differently re‐worked in the south Indian state of Karnataka. We draw on observation and interview data with educators and administrators to shed some light on their roles in reconstituting the meaning and practice of this programme. The intended frameworks of ‘borrowed’ education policies are not always reproduced or sustained in local contexts. Our paper shows how policies, rather than ‘borrowed’ from one context to another, undergo a process of ‘translation’ involving the contextualisation and inevitable transformation of policies.     

The impact of PISA and the interrelation and development of assessment policy and assessment theory in Japan

PISA presents a new image for academic achievement, which has prompted Japanese education reforms over the past decade to innovate teaching and learning for ‘PISA-style literacy’. Supported by theoretical foundations, particularly with regard to the concept of ‘PISA literacy’ and ‘authentic assessment’, these reforms have accomplished progress in the focus on higher order competencies, such as application and the development of new assessment strategies. However, more recently, various critical discussions of ‘PISA literacy’ are underway in the Japanese academy. They interrogate it more critically and reveal the narrow emphasis on functional application and technical operation. Current assessment practices, which tend to fall into ‘criteria compliance’, are in urgent need of review. There is a need to extend the critical discussions in progress to the new assessment strategies. This paper responds to this, by considering the Japanese acceptance of ‘PISA literacy’ and its assessment, discussing the features and limitations.     

Balancing tensions in educational policy reforms: large-scale implementation of Assessment for Learning in Norway

This study investigates how different stakeholders in Norway experienced a government-initiated, large-scale policy implementation programme on Assessment for Learning (AfL). Data were collected through 58 interviews with stakeholders in charge of the policy; Ministers of Education and members of the Directorate of Education and Training in Norway, and the main actors such as municipality leaders, teachers, school leaders and students. Successful implementation of AfL processes was found in municipalities where there were dialogue and trust between the municipality level, school leaders, teachers and students and where the programme was adapted to the local context. Implementation was challenged when the policy was interpreted as a way of controlling the schools. Despite the successful implementation in some municipalities, the programme did not have any effect upon students’ learning outcome, as measured on national tests in reading and mathematics. The results are discussed in relation to how local assessment cultures with particular characteristics influence governing, accountability and trust.     

Technology-based assessment in science: Issues underlying teacher advocacy of testing policy

The theme of this article is that the development of informed teacher advocacy for new advancements in technology-based assessment is an essential requirement if such advancements are to contribute toward the systemic improvement of the quality of school science instruction. The potential for advocacy involvement by teachers is considered a natural reaction toward the increasing tendency for classroom practices to be affected by local, state, or national assessment policy initiatives. In support of such an advocacy process, this article provides an awareness of the principles of good measurement practices in conjunction with the qualitative characteristics of technology-based assessment that together are sufficient to serve as a foundation for teachers whose concerns may motivate them to raise relevant questions regarding assessment policy. Based upon such implied standards of testing practice, the article suggests key evaluative questions for teachers to ask about any forms of science assessment that would have the effect of amplifying the potential value of new technology-based forms of assessment applications to enhance ongoing classroom processes of science teaching.     

Literacy assessment that counts: mediating,interpreting and contesting translocal policy in a primary school

In Australia, as in many western education systems over the last two decades, discourses of accountability and performativity have reshaped education policy that has in turn reorganised the work of school leaders and teachers. One of the effects of this reorganisation is increased attention to the production, analysis and display of student achievement data. In this paper we examine in detail a sequence of the production and reading of literacy assessment data in a small Catholic school. Our analysis uses institutional ethnography's concept of the ‘active text’, the text as occurring in a specific place and time even as it is articulated to social relations beyond its immediate context. Through this process we learn from those involved how their everyday work brings into being formalised, textually authorised processes in a local site that ensure the school meets accountability requirements, while enabling teachers to resist standardisation of literacy teaching and assessment.     

Educational assessment policy and practice: a matter of ethics

Long-standing concerns within the field of educational assessment consider the impact of assessment policy and practice as matters of equity, inequality and social justice. Yet educational assessment policy and practice continues to have powerful social consequences for key users such as children and young people. This paper re-positions these consequences as a matter of ethics. It uses the work of Messick to frame how ethical matters extend beyond test instruments into the realm of uses and impact. A case study of the 11+ school transfer system in Northern Ireland is presented to illustrate ethical dilemmas emerging as a consequence of actions and decisions of using assessment systems for particular purposes. In looking forward to how we might attend to ethical matters in assessment policy and practice, a consideration of a children’s rights approach is outlined that may provide a moral and legal framework for action.     

The International Education Index: A global approach to education policy analysis,performance and sustainable development

Before the COVID-19 pandemic, the world struggled to address growing educational inequalities and fulfil the commitment to Sustainable Development Goal 4, which seeks to ensure inclusive and equitable quality education and promote lifelong learning opportunities for all. The pandemic has exacerbated these inequalities and changed how education functions, moving to online and hybrid methods. The challenges in global education highlighted and worsened by the pandemic make it necessary to re-evaluate education systems and the policies in place to support access, quality and equal opportunity. This article focuses on analysing education policies at a national level. It tests a pilot policy analysis tool, the International Education Index (IEI), developed as a starting point to begin this reconsideration and create an accessible and comprehensive way to evaluate national education systems to inform decision-making and policies in the new context. This research uses Ireland and Northern Ireland to test the IEI pilot tool. The IEI consists of 54 questions across nine indicators, including institutional frameworks, education strategies, digital skills and infrastructure, twenty-first century skills, access to basic social services, adherence to international standards, legal frameworks, data gathering and availability and international partnerships. Countries can score 108 points to be categorised as having developed, emerging or nascent national education systems. Ireland scored 94 and Northern Ireland 81, indicating that they have developed national education systems.     

Investigating the dynamics of formative assessment: relationships between teacher knowledge,assessment practice and learning

This exploratory study of elementary school science examines questions central to policy, practice and research on formative assessment: What is the quality of teachers’ content-pedagogical and assessment knowledge? What is the relationship between teacher knowledge and assessment practice? What is the relationship between teacher knowledge, assessment practice and student learning? Drawing on multiple measures, hierarchical linear modelling and path analysis, results suggest that despite weaknesses in teachers’ content-pedagogical and assessment knowledge, teachers’ formative assessment practices are positively related to student learning. Relationships between teachers’ knowledge and assessment practices are mixed. Findings underscore both the potential and challenge of bringing effective formative practice to fruition as well as the need for continued research.     

Whose knowledge,whose values? the contribution of local knowledge to education policy processes: a case study of research development initiatives in the small state of Saint Lucia

This paper draws upon a case study of education in the small Caribbean state of Saint Lucia (population 154,000) to examine how local knowledge and values can influence the education policy process. It is argued that recent research development initiatives have strengthened the ability of Saint Lucia to mediate international education agendas to suit its distinctive ‘social ecology’ and circumstances as a small state. Saint Lucians, it is argued, are reconceptualizing educational research in ways that better reflect the society's Kwéyòl‐majority culture, oral traditions and postcolonial context. The boundaries of educational research are therefore being stretched to incorporate local knowledge and values and multiple modes of meaning‐making. The paper argues that increased recognition of the epistemological issues and cultural values that are at the heart of efforts to strengthen research capacity can play a crucial role in democratizing the education policy process and educational research itself.     

Federal policy and the teacher labor market: exploring the effects of NCLB school accountability on teacher turnover

The media suggest that accountability pressure increases teacher stress and drives teachers away from teaching, resulting in teachers leaving disadvantaged schools that serve larger proportions of poor and minority students. However, no prior work has systematically examined the changes in the national trends of teacher turnover in response to No Child Left Behind (NCLB) school accountability. Drawing on nationally representative samples of schools and teachers from the Schools and Staffing Surveys and Teacher Follow-Up Surveys from 1993–2009, this study applied difference-in-differences approaches to examine the effects of NCLB on teacher turnover. We found a weak increase in the average rate of teachers transferring involuntarily to other schools following school-initiated separations, particularly in disadvantaged schools that served larger proportions of poor and minority students. We also observed that NCLB reduced the involuntary attrition rates of teachers. Importantly, the policy effect is indistinguishable from zero on voluntary transfer between schools or voluntary leaving the teaching profession.     

Too little and too much trust: performance measurement in Australian higher education

A striking feature of contemporary Australian higher education governance is the strong emphasis on centralized, template style, metric-based, and consequential forms of performance measurement. Such emphasis is indicative of a low degree of political trust among the central authorities in Australia in the intrinsic capacity of universities and academics to do their work efficiently and effectively. At the same time, it is indicative of a deep-seated political trust in highly centralized and top–down forms of performance measurement, and their capacity to effectively control and coordinate the work of universities and academics. In this article, we argue with regard to performance measurement that these patterns of trust and mistrust embody contradictory assumptions regarding the agency and motivations of universities and academics and prevent adequately coming to terms with the unintended effects that the current performance measurement regime has on universities and the academics working in them.     

An exploration of the interaction between global education policy orthodoxies and national education practices in Cambodia,illuminated through the voices of local teacher educators

Abstract

This research is based on a multi-disciplinary and multi-levelled analysis of evidence to present the case that education reform needs to be contextualised far more widely than is currently practised. It focuses on the voices of Cambodian local teacher trainers through interviews over a five-year period. Interview data is triangulated against academic sources, national policy documents and classroom observations. The research reveals how notions of globalisation and knowledge economy have led to education being driven by measurable outcomes resulting in simplified educational policies that have a negative impact on teacher development. By allowing the Khmer teacher trainers to articulate their opinions on a wide range of themes, the interviews provide evidence to support historical and cultural theories proposed by academics working in the social sciences in Cambodia. This approach identifies that a deeper contextual analysis should enable teacher development programmes to be more closely aligned to national contexts.     

The policy discourse of networking and its effect on school autonomy: a Foucauldian interpretation

Policy discourse officially operates to distinctly influence public perception in an irrevocable and normalising manner. In a Maltese educational scenario of gradual decentralisation and increased accountability, I explore the ‘effects’ of both the global and the local policy discourse of networks and networking on the practising leaders, in addition to their reaction to the policy document mandating these multi-site school collaboratives, with a particular interest on their imposed nature and how this reform impinged on individual school autonomy. This research adopts a case study methodology, with data collected through in-depth, semi-structured interviews; participant observation; and documentary analysis, interpreted via a Foucauldian theoretical framework through narrative analysis. The findings reveal an inherent tension among autonomy, centralisation, and decentralisation both within the policy discourse and the unfolding network leadership dynamics. This paper has particular philosophical implications for educational policy, practice, and theory in an educational scenario of school policy globalisation.     

Peer assessment in a project-based engineering course: comparing between on-campus and online learning environments

As the number of participants in online distance learning courses increases, peer assessment is becoming a popular strategy for evaluating open assignments and for breaking the social isolation surrounding distance education. Yet, the quality and characteristics of peer assessment in massive online courses has received little attention. Hence, this study was set to examine peer feedback quality and grading accuracy in a project-based course. The study applied a sequential exploratory mixed methods design. It included 339 participants who studied the same engineering course, but in three different modes: on-campus (n = 77), small private online course (n = 110), and massive open online course (MOOC) (n = 152). Content analysis of feedback comments identified four categories: reinforcement, statement, verification and elaboration, arranged in an ascending scale of cognitive ability. The findings indicated that the MOOC participants provided more feedback comments and volunteered to assess more projects than their counterparts did. However, the on-campus students provided higher quality feedback and their peer grading was better correlated with the grades assigned by the teaching assistants.     

The potential formative benefits of portfolio assessment in second and foreign language writing contexts: A review of the literature

This article identifies the potential formative benefits of using portfolio assessment (PA) in second/foreign language classrooms by reviewing the up-to-date research. Three methods were used to search for relevant research: database search, manual search, and citation search. According to the literature, PA has potential benefits due to its process-oriented, authentic, integrated, interactive, and learner-centered characteristics. PA leads to increased motivation, learner autonomy, and improved writing performance. However, not many studies include observation as a method of collecting data. Through observation a researcher can investigate how the benefits of PA are acted upon and negotiated. There is a need for more empirical research in the juxtaposition of PA and formative assessment, particularly in primary and secondary education.     

Formative assessment: the role of participatory action research in blending policy and practice in Uganda

This paper documents teachers’ assessment practices and pupils’ learning to read and write in large classes. To gain insights into the assessment practices and pupils’ learning, the principles of Vygotsky’s zone of proximal development and participatory action research (PAR) design were adopted. Collaboratively, teachers reflected on their assessment practices, developed, implemented and monitored the assessment practices. Through observation, in-depth interview and review of exercise books, data were obtained. We argue that teachers’ involvement in PAR has contributed to changing assessment practices in large classes and in improving pupils’ competencies in reading and writing. On the other hand to promote better assessment practices, we advocate for reduced number of pupils in a class and teachers’ workload to enable them offer more support to the pupils during assessment of reading and writing.