Evidence-based decision making or a tunnel vision effect? TIMSS,problem definition and policy change in Israeli mathematics education

ABSTRACT

This study examines the influence of the Trends in International Mathematics and Science Study (TIMSS) 1999 results on mathematics education policy in Israel. Analyzing various documents, the study shows how these results transformed policymakers’ discourse regarding mathematics education. In order to achieve improvement in Israel’s ranking, the middle school mathematics curriculum was aligned with the TIMSS test. Following this change, the scores of Israeli students rose considerably in TIMSS 2011. Parallel results from Programme for International Student Assessment (PISA) and Israeli internal standardized tests, however, do not show comparable improvement in mathematics achievement. High school matriculation data for the same period of time show a decline in the percentage of students who took the most advanced mathematics courses. Furthermore, none of these data sources show any noticeable reduction in social inequality in mathematics achievement. On the basis of theoretical insights from political science, it is argued that international tests influence the way educational problems are framed and defined. Thus, instead of enriching educational decision-making processes, these tests can create a tunnel vision effect that restricts policymakers’ attention to ‘problems’ and ‘solutions’ defined by the tests themselves.     

Educational accountability tests,social and legal inclusion approaches to discrimination for students with disability: a national case study from Australia

The UN Convention on the Rights of Persons with Disability promotes equal and full participation by children in education. Equity of educational access for all students, including students with disability, free from discrimination, is the first-stated national goal of Australian education. Australian federal disability discrimination law, the Disability Discrimination Act 1992, follows the Convention, with the federal Disability Standards for Education 2005 enacting specific requirements for education. This article discusses equity of processes for inclusion of students with disability in Australian educational accountability testing, including international tests in which many countries participate. The conclusion drawn is that equitable inclusion of students with disability in current Australian educational accountability testing is not occurring from a social perspective and is not in principle compliant with law. However, given the reluctance of courts to intervene in education matters and the uncertainty of an outcome in any court consideration, the discussion shows that equitable inclusion in accountability systems is available through policy change rather than expensive, and possibly unsuccessful, legal challenges.     

Policy impact of PISA on mathematics education: the case of Norway

This article addresses the policy implications of participation in international large-scale assessments (ILSAs), particularly the Programme for International Student Assessment (PISA), and the ways in which such implications might influence mathematics education. Taking Norway as a special case, this discussion focuses on insights into teaching, learning and assessment practices that can be inferred from the PISA study, and how participation in ILSAs has contributed to educational policy and even changed policymakers’ perspectives on schools, teachers and students. Following publication of the PISA 2000 results, Norway experienced a ‘PISA shock’, leading to the implementation of a national quality assessment system and national tests. In addition, changes were made to the mathematics curriculum for compulsory school and to mathematics teacher education. More recently, public debate has focused less on rank and league tables, shifting instead to the high number of low-achieving students and the low number of high achievers. Moreover, there has been little uptake of policy advice provided by the Organisation for Economic Co-operation and Development (OECD), which focuses on strengthening accountability measures. Furthermore, although the Norwegian educational system in the past decade has undergone a decentralisation process, the educational system still follows the Nordic model, which focuses on equity and ‘education for all’. Analyses of the Norwegian case indicate that policymaking takes place in highly cultural contexts, and that international studies might be used merely to validate existing policy directions.     

Effects of extracurricular activities on postsecondary completion for students with disabilities

The authors focused on participation in extracurricular activities as a way of improving the educational outcomes of children with disabilities. Regarding students in the general population, adolescent involvement in extracurricular activities has been shown to have a positive association with school involvement and adolescent self-esteem, academic achievement, and adolescent development. Using the Educational Longitudinal Study dataset, the authors conducted logistic regression analyses to examine the relationship between participation in high school–sponsored extracurricular activities and postsecondary degree completion for students with disabilities. Findings show a statistically significant association between postsecondary degree completion for students with disabilities and extracurricular activity participation, including extent and type of extracurricular activity. Students with disabilities who participate in high school extracurricular activities may have increased likelihood of completing a postsecondary degree. Implications for policy, practice, and future research are discussed.     

Including students with disabilities in national academic assessments in Korea

In Korea, the effort to include students with disabilities in the educational accountability system has just begun. This paper reviews how Korean students with disabilities have been tested using the National Assessment of Educational Achievement (NAEA) and what issues have emerged as a result of the testing. Analysis of the 2009 and 2010 NAEA data reveals that only a small proportion of students with disabilities participated in these tests and that those who did participate showed low academic achievement compared with general education students. In addition, even though alternate assessments are conducted by special education teachers to assess the academic performance of students with disabilities, national academic assessments in Korea do not include a system for alternate assessment. Based on findings of the analysis, several suggestions are made for improvement in Korea??s educational accountability system for students with disabilities.     

In-service training programmes for mathematics teachers nested in transnational policy discourses

Results in mathematics on international knowledge surveys like Programme for International Student Assessment and Trends in International Mathematics and Science Study have become one of the most important factors for the perceived success or failure of schools and even entire education systems in the policy arena. In this article, we explore the complex recontextualising processes that occur when translating educational policy into actual programmes for teachers’ education. First, the transnational education policy discourse(s) of teachers’ in-service training with a focus on mathematics will be explored. Second, we examine how this transnational discourse is recontextualised in a national policy discourse resulting in a national reform programme for in-service training of mathematics teachers in Sweden. In a third step, concrete teacher training courses in mathematics are examined. The result shows a convergence between the official policy discourse and the pedagogic recontextualising field in terms of a broad teaching repertoire and peer discussions about reflections on certain common objects of learning.     

Media representations of national and international standardized testing in the Israeli education system

This study applies discourse analysis to Israeli media coverage of national and international standardized examinations within Israel's public education system. Through systematic analysis of the topic in the two main Israeli financial publications between the years 2000 and 2013, we explore the nature and narrative of the media and compare the coverage of national and international standardized testing. We find that most of the media attention was devoted to international examinations, while national examinations were covered in a more limited yet critical way, perceived as unnecessary and even dubious. International examinations, in contrast, were described as axiomatic components of the education system. Articles on both national and international standardized testing criticize the education system, blaming teachers, the Ministry of Education, budget constraints, and marginalized populations for Israeli students’ inadequate results. We frame our analysis by alignment of the articles along global–local and also neoliberal–humanistic axes. We structure our assessment within the global–local nexus and discuss the broader implications of the role of the testing in framing the local educational public discourse.     

How not to reason with PISA data: an ironic investigation

International comparative testing, such as the OECD Programme for International Student Assessment (PISA), has considerable impact on policy-makers, the media and the general public. A central assumption underlying PISA is that global variation in students’ academic performance is attributable to national educational structures and policies. The aim of this article is to demonstrate the problematic nature of this assumption. Rather than critiquing it from the outside, we turn the tools, data and presumptions of the current discourse of international comparisons upon themselves, showing that this assumption is refuted by analysis of immigrant student test scores. Data from the PISA 2009 tests show that Chinese immigrant students in New Zealand and Australia achieve math scores that are more similar to those of students in Shanghai than to their non-immigrant Australian and New Zealand peers. Thus, cultural background appears to be more consequential for the educational attainment of Chinese immigrant students than exposure to the educational systems of Australia or New Zealand. We discuss limitations of our analysis, using them as basis to discuss the shortcomings of PISA more generally.     

Why Students Answer TIMSS Science Test Items the Way They Do

The purpose of this study was to explore how Year 8 students answered Third International Mathematics and Science Study (TIMSS) questions and whether the test questions represented the scientific understanding of these students. One hundred and seventy-seven students were tested using written test questions taken from the science test used in the Third International Mathematics and Science Study. The degree to which a sample of 38 children represented their understanding of the topics in a written test compared to the level of understanding that could be elicited by an interview is presented in this paper. In exploring student responses in the interview situation this study hoped to gain some insight into the science knowledge that students held and whether or not the test items had been able to elicit this knowledge successfully. We question the usefulness and quality of data from large-scale summative assessments on their own to represent student scientific understanding and conclude that large scale written test items, such as TIMSS, on their own are not a valid way of exploring students' understanding of scientific concepts. Considerable caution is therefore needed in exploiting the outcomes of international achievement testing when considering educational policy changes or using TIMSS data on their own to represent student understanding.     

Structural Communications: An Instructional Strategy for the Language Arts

The ranking of the United States in major international tests such as the Progress in International Reading Literacy Study (PIRLS), Trends in International Mathematics and Science Study (TIMSS), and Program for International Student Assessment (PISA) is used as the driving force and rationale for the current educational reforms in the United States. Through current reforms, the United States is preparing to “race to the top” of the international rankings. However, using international test results as the ultimate indicator of a country's educational achievement might not be an appropriate measure. Additionally, replicating some aspects of high-achieving countries’ education systems or standards may not guarantee success for U.S. public schools. In this article, four reasons are presented explaining why there is no guaranteed success in comparing the United States to other countries’ educational standards and to their achievement in international tests. In order to successfully implement reforms in U.S. public schools, educational leaders and reformers should carefully consider the four points discussed in this article and use tests to inform their reforms, rather than to create uniformity.     

Standardized Testing + High-Stakes Decisions = Educational Inequity

Changes in assessment policy have increased standardized testing at provincial, national, and international levels, introduced testing at more grade levels, increased the reporting of test results, and attached more significance to those results. Advocates claim that testing will result in greater accountability in education. The research demonstrates that standardized testing has a negative impact on students, perpetuating and intensifying educational inequity through test bias and the misuse of test scores. Test results are increasingly being used to analyse policy, program, school, and teacher success, and they are being inappropriately used as "educational gatekeepers" to make important decisions about students, teachers, schools, and school systems. This paper focuses on how standardized testing is becoming the mechanism that facilitates many questionable education practices that contribute to educational inequity.     

Globalisation and the politics of international tests: the case of Israel

Comparative international tests of students' achievements have become increasingly popular over the past decade. An outcome of this widespread practice is the tendency of various countries to evaluate their national status according to their pupils' international ranking in such achievement tests, partly due to the common belief that high ranking is an indication of the nation's future ability to compete in the global market. Using PISA 2000 and PISA 2006 data, the present study aims to show that the low achievements of Israeli pupils were to be expected, and that their average scores are a reflection of the economic and demographic characteristics of their country. Our findings suggest that the public debate in Israel on school achievements has been magnified out of all proportion. We also discuss the educational policy which derives proposals for educational reform from a conception of globalisation of education. This policy focuses on the setting of standards for educational achievements aimed at improving pupils' place on the international scale. Finally, we claim that the focus on the international grading of educational achievements deflects public attention from an understanding of the significance of inequalities within the country.     

Special educational needs,disability and school accountability: an international perspective

Standards-based accountability approaches in education are increasingly being adopted in many industrialised countries. This tends to involve the scrutiny of student performance in national academic tests and then holding schools accountable for subsequent progress. International assessments such as Programme for International Student Assessment (PISA) are not accountability measures in themselves, but are often utilised in a similar way and national policies are judged against performance in these tests. Drawing upon the examples of PISA and national assessments in the USA, this paper explores how students with special educational needs and disability (SEND) are included in, and served by, these assessment policies. The analysis suggests that while these approaches are inclusive in their intent (most obviously through efforts to include students with SEND in testing) they may poorly represent the progress of these students. It is argued that more equitable inclusion of students with SEND must find a balance between accountability measures which measure performance in traditional attainment-focused tests and other relevant assessments which are meaningful to the given student and/or SEND sub-group.     

Neoliberal versus social justice reforms in education policy and practice: discourses,politics and disability rights in education

This article uses critical discourse analysis in order to discuss the equity and social justice implications of an envisaged education reform agenda in Cyprus, as articulated by two consultation reports commissioned by the World Bank. The reports highlight, inter alia, the imperative to improve teaching and enhance accountability regimes with regard to students’ learning. Selected extracts from these documents are analyzed in order to highlight the absence of a social justice discourse in the rhetoric of educational reforms, despite the alleged centrality of a social justice discourse in official policy. The reports fail to include issues of social justice and learner diversity in discussing the necessity to strengthen the existing teacher policy framework and to mobilize structural educational reforms. This omission is indicative of the neoliberal imperatives that drive the envisaged education policy reforms as well as the low priority attributed to issues of equity and learner diversity, with particular reference to students designated as having special educational needs and/or disabilities.     

Standards-based external exams and students’ science-related career expectations: an international perspective

Students’ science-related career expectations are important for predicting their future science, technology, engineering, and mathematics (STEM)-related educational and occupational attainments. This study examines the degree to which standards-based external examinations are associated with a student’s propensity for pursuing science-related professional occupations. The science-related fields included in the analysis are mathematics, physical and life science, and engineering/computing. Three-level hierarchical generalized linear models are employed to analyse international survey and student achievement data from the Programme for International Student Assessment (PISA). The analyses show that students in national education systems that require standards-based external examinations have lower expectations for science-related professional careers than students in the systems that do not use such examinations. This negative association remained constant by gender as well as across levels of science performance. From an educational policy point of view, the results suggest the negative consequence of standards-based external exams in fostering students’ interests in pursuing science-related careers.     

Change of Emphasis in the Mathematics Assessment in Hungary

International survey data showed that Hungarian students performed well in both mathematics and science in the past. Since 1991 achievement in these 2 areas has declined, and this was most clearly shown in Third International Mathematics and Science Study (TIMSS). Two possible reasons for this phenomenon are investigated here: as a consequence of recent political and economical changes; due to the conservative structure of math and science teaching which differ from the international trend. While the achievement of Hungarian students was high on items requiring awareness of the traditional disciplines, it was lower on literacy and life-skill items and topics such as environment issues, measurement, data representation and interpretation, and so forth. Following international trends, the national monitoring surveys have shown a shift from the “academic” approach to the “real-life” application of mathematics. The paper presents both the new approach and the findings from the most recent national survey.     

Negative school factors and their influence on math and science achievement in TIMSS 2003

The aim of the present study was to conduct an analysis of TIMSS (Trends in International Mathematics and Science Study) 2003 database and to determine how negative school factors, such as aggression, are associated to the mathematical and science achievement of students. The analyses were conducted separately for national and international data. National analyses for Slovenia show significant associations between math and science achievement and the experience of aggressive behaviour. Students who experienced aggressive behaviour scored lower in math and science, both in the fourth and in the eighth grade. The results of the regression analysis show that negative factors, such as aggressive behaviour, are good predictors of educational achievement in Slovenia. International analyses for the selected countries (high‐ and low‐achieving countries from the whole TIMSS population) confirm that this type of finding is culturally impartial as well as valid for the level of achievement both in math and in science.     

International education in systemic educational reforms: the Chinese case and lessons learned from an international perspective

International research findings and country experiences prove that, in coping with the tension between the global and the local in education, three major forces contribute to the success of international reform: firstly, the national public authorities in protecting education as a common good through political commitment and policy actions; secondly, the local community, in responding to the diversifying educational needs through active participation and initiatives for innovation; and thirdly, the international community, through technical, professional as well as financial assistance to member countries for desired systemic changes. This paper examines the roles of international education in facilitating systemic reforms, and explores strategies in promoting international education for educational change, citing evidence-based cases to examine the interaction between international education and systemic educational reforms.     

Using Adjustments to Support the Learning and Assessment Needs of Students with Disabilities: Macau and Mainland China Teachers’ Report

Adjustments are considered necessary for students with disabilities to be fully included in classroom instruction, classroom assessment and external accountability tests. The 67 item Checklist of Learning and Assessment Adjustments for Students (CLAAS), translated for the Chinese community, was used by 74 teachers from Macau and Mainland China to document their application of adjustments for 319 students with special educational needs across these three settings. Results indicated consistently large gaps between adjustment use in classrooms compared with national tests, with the allowable adjustments for public testing reportedly used very little. Findings also provided evidence for the content validity of the checklist for teachers of students with disabilities in China and its utility in documenting applied adjustments. This study also indicates the potential for CLAAS to provide teachers across China with a comprehensive list of adjustments, to reflect on and review adjustment decision-making and assessment protocols for all students.     

Large-scale academic achievement testing of deaf and hard-of-hearing students: past, present, and future

The first large-scale, nationwide academic achievement testing program using Stanford Achievement Test (Stanford) for deaf and hard-of-hearing children in the United States started in 1969. Over the past three decades, the Stanford has served as a benchmark in the field of deaf education for assessing student academic achievement. However, the validity and reliability of using the Stanford for this special student population still require extensive scrutiny. Recent shifts in educational policy environment, which require that schools enable all children to achieve proficiency through accountability testing, warrants a close examination of the adequacy and relevance of the current large-scale testing of deaf and hard-of-hearing students. This study has three objectives: (a) it will summarize the historical data over the last three decades to indicate trends in academic achievement for this special population, (b) it will analyze the current federal laws and regulations related to educational testing and special education, thereby identifying gaps between policy and practice in the field, especially identifying the limitations of current testing programs in assessing what deaf and hard-of-hearing students know, and (c) it will offer some insights and suggestions for future testing programs for deaf and hard-of-hearing students.