The ambiguous influence of high-stakes testing on science teaching in Sweden

ABSTRACT

Tests convey messages about what to teach and how to assess. Both of these dimensions may either broaden or become more uniform and narrow as a consequence of high-stakes testing. This study aimed to investigate how Swedish science teachers were influenced by national, high-stakes testing in science, specifically focusing on instances where teachers’ pedagogical practices were broadened and/or narrowed. The research design is qualitative thematic analysis of focus group data, from group discussions with Swedish science teachers. The total sample consists of six teachers, who participated in 12 focus group discussion during three consecutive years. Findings suggest that the national tests influence teachers' pedagogical practice by being used as a substitute for the national curriculum. Since the teachers do not want their students to fail the tests, they implement new content that is introduced by the tests and thereby broaden their existing practice. However, when this new content is not seen as a legitimate part of teachers' established teaching traditions, the interpretation and implementation of this content may replicate the operationalisations made by the test developers, even though these operationalisations are restricted by demands for standardisation and reliable scoring. Consequently, the tests simultaneously broaden and narrow teachers’ pedagogical practices.     

The impact of high-stakes testing on curriculum and pedagogy: a teacher perspective from Australia

Debates continue about how high-stakes testing regimes influence schools at all levels: their impact on teaching practices, distribution of resources and curriculum provision, and whether they achieve the intended increases in student achievement in targeted areas. In 2008, the Australian government introduced a national testing scheme, the National Assessment Program – Literacy and Numeracy (NAPLAN), in which all Australian students in Years 3, 5, 7 and 9 are required to participate, and a national website, MySchool, was established in 2010 to publish the results of all schools. This paper reports on the first national study of educators’ views on the impact of NAPLAN on Australian schools and students. Over 8000 educators from all states and territories participated in the study, and the findings indicate that the testing regime is leading to a reduction in time spent on other curriculum areas and adjustment of pedagogical practice and curriculum content to mirror the tests. The findings suggest that the modification of teaching and curricular practices is in response to concerns regarding the use and reporting of NAPLAN data and the potential impact on schools. This confirms findings of researchers in other countries on the capacity of high-stakes regimes to distort teaching practices, constrain the curriculum and narrow students’ educational experiences.     

The impact of a universal class-size reduction policy: Evidence from Florida's statewide mandate

Class-size reduction (CSR) mandates presuppose that resources provided to reduce class size will have a larger impact on student outcomes than resources that districts can spend as they see fit. I estimate the impact of Florida's statewide CSR policy by comparing the deviations from prior achievement trends in districts that were required to reduce class size to deviations from prior trends in districts that received equivalent resources but were not required to reduce class size. I use the same comparative interrupted time series design to compare schools that were differentially affected by the policy (in terms of whether they had to reduce class size) but that did not receive equal additional resources. The results from both the district- and school-level analyses indicate that mandated CSR in Florida had little, if any, effect on student achievement.     

The impact of Covid-19 on student achievement: Evidence from a recent meta-analysis

This work attempts to synthetize existing research about the impact of Covid-19 school closure on student achievement. It extends previous systematic reviews and meta-analyses by (a) using a more balanced sample in terms of country composition, (b) considering new moderators (type of data and research design), and (c) including studies on tertiary education students in addition to primary and secondary education students. Our meta-analysis findings show that the pandemic had, on average, a detrimental effect on learning. The magnitude of this learning deficit (about 0.19 standard deviations of student achievement) appears to be roughly comparable to that suffered by students who have experienced a significant disruption in their schooling due to a major natural disaster (e.g., Hurricane Katrina). Students are also found to have lost more ground in math/science than in other subjects. Additionally, one year or more after the first lockdown, students seem to have been unable to catch up on unfinished learning from the pandemic. This result suggests that more efforts should be made to ensure students recover their missed learning in order to avoid negative long-term consequences for them and society.     

The impact of science curricula on student views about the nature of science

This review of the literature focused on three decades of research related to precollege student understandings about the nature of science. Various interpretations of what aspects characterize the nature of science were examined, revealing an agreement among scientists, science educators, and those involved in policy-making arenas that the nature of science is multifaceted and an important component of scientific literacy. A summary of the research regarding the adequacies of student conceptions about the nature of science revealed inconsistent results. Although the majority of studies show that student understandings are less than desirable, there is research that indicates that student conceptions are acceptable. Research on the impact of instructional materials and techniques on student understandings was also reviewed. The effects of language in science instruction, the content emphasis of instructional materials, integrated science curricula, and instruction in general were curricular variables found to have a negative impact on student understandings about the nature of science. Empirical evidence about the success of innovative instructional materials and techniques designed to facilitate more adequate understandings of the nature of science is needed.     

The effect of same-gender teacher assignment on student achievement in the elementary and secondary grades: Evidence from panel data

We utilize information from a rich administrative panel dataset following the universe of test-taking public school students in Florida over a period of five years to estimate the relationship between same-gender teacher assignment and student achievement. We estimate how a student's achievement changes as he/she is assigned to teachers of different genders throughout his/her academic career, holding constant both observed and unobserved factors related to academic outcomes. We also provide estimates from models that evaluate how the relative performance of male and female student assigned to the same teacher or in the same classroom relates to the gender of the teacher. We find no statistically distinguishable relationship between same-gender teacher assignments and student math or reading achievement in elementary school. We find a statistically significant relationship between being assigned to a female teacher and student achievement in middle and high school, however the magnitude of the effect is small.     

Uneasy writing: The defining moments of high-stakes literacy testing

The authors report on an exploratory study that investigated Grade 10 students’ responses to a large-scale, high-stakes literacy test, the results of which determine high school graduation in Ontario, Canada. Through focus-group and individual interviews, the authors found that students perceived the test as evaluating their English skills rather than their cross-curricular literacy as is the stated purpose by the test administrators. Furthermore, the test contributed to students’ narrow definition of writing. Rather than promoting writing as a powerful tool for thinking and expressing ideas, students understood writing to be a fill-in-the-blanks response to particular genres. The authors conclude by pointing out the power of such tests to define unintended learning in schools.     

The impact of grade ceilings on student grades and course evaluations: Evidence from a policy change

This paper analyzes the effects of a grade ceiling policy on grade distributions and course evaluations. Results show that the effects vary based upon the level of the grade ceiling. A ceiling set at 2.8 decreased overall grade point average (GPA) by reducing the number of As and Bs and increasing the number of lower grades given. This low ceiling also increased the number of withdrawals and significantly lowered course evaluations. A ceiling set at 3.2 decreased overall GPA by reducing the number of As and increasing the number of Bs given, but the effects on course evaluations were smaller in magnitude and insignificant.     

The impact of labor market work and educational tracking on student educational outcomes: Evidence from Taiwan

This study empirically investigates how working while enrolled in high school affects educational outcomes, while accounting for self-selected educational tracking. Using a longitudinal survey of Taiwanese youth and exploiting the inter-zip-code geographic variations in youth-preferred industries, we find a negative effect of school-year work on educational achievement, and the negative marginal impact is much stronger for academic-track than for vocational-track students. An exogenous increase in school-year hours worked of 10 hours per week lowers college entrance scores by a 0.117 (0.083) standard deviation for academic-track (vocational-track) students. The negative impact of school-year work tends to be overstated if the endogeneity arising from educational tracking is not acknowledged—with upward biases as large as 16% and 30% for academic- and vocational-track students, respectively. Among subjects, math scores suffer most from working during the school year.     

The effect of technology funding on school-level student proficiency

This study presents new evidence on the effect of technology funding on school-level student proficiency. I exploit exogenous variation in school-level technology funding using the California Education Technology K-12 Voucher Program. The program provided eligible schools with technology vouchers to purchase qualifying hardware and software products and services. Using a regression discontinuity difference-in-difference design and data on voucher eligibility, voucher use, and school-level student proficiency, I find that voucher eligibility had no significant impact on school-level student proficiency, while voucher use had positive impacts on school-level student proficiency. The voucher use results are driven entirely by schools using the voucher funds for technology resources, but reallocating dollars initially earmarked for technology to other school inputs.     

Group differences in test-taking behaviour: an example from a high-stakes testing program

This study investigated whether different groups of test-takers vary in their reported test-taking behaviour in a high-stakes test situation. A between-group design (N = 1129) was used to examine whether high and low achievers, as well as females and males, differ in their use of test-taking strategies, and in level of reported test anxiety and motivation. The results showed differences between high and low achievers on a number of test-taking strategies, where high achievers reported using successful strategies to a higher extent. There were also gender differences: females, for example, reported using random guessing to a higher extent than males. Further, low achievers, especially females, reported significantly higher levels of test anxiety than high achievers, and high achievers reported slightly higher levels of motivation when compared to low achievers. To conclude, test-taking behaviour might bring additional variance to test scores, whether or not this is irrelevant variance is discussed.     

The impact of Geospatial Inquiry lessons on student interest in science and technology careers

Advances in online geospatial technologies (GST) have expanded access to K-12 classrooms which has implications for the support teachers require to effectively integrate GSTs to promote learning. Previous studies have shown the impact of GST-integrated lessons on student engagement, spatial thinking skills, and/or content knowledge; however, most of these studies have been small in scope and scale and frequently focus on the affordances of the technology, without addressing the context of the implementation and student characteristics for whom GST is most impactful. We attempt to address some of these gaps. Our program scaled an effective GST-focused professional learning and development program to a national audience through a facilitator development model. This paper explores the student characteristics and lesson factors that resulted in student interest in science and technology and careers in those fields. After teaching a Geospatial Inquiry lesson created during a teacher workshop, teachers (n = 82) submitted the lessons and surveys on the implementation of Geospatial Inquiry lessons. The implementation surveys and lessons were scored for alignment to the principles of high-quality Geospatial Inquiry. Students (n = 1924) completed a post-lesson retrospective survey and indicated the extent to which their perceptions and attitudes toward science and technology changed because of the lesson. Results indicate that teacher GST performance is associated with increases in student outcomes. Students with previous exposure to science activities were more likely to have increased interest and excitement in science and careers in science but decreased interest in technology careers. Students who had previous exposure to technology activities had increased interest and excitement in technology and careers in technology but decreased interest in science careers. Geospatial Inquiry lessons also had a significant impact on students who are traditionally underrepresented in STEM fields. After participating in the lessons, students who identify as female reported higher engagement and interest in science and higher interest in science careers. Students who identified as Black or Hispanic also reported higher interest and excitement in science and technology, and students who identified as Black reported marginally higher interest in science careers.     

The influence of high-stakes testing on teacher self-efficacy and job-related stress

In the United States, teachers’ job-related stress and self-efficacy levels across all grades are influenced in some manner by the demands of high-stakes testing. This sequential mixed-methods study aimed at examining the dynamics among assigned subject matter, teacher job-related stress, and teacher self-efficacy in a large south-eastern Texas school district. Quantitative data were collected via the High Stakes Testing on Self-Efficacy and Teacher Stress Survey from a convenience sample of 145 teachers, while qualitative data were gathered during focus-group sessions at the elementary, middle and high school levels. Findings indicated that assigned subject matter did not influence teacher self-efficacy, but did influence teachers’ job-related stress. Additionally, a relationship was found to exist between teacher job-related stress and teacher self-efficacy. Teachers reported lack of time, modifications to curriculum, and testing of special education populations as contributions to their job-related stress, and school leadership and educational triage as impacting their teacher self-efficacy.     

The impact of an integrated approach to science and literacy in elementary school classrooms

This study investigates the efficacy of an integrated science and literacy approach at the upper‐elementary level. Teachers in 94 fourth grade classrooms in one Southern state participated. Half of the teachers taught the treatment unit, an integrated science–literacy unit on light and energy designed using a curriculum model that engages students in reading text, writing notes and reports, conducting firsthand investigations, and frequent discussion of key concepts and processes to acquire inquiry skills and knowledge about science concepts, while the other half of the teachers taught a content‐comparable science‐only unit on light and energy (using materials provided by their districts) and provided their regular literacy instruction. Students in the treatment group made significantly greater gains on measures of science understanding, science vocabulary, and science writing. Students in both groups made comparable gains in science reading comprehension. © 2012 Wiley Periodicals, Inc. J Res Sci Teach 49: 631–658, 2012     

Differential effects of three professional development models on teacher knowledge and student achievement in elementary science

To identify links among professional development, teacher knowledge, practice, and student achievement, researchers have called for study designs that allow causal inferences and that examine relationships among features of interventions and multiple outcomes. In a randomized experiment implemented in six states with over 270 elementary teachers and 7,000 students, this project compared three related but systematically varied teacher interventions—Teaching Cases, Looking at Student Work, and Metacognitive Analysis—along with no‐treatment controls. The three courses contained identical science content components, but differed in the ways they incorporated analysis of learner thinking and of teaching, making it possible to measure effects of these features on teacher and student outcomes. Interventions were delivered by staff developers trained to lead the teacher courses in their regions. Each course improved teachers' and students' scores on selected‐response science tests well beyond those of controls, and effects were maintained a year later. Student achievement also improved significantly for English language learners in both the study year and follow‐up, and treatment effects did not differ based on sex or race/ethnicity. However, only Teaching Cases and Looking at Student Work courses improved the accuracy and completeness of students' written justifications of test answers in the follow‐up, and only Teaching Cases had sustained effects on teachers' written justifications. Thus, the content component in common across the three courses had powerful effects on teachers' and students' ability to choose correct test answers, but their ability to explain why answers were correct only improved when the professional development incorporated analysis of student conceptual understandings and implications for instruction; metacognitive analysis of teachers' own learning did not improve student justifications either year. Findings suggest investing in professional development that integrates content learning with analysis of student learning and teaching rather than advanced content or teacher metacognition alone. © 2012 Wiley Periodicals, Inc. J Res Sci Teach 49: 333–362, 2012     

An investigation of the gender differential performance on a high-stakes language proficiency test in Iran

There has been a growing consensus among the educational measurement experts and psychometricians that test taker characteristics may unduly affect the performance on tests. This may lead to construct-irrelevant variance in the scores and thus render the test biased. Hence, it is incumbent on test developers and users alike to provide evidence that their tests are free of such bias. The present study exploited generalizability theory to examine the presence of gender differential performance on a high-stakes language proficiency test, the University of Tehran English Proficiency Test. An analysis of the performance of 2,343 examinees who had taken the test in 2009 indicated that the relative contributions of different facets to score variance were almost uniform across the gender groups. Further, there is no significant interaction between items and persons, indicating that the relative standings of the persons were uniform across all items. The lambda reliability coefficients were also uniformly high. All in all, the study provides evidence that the test is free of gender bias and enjoys a high level of dependability.