On the value of data-based decision making in education: The evidence from six intervention studies

Although some studies have investigated the impact of data-based decision making (DBDM) on student achievement, the overall findings are not straightforward, because of the studies’ methodological flaws and their mixed results. This article first presents a breakdown of the DBDM concept as applied in the Dutch context. Next, it explains the theoretical foundations of DBDM in feedback and goal-setting theory and then discusses various factors influencing DBDM effectiveness. The results of six Dutch DBDM interventions with an explicit focus on student achievement effects of DBDM and with strong research designs are then presented. Significant positive effects on student performance as measured by means of standardized tests are reported for four interventions. The interpretation of student progress data from student monitoring systems does not seem to be problematic for teachers, if they are deliberately trained for it, but teachers do find it difficult to translate student progress data into tailor-made instruction.     

Factors promoting and hindering data-based decision making in schools

Although data-based decision making can lead to improved student achievement, data are often not used effectively in schools. This paper therefore focuses on conditions for effective data use. We studied the extent to which school organizational characteristics, data characteristics, user characteristics, and collaboration influenced data use for (1) accountability, (2) school development, and (3) instruction. The results of our hierarchical linear modeling (HLM) analysis from this large-scale quantitative study (N = 1073) show that, on average, teachers appear to score relatively high on data use for accountability and school development. Regarding instruction, however, several data sources are used only on a yearly basis. Among the factors investigated, school organizational characteristics and collaboration have the greatest influence on teachers’ data use in schools.     

The complexity of data-based decision making: An introduction to the special issue

This special issue explores the complexity and interconnectedness of the many components of data-based decision making. The selection of papers represents many countries (i.e., Belgium, The Netherlands, New Zealand, Norway, and the United States), theories, methods, and foci. All the papers seek to explicate how data are used at the different level of the system, ranging from students, teachers, schools, and districts. Together these papers offer a view of the current data-based decision making landscape, including in- and pre-service professional development, district and school organizational capacity, the data use process (from goal setting to collaborative instructional decision making), and effects on student achievement. The intent of the special issue is to stimulate future work in terms of impact on research, theory, policy, and practice.     

School characteristics influencing the implementation of a data-based decision making intervention

There is an increasing global emphasis on using data for decision making, with a growing body of research on interventions aimed at implementing and sustaining data-based decision making (DBDM) in schools. Yet, little is known about the school features that facilitate or hinder the implementation of DBDM. Based on a literature review, the authors identified 12 potentially critical factors for the successful implementation of DBDM. Following an intensive DBDM intervention, 16 schools’ characteristics were studied and related to the success of DBDM implementation after participating in an intensive DBDM intervention for 2 subsequent academic years. Strong instructional leadership, maximum exposure to the intervention, standardization of work processes, as well as staff continuity and a strong academic coach appear to be strongly related to implementation success. The remaining school context features and the culture and structure of the schools were not associated with the success of DBDM implementation.     

Misconceptions about data-based decision making in education: An exploration of the literature

Research on data-based decision making has proliferated around the world, fueled by policy recommendations and the diverse data that are now available to educators to inform their practice. Yet, many misconceptions and concerns have been raised by researchers and practitioners. To better understand the issues, a session was convened at AERA’s annual convention in 2018, followed by an analysis of the literature based on misconceptions that emerged. This commentary is an outgrowth of that exploration by providing research, theoretical, and practical evidence to dispel some of the misconceptions. Our objective is to survey and synthesize the landscape of the data-based decision making literature to address the identified misconceptions and then to serve as a stimulus to changes in policy and practice as well as a roadmap for a research agenda.     

Data-based decision making for teacher and student learning: a psychological perspective on the role of the teacher

Data-based decision-making has the potential to increase student achievement results. Data-based decision-making can be defined as teachers’ systematic analysis of data sources in order to study and adapt their educational practices for the purpose of maximizing learning results. Teachers must apply the findings from their data use to their personal teaching activities. Therefore, data-based decision-making may be influenced by individual teachers' psychological characteristics. The present study aimed to explore which psychological factors contribute to teachers’ data use in a Dutch primary school context. A questionnaire-based quantitative methodology was employed. We included the following psychological constructs: affective and instrumental attitudes, perceived control, social norms, self-efficacy, collective efficacy, and intentions regarding data use. Results of the path analysis showed that perceived control, instrumental attitude, and intention regarding data use all significantly influenced data use. Additionally, intention was found to be a mediator of the relation between affective attitude and data use. Interventions aimed at data-based decision-making should take these psychological factors into account to increase teachers’ implementation of data-based decision-making for instruction and, consequently, educational quality.     

Exploring data use practices around Europe: Identifying enablers and barriers

In this article we explore what data-based decision making use looks like in schools in five different countries (United Kingdom, Germany, Poland, Lithuania and the Netherlands). We explore for what purposes data are used in these countries and what the enablers and barriers to data use are. The case study results show that schools in all five countries use data for school development, accountability, and instructional improvement. Also, the schools in the five countries struggle with the same type of problems: e.g. lack of access to high quality data, lack of professional development in using data, and a lack of collaboration around the use of data. Finally, we discuss how some enablers can turn into barriers for effective data use.     

The impact of formative assessment on student achievement: A study of the effects of changes to classroom practice after a comprehensive professional development programme

A random sample of 22 Year 4 teachers in mathematics from a middle-sized Swedish municipality participated in a teacher professional development programme in formative assessment. The content of the programme was formative assessment conceptualised as a unity of different, integrated strategies. The study examines the effects on student achievement of the changes in the teachers’ formative classroom practice that followed the professional development input. Results show that, after controlling for pretest scores, the classes in the intervention group significantly outperformed the classes in the control group in a posttest administered one school year after the end of the programme (p = 0.036, d = 0.66). The study contributes to the understanding of under-studied areas of the impact of professional development in formative assessment, and the impact of teacher practice based on formative assessment conceptualised as a unity of different formative assessment strategies.     

Effects of student participation in decision making at school. A systematic review and synthesis of empirical research

This article reviews empirical research on the effects of student participation in school decision-making processes. Out of 3102 searched citations, a total of 32 publications met the inclusion criteria. The qualitative analyses employed in this review yielded a typology of student participation, a categorisation of the diverse effects of student participation and an overview of the quality and quantity of related research (available evidence). We found moderate evidence of positive effects of student participation on life skills, self-esteem and social status, democratic skills and citizenship, student-adult relationships and school ethos. We found limited evidence of positive effects on academic achievement, facilities, rules or policies, and health; we also found low levels of evidence of negative effects. We did not find sufficient research on the positive effects of student participation on peer relationships or on effects on teachers. Different forms of student participation seem to have different effects, but first and foremost, this review indicates a need for more comprehensive high quality research.     

The development of data use,data skills,and positive attitude towards data use in a data team intervention for teacher educators

This study focuses on understanding how a participation in a data team develops data skills and data use in individual teacher educators. Five teacher educators collaborated in a data team that used data to solve the problem of student teachers dropping out during their course of study. This study aimed at understanding how teacher educators learn from their participation in the data team. We collected data through interviews, surveys, and a knowledge test, and gained insight into the development of data skills, attitude towards data use, and the teacher educators' data use in daily practice. The results show that the data team members’ data skills and attitudes towards data use changed in different ways during the data team intervention depending on their initial situation, and that overall, their data use for school development increased.     

State teacher union strength and student achievement

A new and very small literature has provided evidence that students have lower test scores in larger school districts and in districts in which the district's teachers union has negotiated a contract that is more favorable to the district's teachers. The teachers’ unions at the state and national levels contribute a great deal of money to candidates for state and federal offices. This gives the unions some influence in passing (defeating) bills that would help (harm) the state's teachers. We introduce two novel measures of the strength of the state-wide teachers union: union dues per teacher and union expenditures per student. These reflect the key role of political influence for state-wide unions. We provide remarkably strong evidence that students in states with strong teachers unions have lower proficiency rates than students in states with weak state-wide teacher unions.     

Social order perspective on raising student achievement

Despite significant and sustained gains recorded on the national secondary school leaving examinations between 1999–2004, South Africa’s large-scale secondary school reform has receive little international attention. Defenders of the reforms have argued that the ‘success’ in raising student achievement extended beyond gains in the percentage pass rates to include increased numbers of students completing secondary schooling, a growth in the number of students eligible for admissions to university and a decline in the number of ‘collapsed’ secondary schools in disadvantaged communities. Using a comprehensive dataset that includes national examination results for all candidates between 1996 and 2004, survey information from a group of ‘collapsed’ secondary schools located in disadvantaged communities, qualitative data from school ethnographies, as well as official and unofficial documentary sources, the paper locates student achievement within a framework of structural change in the post-apartheid social order. It shows how student achievement is embedded in complex shifts in the state, economy and civil society. Specifically, the paper examines how structural shifts are enacted at the levels of time-in-school, student expectations, school choice and student selectivity.

Integrating data-based decision making,Assessment for Learning and diagnostic testing in formative assessment

Recent research has highlighted the lack of a uniform definition of formative assessment, although its effectiveness is widely acknowledged. This paper addresses the theoretical differences and similarities amongst three approaches to formative assessment that are currently most frequently discussed in educational research literature: data-based decision making (DBDM), Assessment for Learning (AfL) and diagnostic testing (DT). Furthermore, the differences and similarities in the implementation of each approach were explored. This analysis shows that although differences exist amongst the theoretical underpinnings of DBDM, AfL and DT, the combination of these approaches can create more informed learning environments. The thoughtful integration of the three assessment approaches should lead to more valid formative decisions, if a range of evidence about student learning is used to continuously optimise student learning.     

Effects of teacher professional learning activities on student achievement growth

The authors examined the effects of six types of teacher professional learning activities on student achievement growth over 4 years using statewide longitudinal survey data collected from 467 middle school mathematics teachers in 91 schools merged with 11,192 middle school students' mathematics scores in a standardized assessment in Missouri. The data showed that teacher-centered collaborative activities to learn about mathematics teaching and learning (teacher collaboration and informal communication) seem to be more effective in improving student mathematics achievement than learning activities that do not necessarily involve such teacher-centered collaborative opportunities (professional development programs, university courses, individual learning activities). Teacher-driven research activities through professional conference presentation and participation were also found to be associated with student achievement growth in mathematics. The districts and schools may benefit from investing their professional development funds and resources in facilitating teacher-centered collaborative and research-based learning activities in order to improve student learning.     

The impact of Covid-19 on student achievement: Evidence from a recent meta-analysis

This work attempts to synthetize existing research about the impact of Covid-19 school closure on student achievement. It extends previous systematic reviews and meta-analyses by (a) using a more balanced sample in terms of country composition, (b) considering new moderators (type of data and research design), and (c) including studies on tertiary education students in addition to primary and secondary education students. Our meta-analysis findings show that the pandemic had, on average, a detrimental effect on learning. The magnitude of this learning deficit (about 0.19 standard deviations of student achievement) appears to be roughly comparable to that suffered by students who have experienced a significant disruption in their schooling due to a major natural disaster (e.g., Hurricane Katrina). Students are also found to have lost more ground in math/science than in other subjects. Additionally, one year or more after the first lockdown, students seem to have been unable to catch up on unfinished learning from the pandemic. This result suggests that more efforts should be made to ensure students recover their missed learning in order to avoid negative long-term consequences for them and society.     

The challenge of separating effects of simultaneous education projects on student achievement

When multiple education projects operate in an overlapping or rear-ended manner, it is always a challenge to separate unique project effects on schooling outcomes. Our analysis represents a first attempt to address this challenge. A three-level hierarchical linear model (HLM) was presented as a general analytical framework to separate program effects while taking into account the hierarchy in educational data. The HLM model was then applied to data from the Commonwealth Accountability Testing System that the State of Kentucky has implemented for years, in an attempt to separate the effects of two education projects aimed at improving mathematics and science education in the Appalachian region: the Appalachian Rural Systemic Initiative (ARSI) closely followed by the Appalachian Mathematics and Science Partnership (AMSP). Even though the HLM model successfully separated ARSI and AMSP effects, relevant statistical issues were discussed to improve future efforts in separating effects of simultaneous education projects on schooling outcomes.     

The effects of professional development on the attitudes,knowledge and skills for data-driven decision making

The Dutch government and School Inspectorate encourage schools to use the student performance data they can obtain from their student monitoring systems to maximize student performance in a systematic and goal-oriented way. Research by the same Inspectorate (Inspectie van het Onderwijs, 2010) shows that students in schools which do so outperform students in schools where data-driven decision making (DDDM) is as yet less developed. The University of Twente developed a training course in which school teams learn to utilize data from computerized student monitoring systems in order to improve instructional quality and student performance. Parallel to the training activities, training effects are studied. The research findings show that the training activities had a positive effect on school staff's DDDM knowledge and DDDM skills. Staff attitudes towards DDDM were already high on the pre-tests and remained high on the post-tests.     

Fostering aspects of pre-service teachers’ data literacy: Results of a randomized controlled trial

The present study reports the results of a randomized controlled trial aimed at fostering specific aspects of pre-service teachers’ data literacy. The 6-h intervention focused on data types, reference norms, scale transformations, graphic displays of the properties of frequency distributions, and judgments about the magnitude of mean differences. Pretest-posttest comparisons of a data literacy test showed a large and significant effect of the intervention. Furthermore, main effects but no significant interaction effects with pretest scores were found for personal and motivational covariates (academic self-concept, value beliefs, study interest) on posttest data literacy.     

Teaming up: Linking collaboration networks, collective efficacy, and student achievement

Improving student achievement through teacher collaboration networks is a current focus of schools in many countries. Yet, empirical evidence on the relationship between teacher networks and student achievement and mechanisms that may explain this relationship is limited. This study examined the relationship between teacher networks and student achievement and the mediating role of teachers’ collective efficacy beliefs. Data were collected from 53 Dutch elementary schools. Using social network analysis and multiple regression we analyzed teacher survey and student achievement data. Findings indicate that well-connected teacher networks were associated with strong teacher collective efficacy, which in turn supported student achievement.