Methods in School Effectiveness Research∗

This paper discusses the methodological requirements for valid inferences from school effectiveness research studies. The requirements include long term longitudinal data and proper statistical modelling of hierarchical data structures. The paper outlines the appropriate multilevel statistical models and shows how these can model the complexities of school, class and student level data.

     

A systematic review on the deployment and effectiveness of data analytics in higher education to improve student outcomes

Abstract

This is a systematic review conducted of primary research literature published between 2007 and 2018 on the deployment and effectiveness of data analytics in higher education to improve student outcomes. We took a methodological approach to searching databases; appraising and synthesising results against predefined criteria. We reviewed research on the effectiveness of three differentiated forms of data analytics: learning, academic and learner analytics. Student outcomes are defined as retention, academic performance and engagement. Our results find that three quarters of studies report the use of educational data analytics to be effective in improving student outcomes but their relationship with student outcomes requires further and more robust investigation and assessment. We argue that research must interpret and communicate effectiveness qualitatively, as well as quantitatively, by including the student voice in assessments of impact.     

Student Mobility in Multilevel Growth Modeling: A Multiple Membership Piecewise Growth Model

This study proposes a new model, termed the multiple membership piecewise growth model (MM-PGM), to handle individual mobility across clusters frequently encountered in longitudinal studies, especially in educational research wherein some students could attend multiple schools during the course of the study. A real data set containing some students who switched elementary schools was used to demonstrate and explain the MM-PGM. Parameter and model fit differences were compared between the MM-PGM and two other techniques for handling student mobility: the first school-PGM, which only used school membership at the first measurement occasion, and the delete-PGM, which removed mobile students from the analysis. Results indicated that the three approaches of handling mobile students led to different conclusions about the impact of school-level predictors of growth parameters and the school-level variability in the growth rates. Furthermore, deleting mobile students altered the impact of student-level predictors compared to the other two approaches.     

Multilevel Modelling in Repeated Measures of the Quality of Finnish School Life

The main interest of the paper is methodological: to exploit multilevel modelling in analysing quality of school life data gathered in 1991 and 1995 in Finnish comprehensive schools, with the intention of investigating differences and related changes in the quality of school life over time. The specific methodological aim is to explore the advantages gained and problems encountered while applying multilevel models and the statistical program MLwiN in analysing large scale educational data with repeated measures. The quality of school life was assessed by a representative sample of all the Grade 8 pupils throughout the country with a 4 year interval in the same schools. The measurement of the quality of school life focused on general school satisfaction and on teacher-student relations. The validity of the instrument was confirmed by means of factor analysis. The findings of the multilevel statistical analysis are discussed both from an educational and a methodological point of view.     

Long-term effects of primary schools on educational positions of students 2 and 4 years after the start of secondary education

Although the importance of primary schools in the long term is of interest in educational effectiveness research, few studies have examined the long-term effects of schools over the past decades. In the present study, long-term effects of primary schools on the educational positions of students 2 and 4 years after starting secondary education are investigated. Moreover, it is examined which school factors play a role in this process. We specifically investigated whether effective primary schools make a difference in the long term. This study uses data from the longitudinal SiBO project, which followed 6,000 pupils in primary education in Flanders, Belgium, and has follow-up data during secondary education. Two-level models and cross-classified multilevel models show that primary schools have long-term effects on the educational positions of students in secondary education.     

comparison of three growth modeling techniques in the multilevel analysis of longitudinal academic achievement scores: Latent growth modeling, hierarchical linear modeling, and longitudinal profile analysis via multidimensional scaling

This study introduces three growth modeling techniques: latent growth modeling (LGM), hierarchical linear modeling (HLM), and longitudinal profile analysis via multidimensional scaling (LPAMS). It compares the multilevel growth parameter estimates and potential predictor effects obtained using LGM, HLM, and LPAMS. The purpose of this multilevel growth analysis is to alert applied researchers to selected analytical issues that are required for consideration in decisions to apply one of these three approaches to longitudinal academic achievement studies. The results indicated that there were no significant distinctions on either mean growth parameter estimates or on the effects of potential predictors to growth factors at both the student and school levels. However, the study also produced equivocal findings on the statistical testing of variance and covariance growth parameter estimates. Other practical issues pertaining to the three growth modeling methods are also discussed.     

Variations Between Latino Groups in us Post-Secondary Educational Attainment*

Hispanics/Latinos are the fastest growing minority group in the US, and represent a diverse variety of ethnic groups with unique heritages. Yet educational and social research often analyzes this group in aggregate. This research, employing 1980 high school senior longitudinal data from the High School and Beyond project, demonstrates substantial differences in post-secondary educational attainment between some Hispanic/Latino groups. Net of a number of individual, socio-economic, family and high school background variables, logit analysis shows that Mexican Americans are significantly below others in post-secondary educational attainment after high school. The results are discussed in regard to future research adopting analysis of individual Hispanic/Latino groups rather than employing the aggregate measure which can substantially misrepresent the relationship between component Latino/a groups and educational attainment.We are indebted to Robert Kaufman for consultation on methodological and statistical approaches; and to Michael Hughes, James Michaels, and Ellen Plummer for helpful suggestions for revision on earlier versions of this paper. This paper is based in part on a Master’s thesis by the first-named author.     

Measuring equity of education resource allocation: An output-based approach

This paper provides a methodological framework to measure and analyze educational resource allocation within and across systems and ascertain potential equity implications. The proposed approach employs an outputs-driven method to provide a snapshot of the equitable distribution of key educational resources available in relation to student populations that are most in need. Our approach standardizes educational resources into three broad dimensions including teacher quality, school physical environment, and school instructional environment while contrasting the allocation of these resources between low- and high-needs schools. We implement this approach using real world data from Brazil to demonstrate its adaptability to the context and available data while still maintaining a consistent framework across applications.     

The Challenges and Tensions in Reconstructing Teacher - Parent Relations in the Context of School Reform: A case study

This paper reports on a comparison of two research approaches of the study of the reflective practice in two different modes of tele‐guidance of the school practicum. The results of two studies were contrasted encouraging the dialogue between researchers coming from different research paradigms. Although the researchers in the two studies shared some perspectives on educational practices and student teachers’ reflection, the studies resulted in two different reports on student teachers’ reflection. Ontological, epistemological and methodological beliefs in both studies are related to a rationalistic and a naturalistic paradigm informing the two research sites. Researchers’ voices in both studies have been interacting, changing and developing but the differences in beliefs between the two sites seem to have contributed to the different analyses and results.     

Modelling Student Progress in Studies of Educational Effectiveness∗

It is argued that a crucial requirement in studies of educational effectiveness is the modelling of change or growth in student learning. To illustrate one approach to achieving this end, results are presented from multivariate multilevel analyses of three‐waves of data for three Grade level cohorts of students from a longitudinal study designed to explain variation in elementary school students' progress in literacy achievement. The article provides estimates of the influence of prior achievement and social background factors including ‘critical events’, on students’ progress, examines the extent to which progress can be accounted for by the grouping effects of students within classes and schools over successive years, and provides estimates of the effects of explanatory variables at the student‐and class/teacher‐levels.

     

Institutional Characteristics and College Student Dropout Risks: A Multilevel Event History Analysis

In the past two decades, although access to higher education for American students has improved, student persistence in 4-year institutions is far from assured. There have been a number of research studies on student persistence/dropout in higher education, but most have focused on the characteristics and behavior of students as illustrated by the ??student-centered research tradition??. This study focuses on what institutional characteristics contribute to conditions that reduce student dropout risks. By analyzing longitudinal and hierarchical data, this research proposes and tests a multilevel event history model that identifies the major institutional attributes related to student dropout risk in a longitudinal process. Evidence indicates that institutional expenditure on student services is negatively associated with student dropout behavior. Implications of the results for institutional practices and future research are discussed.     

Do Students Develop Towards More Deep Approaches to Learning During Studies? A Systematic Review on the Development of Students’ Deep and Surface Approaches to Learning in Higher Education

The focus of the present paper is on the contribution of the research in the student approaches to learning tradition. Several studies in this field have started from the assumption that students’ approaches to learning develop towards more deep approaches to learning in higher education. This paper reports on a systematic review of longitudinal research on how students’ approaches to learning develop during higher education. A total of 43 studies were included in the review. The results give an unclear picture of the development of approaches to learning and, thus, do not provide clear empirical evidence for the assumption that students develop towards more deep approaches during higher education. Neither methodological nor conceptual aspects of the studies investigated explained the ambiguity of the research results. Both theoretical and empirical implications for further research are discussed.     

Teaching Groups as Foci for Evaluating Performance in Cost-Effectiveness of GCE Advanced Level Provision: Some Practical Methodological Innovations1

Some key methodological issues in factors affecting educational effectiveness within schools are discussed. The motivation is an analysis of subject teaching group effectiveness in English and Welsh GCE Advanced Level prior to a linking to resources. Cross-classified multilevel models with weighted random effects are suggested for disentangling student, group, and teacher effects. Analyses show that there is considerable variation between subjects within students. Teacher effects are shown to be considerable but explanations for variation cannot be sought in the conventional personal characteristics of teachers. One conclusion is that it is availability of data which constrains such investigations and not methodology.     

Qualitative sampling in research on international student mobility: insights from the field in Germany1

ABSTRACT

Although over the past decades the numbers of studies investigating international student mobility and migration (ISM) increased, methodological challenges in empirical research on the topic have remained mainly unaddressed. This is particularly the case for sampling, which is a crucial but often less considered part of qualitative research designs. In this article, we identify three main challenges in qualitative sampling for research into ISM: time, space and international students’ heterogeneities. In addressing those challenges, we theoretically discuss their implications and give empirical examples drawing on our research experiences. We argue for a more reflexive research procedure in studying educational mobility.     

Researching social inclusion in student mobility: methodological strategies in studying the Erasmus programme

This article explores different methodological approaches to the study of social inclusion in the European Commission’s Erasmus undergraduate exchange programme, elaborating upon the approach taken in a study of Erasmus conducted in Portugal. The opening section of the discussion acknowledges the prior tradition of quantitative research on student mobility in Europe as a means of providing indicators of programmatic success to policymakers and stakeholders, and the existence of micro level studies that examine specific aspects of the Erasmus experience such as the emergence of shared European identities and building work competencies. What these approaches share is an emphasis on student perspectives: learning about the impact of exchange visits from past participants. As an alternative approach, I have focused on the institutional level of the programme, using qualitative interviews with individuals involved in managing Erasmus mobility at a diverse range of universities, an approach that I argue unlocks valuable knowledge about issues such as social inclusion within the programme.     

Assessing the impact of learning environments: How to use student ratings of classroom or school characteristics in multilevel modeling

In educational research, characteristics of the learning environment (e.g., social climate, instructional quality, goal orientation) are often assessed via student reports, and their associations with outcome variables such as school achievement or student motivation then tested. However, studying the effects of the learning environment presents a series of methodological challenges. This article discusses three crucial elements in research that uses student reports to gauge the impact of the learning environment on student outcomes. First, from a conceptual point of view, it is argued that ratings aggregated at the relevant level (e.g., class or school level), and not individual student ratings, are of primary interest in these studies. Second, the reliability of aggregated student ratings must be routinely assessed before these perceptions are related to outcome variables. Third, researchers conducting multilevel analyses need to make very clear which centering option was chosen for the predictor variables. This article shows that conclusions about the impact of learning environments can be substantially affected by the choice of a specific centering option for the individual student ratings.     

Comparisons of two statistical approaches to study growth curves: The multilevel model and the latent curve analysis

This article compares two statistical approaches for modeling growth across time. The two statistical approaches are the multilevel model (MLM) and latent curve analysis (LCA), which have been proposed to depict change or growth adequately. These two approaches were compared in terms of the estimation of growth profiles represented by the parameters of initial status and the rate of growth. A longitudinal data set obtained from a school‐based substance‐use prevention trial for adolescents was used to illustrate the similarities and differences between the two approaches. The results indicated that the two approaches yielded very compatible results. The parameter estimates associated with regression weights are the same, whereas those associated with variances and covariances are similar. The MLM approach is easier for model specification and is more efficient computationally in yielding results. The LCA approach, however, has the advantage of providing model evaluation, that is, an overall test of goodness of fit, and is more flexible in modeling and hypothesis testing as demonstrated in this study.     

Using video surveillance footage to support validity of self-reported classroom data

The use of video surveillance footage presents a new possibility in educational research as a reliable and valid source of learning and teaching activities in classrooms. However, the unique nature of surveillance footage requires different approaches and poses distinctive challenges in utilizing it in research, yet no methodological guides are currently available in educational research. The purpose of this reflective methodological review is to share our lessons learned from a mixed-methods study that utilized video surveillance footage for triangulating self-reported classroom data. We discuss (1) the importance of triangulation of self-reported data to reduce biases of it, (2) the unique benefits and limitations of using video surveillance data, (3) a methodological framework for video surveillance data analysis, and (4) our process of data analysis with technical, ethical, and other issues in mind when using such data. Software programs to aid video data analysis are briefly introduced as well.     

The transnational in-between identity of Chinese student returnees from the UK: Mobility,variations and pathways

Considering the rising tide of international student return mobility, the UK needs to pay attention to the long-term circulation and development of international student returnees. This study analyses the mobility, variations and pathways of the transnational in-between identity of Chinese student returnees from the UK, an under-researched group in the changing dynamics of international student mobility. This longitudinal study uses original data from 48 semi-structured interviews to show how student returnees’ identities change over time in transnational sociocultural contexts. This study develops a spider chart comprising multiple manifestations of transnational selves to account for returnees’ intermediary and heterogeneous states of transnational in-between identities. Based on the empirical evidence, the study finds that transnational in-between identity is characterised by a ‘self-in-between mobility’, seeking compromises and connections between the polar extremes of essentialist and non-essentialist approaches to identity. In such an identification process, the changing dynamics of agency play a pivotal role in (re)shaping multiple identity pathways as variegated temporal outcomes of mobility. The findings not only advance the theoretical and methodological constructs of identity, but also have implications for relevant concerns in international education.     

Integrating data-based decision making,Assessment for Learning and diagnostic testing in formative assessment

Recent research has highlighted the lack of a uniform definition of formative assessment, although its effectiveness is widely acknowledged. This paper addresses the theoretical differences and similarities amongst three approaches to formative assessment that are currently most frequently discussed in educational research literature: data-based decision making (DBDM), Assessment for Learning (AfL) and diagnostic testing (DT). Furthermore, the differences and similarities in the implementation of each approach were explored. This analysis shows that although differences exist amongst the theoretical underpinnings of DBDM, AfL and DT, the combination of these approaches can create more informed learning environments. The thoughtful integration of the three assessment approaches should lead to more valid formative decisions, if a range of evidence about student learning is used to continuously optimise student learning.