What did PISA and TIMSS ever do for us?: the potential of large scale datasets for understanding and improving educational practice

ABSTRACT

There appears to be something of an intellectual and philosophical gulf between education researchers who seek insights from statistical analyses of complex data-sets such as those provided by the OECD (PISA), and others who seek to develop rich, contextualised socio-historical understandings that can shed light upon why particular classroom practices operate and are sustained within a given milieu. This paper outlines these different perspectives, with particular reference to non-cognitive factors. Detailed analysis of the roots of high academic achievement, and associated challenges to student wellbeing, in many East Asian countries, is provided. The important influence of broad political and societal factors is highlighted by reference to cross-cultural differences across a large number of countries. The paper concludes by stating that while data from large datasets can assist in gaining greater understanding of cross-cultural differences, to be meaningful, such analyses should be incorporated within complex ecosystemic accounts.     

The visible hand of research performance assessment

Far from allowing a governance of universities by the invisible hand of market forces, research performance assessments do not just measure differences in research quality, but yield themselves visible symptoms in terms of a stratification and standardization of disciplines. The article illustrates this with a case study of UK history departments and their assessment by the Research Assessment Exercise (RAE) and the Research Excellence Framework (REF), drawing on data from the three most recent assessments (RAE 2001, 2008, REF 2014). Symptoms of stratification are documented by the distribution of memberships in assessment panels, of research active staff, and of external research grants. Symptoms of a standardization are documented by the publications submitted to the assessments. The main finding is that the RAEs/REF and the selective allocation of funds they inform consecrate and reproduce a disciplinary center that, in contrast to the periphery, is well-endowed with grants and research staff, decides in panels over the quality standards of the field, and publishes a high number of articles in high-impact journals. This selectivity is oriented toward previous distributions of resources and a standardized notion of “excellence” rather than research performance.     

Note About Possible Bias Resulting When Under-Statisticized Studies are Excluded from Meta-Analyses

Reviews and meta-analyses of research on a given topic may exclude' a sizable percentage of reports because they do not lend themselves to the type of summarizing procedures used. If the excluded articles contain relevant information, this may bias the conclusions of the analysis. It seems likely that, when computing statistics from their data, researchers will need to consider this aspect. A simple illustration of how that can sometimes be done readily is presented. A robust correlation coefficient easily computable from published data is shown to indicate a sizable relationship that is contrary to the main conclusion of a meta-analysis.