The jurisdiction of the subjective expected utility (SEU) approach to risk-taking in science – A response to Franzoni and Stephan (2023), ‘uncertainty and risk-taking in science’

This paper discusses the SEU approach with regard to its “jurisdiction”; i.e. where it should and where it should not be applied. It is argued that the SEU approach must address the fundamental tension between exploration and exploitation; that it should be applied only to disciplines or fields that have low degrees of both technical and strategic uncertainties; and that funders need to prevent the SEU approach from spilling over to evaluations of exploratory projects, for which it is clearly not suited. In addition, the paper argues that the SEU approach needs to respond to the “fund people, not projects” argument. Finally, it calls for many more empirical and preferably experimental studies with the aim of shedding light on how the SEU approach works in practice. In this regard, novel techniques should be used more frequently than today, such as specification curve analysis that encompasses all reasonable explanatory specifications that are both theoretically consistent and statistically valid and non-redundant.     

Against misleading technocratic precision in research evaluation and wider policy – A response to Franzoni and Stephan (2023), ‘uncertainty and risk-taking in science’

Franzoni and Stephan (2023) recommend a probabilistic ‘subjective expected utility’ technique for addressing challenges of uncertainty in research evaluation. Whilst acknowledging strengths in F&S's analysis, this Response highlights a series of important practical, theoretical and methodological deficiencies. The stakes are raised, in that these are widely shared in a growing body of practice across research policy and beyond. This practice seeks to reduce and aggregate real-world complex, ambiguous, qualitative, multidimensional and contested challenges through ostensibly precise calculation. Taking associated problems in turn, this Response shows how F&S: make scientifically dubious claims; understate the depths of uncertainty; overstate the sufficiency of quantification; neglect foundational limits to calculation; and ignore crucial interpretive dimensions of policy making. Highlighting roles for greater methodological diversity, this Response points at the end to alternative methods that collectively allow more robustly plural approaches to contrasting aspects of incertitude. In the process, the steering of directions for research can become more rigorous and accountable and less vulnerable to manipulation and inadvertent bias. With globally growing ‘post-truth’ authoritarian populism arguably partly provoked by the kind of technocracy criticised here, research evaluation may in a small way help re-invigorate democracy by ‘opening up’ in this particular area, the hiding of politics behind expertise.