Indicators for national science and technology policy: how robust are composite indicators?

This article addresses a set of issues that were central to Keith Pavitt's research, that is, the construction and use of tools to measure national innovative performance and to design national policies relating to innovation. It presents an overview of the development of science and technology (S&T) indicators and their use in national policy making and provides evidence of the vulnerability of composite S&T indicators to manipulation. A brief history of the development of S&T indicators begins with the role of the United States followed by their worldwide diffusion with particular emphasis on Europe. Newer developments towards composite indicators, benchmarking and scoreboarding are discussed. To investigate the robustness of innovation scoreboards empirically, a sensitivity analysis of one selected case is presented. It is shown that composite scores and country rank positions can vary considerably depending on the selection process. Thus, the use of scoreboards leaves room for manipulation in the policymaking system. Further research is needed on alternative methods of calculation to prevent their misuse and abuse.