EVALUATING AND PREDICTING SURVEY EFFICIENCY USING GENERALIZABILITY THEORY

A long-term science performance monitoring program began in England, Wales, and Northern Ireland in 1980 with the first in an initial series of annual national sample surveys of the science performances of II-, 13- and 15-year-old pupils. The assessment framework underlying this program is process-oriented, consisting of a number of subcategories of science activity, some of which are assessed in practical mode. Pupils are randomly selected for testing according to a complex sampling scheme. Questions are also selected randomly to represent the various subcategories.
From the start of this program, it was intended to appeal to generalizability theory for a suitable estimation paradigm, and in this paper some preliminary applications of G-theory are described. The results of these applications would suggest that computerized question-banking, domain-sampling of questions, and G-theory together provide a useful new technology for this kind of performance monitoring exercise. The issue of interpretability might still remain a problem, however, unless the question domains can be clearly defined, and can be reflected in the question pools with consistency over time.     

On shifting sands: assessing the financial sustainability of UK learned societies

This article provides a quantitative assessment of the extent to which UK learned societies rely on publishing revenues. Drawing on work completed as part of a Universities UK project to monitor the transition to open access in the United Kingdom, it considers the risks that increased market consolidation and a shift to open access publishing present for societies' financial sustainability in the coming years. The project identified 279 UK societies that publish peer‐reviewed publications. It is estimated that publishing accounts for just over £300 million, or 26%, of these societies' overall revenues of £1.2 billion, but an in‐depth analysis of 30 societies found that the proportion is as high as 80% in some cases. Publishing is typically a profitable activity for societies, and thereby supports their charitable activities and makes an important contribution to their overall financial sustainability. Although most societies are presently in good financial health, the combined pressures of market consolidation and open access, coupled with early indications of an increase in the costs of publishing, suggests that their reliance on publishing could prove an uncertain foundation in the years to come.