Do altmetrics assess societal impact in a comparable way to case studies? An empirical test of the convergent validity of altmetrics based on data from the UK research excellence framework (REF)

Altmetrics have been proposed as a way to assess the societal impact of research. Although altmetrics are already in use as impact or attention metrics in different contexts, it is still not clear whether they really capture or reflect societal impact. This study is based on altmetrics, citation counts, research output and case study data from the UK Research Excellence Framework (REF), and peers’ REF assessments of research output and societal impact. We investigated the convergent validity of altmetrics by using two REF datasets: publications submitted as research output (PRO) to the REF and publications referenced in case studies (PCS). Case studies, which are intended to demonstrate societal impact, should cite the most relevant research papers. We used the MHq’ indicator for assessing impact – an indicator which has been introduced for count data with many zeros. The results of the first part of the analysis show that news media as well as mentions on Facebook, in blogs, in Wikipedia, and in policy-related documents have higher MHq’ values for PCS than for PRO. Thus, the altmetric indicators seem to have convergent validity for these data. In the second part of the analysis, altmetrics have been correlated with REF reviewers’ average scores on PCS. The negative or close to zero correlations question the convergent validity of altmetrics in that context. We suggest that they may capture a different aspect of societal impact (which can be called unknown attention) to that seen by reviewers (who are interested in the causal link between research and action in society).     

The Artist Initiative at San Francisco Museum of Modern Art

The Artist Initiative at the San Francisco Museum of Modern Art (SFMOMA) is piloting models for increased collaboration between conservators and curators through joint work with artists. We seek a more integrated, holistic approach to the care and research of our collection. The project, funded by the Andrew W. Mellon Foundation, comprises five robust research engagements that serve the curatorial collecting departments of the museum (Photography, Painting and Sculpture, Media Art, and Architecture and Design). Three of the projects are monographic studies, examining the work of Ellsworth Kelly, Vija Celmins, and Julia Scher in depth, while two more are thematic, exploring modes of displaying digitally-driven design objects, and developing strategies for addressing the problem of color shift common to photographic prints made with experimental materials during the 1970s and 1980s. The Artist Initiative is also charged with developing hybrid working spaces to advance collaborative approaches to collections research at the museum's new downtown campus and at SFMOMA's new Collections Center in South San Francisco. These spaces include the Collections Workroom, a 56?sq?m (600?sq?ft) space that functions as a studio for visiting artists, a conservation laboratory, an interview suite, and a classroom at SFMOMA's downtown campus. At the Collections Center, a 121?sq?m (1300?sq?ft) Mock-Up Gallery has been built as a working model of one of the museum's new galleries. A functional exhibition space, the Mock-Up Gallery is also a venue for interviewing artists, prototyping exhibition formats, and meeting with students, scholars, museum staff, and community members. With the goal of contributing to critical discourses in contemporary art history, art conservation, and public engagement, each of the Artist Initiative projects includes a colloquium that will bring experts from multiple fields together with the featured artists. Thereafter we aim to share our findings widely through public programs and a range of publications, both digitally and in print.     

The effectiveness of intelligent tutoring systems on K-12 students' reading comprehension: A meta-analysis

This meta-analysis examined the effectiveness of improving reading comprehension for students in K-12 classrooms using intelligent tutoring systems (ITSs), a computer-based learning environment that provides customizable and immediate feedback to the learner. Nineteen studies from 13 publications incorporating approximately 10 000 students were included in the final analysis; using robust variance estimation to account for statistical dependencies, the 19 studies yielded 88 effect size estimates. The meta-analysis indicated that the overall random effect size of ITSs on reading comprehension was 0.60 (using a mix of standardized and researcher-designed measures) with a 95% confidence interval 0.36 to 0.85 (p < 0.001). This review confirms previous studies comparing ITSs to human tutoring: ITSs produced a small effect size when compared to human tutoring (0.20, 0.02–0.38, p = 0.036, n = 21). All comparisons to human tutoring used standardized measures. This review also found that ITSs produced a larger effect size on reading comprehension when compared to traditional instruction (0.86) for mixed measures and (0.26) for standardized measures. These findings may be of interest to practitioners and policy makers seeking to improve reading comprehension using consistent and accessible ITSs. Recommendations for researchers include conducting studies to understand the difference between traditional and updated versions of ITSs and employing valid and reliable standardized tests and researcher-designed measures.     

Developing a virtual engineering management community

The paper reviews the lessons of planning and running an Engineering Management practitioner development programme in a partnership between BP and the University of Manchester. This distance-learning programme is for professional engineers in mid-career experienced in the engineering and support activities for delivering safe, compliant and reliable projects and operations worldwide. The programme concentrates on the why and how of leadership and judgement in managing the engineering of large and small projects and operational support. Two intensive residential weeks are combined with a virtual learning environment over one year. Assessed assignments between and after the residential weeks provide opportunities for individual reflective learning for each delegate through applying concepts and the lessons of case studies to their experience, current challenges and expected responsibilities. This successful partnership between a major global company and a university rich in research and teaching required a significant dedication of intellectual and leadership effort by all concerned. The rewards for both parties and most importantly for the engineers themselves are extensive.     

“We Keep the Education Goin’ at Home All the Time”: Family Literacy in Low-Income African American Families of Preschoolers

Researchers have examined the impact of family on child literacy among low-income African American families and preschoolers considered to be at risk for not being ready for kindergarten. Quantitative studies identify family-parental variables associated with poorer literacy outcomes, whereas qualitative studies detail family practices that promote child literacy development. Addressing the limitations of social address variables in quantitative research, and the paucity of research on preschoolers in qualitative research, this study examines the home-based literacy practices of 20 low-income, African American families with preschoolers in Head Start transitioning to kindergarten. Using qualitative interviews informed by a resilience framework, we found that home-based literacy activities were carried out within teams of diverse kin who worked together to promote children's school readiness. Family literacy teams expanded the literacy resources available to preschoolers, providing a rich literacy environment for children's development. These findings contribute to our substantive understanding of literacy practices within low-income African American families, resilience theory, and culturally relevant home-school collaborations.     

Changing practices: influences on classroom assessment

The pedagogical potential of classroom assessment to support student learning has increasingly been evidenced in research over the past decade. Constructive classroom assessment has been championed by assessment specialists, and endorsed by professional organizations. In practice, however, the process of changing classroom assessment from its traditionally summative orientation is not straightforward. This methodical review looks at how six sources, which are educational research, evaluative inquiry, large‐scale assessment, educational policy, professional development, and teachers’ beliefs, influence and mediate assessment practices. A group of purposively selected research articles are analysed as evidence of the dynamics in this complex process. Cross‐currents relating to research perspective, collaboration, and time are discussed. For the movement seen in this study to continue, the tension between teacher autonomy and school community, and the relationship between collective commitment and assessment literacy should be considered.