Evaluating impact from research: A methodological framework

BackgroundInterest in impact evaluation has grown rapidly as research funders increasingly demand evidence that their investments lead to public benefits.AimsThis paper analyses literature to provide a new definition of research impact and impact evaluation, develops a typology of research impact evaluation designs, and proposes a methodological framework to guide evaluations of the significance and reach of impact that can be attributed to research.MethodAn adapted Grounded Theory Analysis of research impact evaluation frameworks drawn from cross-disciplinary peer-reviewed and grey literature.ResultsRecognizing the subjective nature of impacts as they are perceived by different groups in different times, places and cultures, we define research impact evaluation as the process of assessing the significance and reach of both positive and negative effects of research.Five types of impact evaluation design are identified encompassing a range of evaluation methods and approaches: i) experimental and statistical methods; ii) textual, oral and arts-based methods; iii) systems analysis methods; iv) indicator-based approaches; and v) evidence synthesis approaches.Our guidance enables impact evaluation design to be tailored to the aims and context of the evaluation, for example choosing a design to establish a body of research as a necessary (e.g. a significant contributing factor amongst many) or sufficient (e.g. sole, direct) cause of impact, and choosing the most appropriate evaluation design for the type of impact being evaluated.ConclusionUsing the proposed definitions, typology and methodological framework, researchers, funders and other stakeholders working across multiple disciplines can select a suitable evaluation design and methods to evidence the impact of research from any discipline.     

Lessons Learned From a Consultation Process Overseas

In this commentary I discuss three international school consultation experiences, highlighting aspects that serve as lessons for professional development and the implementation of effective and helpful strategies that meet the needs of children and youth in school systems. Relationships developed and maintained between the consulting teams and the consultees were oriented toward three intervention goals: psychoeducational services, well-being in adolescents, and instructional procedures. In general, the projects described in the three articles were successful and support the utility of international systems-level consultation as a means of helping improve the quality of outcomes in the school contexts in which they worked. I end my commentary by providing some general observations and questions to be considered when engaging in international school consultation.     

Improving Chilean In-service Elementary Teachers’ Understanding of Nature of Science Using Self-contained NOS and Content-Embedded Mini-Courses

Understanding nature of science (NOS) is considered critical to the development of students’ scientific literacy. However, various studies have shown that a large number of elementary and secondary science teachers do not possess an adequate understanding of NOS. This study investigated how elementary teachers’ understanding of NOS was impacted through a 1-year professional development program in Chile that included NOS instruction as a theme throughout two types of mini-courses in the program. Twelve teachers attended a 1-year development program focused on improving teacher content knowledge and included the instruction of NOS embedded in two self-contained NOS mini-courses (36 h) and two lessons (3 h each) within five science content mini-courses (30 h). The Views of NOS (version D+) questionnaire and interviews were used to assess teachers’ understanding of NOS at the beginning (January) and end of the program (December). Elementary teachers’ understanding of the creative, inferential, and tentative aspect of NOS showed improvement. According to the teachers’ perceptions, the most significant activities for improving their NOS understanding were decontextualized activities in both types of mini-courses (self-contained NOS and science content mini-courses). The implications for professional development programs are also discussed.     

Getting Students Outside: Using Technology as a Way to Stimulate Engagement

Informal environments provide students with unique experiences that allow them to actively participate in activities while promoting a positive attitude toward and an increased interest in science. One way to enhance informal science experiences is through the integration of mobile technologies. This integration is particularly useful in engaging underrepresented students in learning science. Our informal environmental science program engages underrepresented, fifth-grade students in an informal learning environment supplemented with mobile tablet technology (iPads). The purpose of this study was to explore how fifth-grade students interacted with nature using mobile technology during a nature hike series. Participants included 55 fifth-grade students from two low-income schools. We found that students used the mobile technology to explore nature and stay engaged throughout the hike. The iPads were used as references, data collectors, and engagement tools. Students had an intense desire in returning to the site and responded positively toward interacting with nature. Prior research has indicated that students in this age group are likely to lose interest in science and the incorporation of field-friendly technology that engages students with nature, not technology alone, is a useful tool for keeping students interested in science.     

Elitebildungsforschung im Vergleich – Frankreich,USA und Großbritannien

Research on elites (that is, on status groups that occupy dominant positions) is characterized by the lack of connection between studies that focus on elite recruitment and those that focus on the exercise of power by elites. As underlined by Giddens (Elites and power in British society, 1974), both types of approach are important and should complement each other in the analysis of mediations between the class structure, the organizational structure and the power structure in a given society. Giddens also insists on the need for recruitment studies to take account of two different dimensions: the types of channel that are privileged by elite groups to reproduce their social position, and the degree of social closure or openness of these channels to other groups (Parkin, The social analysis of class structure, 1974). This distinction is used to organize the present chapter, which focuses on a single channel that has come to play a crucial role in post-industrial societies, that is schools and, more precisely, upper secondary and higher education institutions, and on their influence in three different national contexts: France, the United Kingdom and the United States. In the first section, the specific features of elite education are examined. The second section explores the extent and modes of institutional and social closure.     

Latino High School Students’ Perceptions of Caring: Keys to Success

This mixed methods investigation specifically examined Latino high school adolescents’ perceptions of teacher behaviors that demonstrate caring. A chi-square test was conducted to analyze the frequency of responses, and focus group interviews were conducted to expand on the results. The data indicated that although Latino male students were as likely to perceive the same behaviors Latina female students perceived as important or very important, a higher percentage of female students rated the same items as important. This investigation situates Latino high school adolescents’ voices as a pivotal agent to inform about the critical nature of caring for all students.     

Teachers' beliefs about standardised testing and test-based accountability: Comparing the perceptions and experiences of teachers in Chile and Norway

The global popularity of test-based accountability appears to signal political trust in standardised assessments as valid and relevant measures of education quality. Nonetheless, research shows that educators' perceptions of standardised testing and test-based accountability can vary significantly, as do their responses to accountability demands. Considering the key influence of teachers' beliefs on the way in which they respond to education reforms, in this article we examine teachers' beliefs and opinions about standardised tests and test-based accountability. We analyse a comparative study on interpretations and experiences of standardised testing and test-based accountability demands of compulsory education teachers in Chile and Norway. These cases were selected following a most-different-systems design approach. The data was derived from an electronic survey (n = 2,531) and in-depth interviews (n = 41). The analysis shows how in both contexts, teachers are relatively critical about the validity, usefulness and fairness of standardised tests. This indicates lacking teacher trust in standardised testing and test-based accountability. Still, despite similar trends, some key differences in the beliefs of Chilean and Norwegian teachers are found, which highlight the influence of the sociocultural context in shaping teachers' beliefs. By illuminating how teachers in different contexts make sense of test-based accountability, our analysis contributes to the understanding of why the often-reported mismatch between policy expectations and policy outcomes might occur.     

The importance of a collaborative culture for teachers’ job satisfaction and affective commitment

European Journal of Psychology of Education - Prior research has shown the importance of collaborative culture as well as teacher job satisfaction and commitment in navigating complex school...     

Nonuniform Fuchsian codes for noisy channels

We develop a new transmission scheme for additive white Gaussian noisy (AWGN) channels based on Fuchsian groups from rational quaternion algebras. The structure of the proposed Fuchsian codes is nonlinear and nonuniform, hence conventional decoding methods based on linearity and symmetry do not apply. Previously, only brute force decoding methods with complexity that is linear in the code size exist for general nonuniform codes. However, the properly discontinuous character of the action of the Fuchsian groups on the complex upper half-plane translates into decoding complexity that is logarithmic in the code size via a recently introduced point reduction algorithm.     

Determinants of the Web accessibility of European banks

This paper aims at identifying the factors influencing the implementation of Web accessibility (WA) by European banks. We studied a database made up of 49 European banks whose shares are included in the Dow Jones EURO STOXX® TMI Banks [8300] Index. Regarding the factors for the implementation, we considered three feasible reasons. Firstly, WA adoption can be motivated by operational factors, as WA can aid in increasing operational efficiency. Secondly, we expect large banks to have higher WA levels, as small firms face competitive disadvantages with regard to technology adoption. Lastly, WA can also be understood as a part of the Corporate Social Responsibility (CSR) strategy, so, the more committed a bank is to CSR, the more prone it will be to implement WA. Our results indicate that neither the operational factors nor the firm size seem to have exerted a significant influence on WA adoption. Regarding CSR commitment, results indicate a significant influence on WA adoption. However, the effect of the influence is contrary to that hypothesized, since more CSR-committed banks have less accessible Web sites. A possible reason for this result is that banks not included in the CSR indexes try to overcome this drawback by engaging in alternative CSR activities such as WA.