The Chartered Society of Physiotherapy The Chartered Society of Physiotherapy

Basket

View your shopping cart.

Research impact

Definitions of research impact

The definition used by the Research Excellence Framework (REF) is particularly important as assessment of impact is now a significant part of the REF. 

'For the purposes of the REF, impact is defined as an effect on, change or benefit to the economy, society, culture, public policy or services, health, the environment or quality of life, beyond academia.

Impact includes, but is not limited to, an effect on, change or benefit to:

  • the activity, attitude, awareness, behaviour, capacity, opportunity, performance, policy, practice, process or understanding
  • of an audience, beneficiary, community, constituency, organisation or individuals
  • in any geographic location whether locally, regionally, nationally or internationally.

Impact includes the reduction or prevention of harm, risk, cost or other negative effects.'

Research Excellence Framework (REF). Assessment framework and guidance on submissions. REF02.2011 (2011) p48.

For other definitions of impact, see the following external resources:

Scope of research impact

Research has an impact on a range of research users and in terms of definition, ‘academic’ impact is normally distinguished from other types of impact.

The REF definition of impact specifically excludes academic impact. Academic impact is defined as follows:

'The demonstrable contribution that excellent research makes to academic advances, across and within disciplines, including significant advances in understanding, methods, theory and application.’

Research Councils UK definition refers to economic and societal impacts:

‘...the demonstrable contribution that excellent research makes to society and the economy.’

and notes the diversity of economic and societal impacts:

‘Economic and societal impacts embrace all the extremely diverse ways in which research-related knowledge and skills benefit individuals, organisations and nations by:

  • fostering global economic performance, and specifically the economic competitiveness of the United Kingdom,
  • increasing the effectiveness of public services and policy,
  • enhancing quality of life, health and creative output.’


Other terms for impact

Other terms used in relation to the concept of research impact include ‘benefit’, ‘payback’, ‘translation’, ‘transfer’, ‘uptake’ and ‘utilisation’.

Measuring impact

Demonstrating the value of research is complicated by many factors which need to be considered in order to develop a comprehensive method of measuring impact.

Quantification

Research impacts are often not easy to quantify; for example, it is difficult to show how behaviours have changed as a result of research findings alone.

Timescale

Impacts occur at various stages in the research cycle, and there is often a time lag, possibly 5-20 years between undertaking research and when its full benefits are realised.

Attribution

Impact is often associated with a body of knowledge from multiple sources rather than one specific research finding.

Types of impact and research use

Two main types of research impact have been described:

  • Instrumental impact is direct and shows a clear connection between a particular study and a specific outcome
  • Conceptualimpact is less tangible; it advances knowledge and informing decisions without showing a direct link. It is associated with changes in people’s level of knowledge, understanding, values and attitudes.

For further information on types of impact, see:

Beacham B, Kalucy L, McIntyre E. Focus on understanding and measuring research impact. Focus on... 2005 Dec; Issue 2: Primary Health Care Research and Information Service, Adelaide, Australia.

Users of research

Users of health research include the public, patients, carers and healthcare consumers; clinicians, clinical service providers, healthcare managers, policy makers, researchers and research funders.

Approaches to increase research use

Approaches include interventions to change behaviour and develop linkage and exchange between researchers and users.

The Canadian Health Services Research Foundation recommends the following approaches:

  • ‘Producer push’: adaptation and reproduction of research results
  • 'User pull’: user knowledge acquisition, efforts and receptivity
  • ‘Exchange/collaboration':linkage and exchange between researchers and users

Lomas J. Using ‘linkage and exchange’ to move research into policy at a Canadian foundation. Health Affairs 2000;19(3):236-40

Categories of research use

Research impact can fall into a range of categories. For example, in health services research five categories have been defined:

  • Knowledge
  • Benefits to future research and research use
  • Benefits from informing policy and product development
  • Health and health sector benefits
  • Broader economic benefits

Donovan C, Hanney S. The ‘Payback Framework’ explained. Research Evaluation. 2011;20(3):181-3

The REF assessment framework defines nine categories of impact; some are more likely to be relevant to physiotherapy and health research than others.

The nine categories are:

  • Health and welfare
  • Society, culture and creativity
  • Economy
  • Commerce
  • Public policy and services
  • Production
  • Practitioners and services
  • Environment
  • International development

Research Excellence Framework. Panel criteria and working methods. REF 01.2012 (2012): Part 2A, Main Panel A criteria, Table A1, p28.

Methods for measuring impact

RAND Corporation

RAND Europe, a non-profit research organisation, has developed a research measurement toolkit which includes a range of evaluation methods. It recommends a multi-indicator, multi-method approach to measuring research impact.

Research Excellence Framework

Impact is assessed through the submission of:

  • a template describing the submitted unit’s approach to enabling impact from its research
  • case studies describing specific examples of impacts achieved together with corroborating evidence of the impact

For further information see Research Excellence Framework. Assessment framework and guidance on submissions. REF02.2011 (2011).

  • Part 3,Section 3: Impact template and case studies p 6
  • Annex G: Impact case study template and guidance p 52

Impact assessment frameworks

A number of frameworks for measuring impact have been developed. Different models take into account different aspects of impact. For example, the frameworks described below reflect timescale of impact, approaches to increasing impact, and categories of research impact.

The ‘payback’ framework

Buxton M, Hanney S. How can payback from health services research be assessed? Journal of Health Services Research & Policy 1996;1(1):35-43

This framework was originally developed to assess the impact or ‘payback’ of health services research. The model relates impact to a timeframe. It incorporates seven different stages of research, from topic identification through to dissemination and then implementation/adoption.

Examples of effects or outcomes and data sources/measures are provided at different stages.

For example:

Stage of research

Effects or outcomes

Data sources/measures

Dissemination

Translation/adoption as:

  • Policy or administrative decisions
  • Clinical guidelines
  • Training packages
  • Improved allocation of resources
  • Setting of health targets
  • How-to manuals
  • Criteria used by evaluative and inspecting bodies
  • Commercial development

Survey of decision makers (central and local levels) to establish if they were influenced by the research

Examination of official statements, guidelines, evaluative criteria etc, to assess how far they reflect and cite research findings


Adapted from: Beacham B, Kalucy L, McIntyre E. Understanding & measuring research impact. Focus on ...2005 Dec;Issue 2: Primary Health Care Research and Information Service, Adelaide, Australia.

Framework based on approaches to increasing research use – ‘producer-push’, ‘user-pull’ and ‘exchange’ models

Lavis J, Ross S, McLeod C, Gildiner A. Measuring the impact of health research. Journal of Health Services Research and Policy. 2003 Jul;8(3):165-70.

This model assesses research impact according to whether the active role in promoting research use in decision-making is undertaken by researchers (producer-push measures), decision-makers (user-pull measures) or both researchers and decision-makers (exchange measures).

Outputs are grouped according to whether they measure a process (e.g. the number of papers published); an intermediate outcome (e.g. change in awareness about a particular issue); or a final outcome (e.g. policy decision based on research knowledge).

For example:

Exchange

Effect/outcomes

Data sources/measures

Process outcomes

Research organisations involve decision-makers in the research process

Decision-making organisations involve researchers in decision-making processes

Research organisation files

Decision-making organisation files

Intermediate outcomes

Decision-maker’s and researcher’s assessment of how they were involved in the decision-making process

Survey of, or structured interviews with, decision-makers

Final outcomes

Research organisation’s research reflects the research needs and context of decision-makers

Decision-making organisation’s decisions reflect the research available to them

Research organisation’s files and survey or interviews with decision-makers

Decision-making organisation’s files and survey or structured interviews with researchers


Adapted from: Beacham B, Kalucy L, McIntyre E. Understanding & measuring research impact. Focus on ...2005 Dec;Issue 2: Primary Health Care Research and Information Service, Adelaide, Australia.

REF assessment framework

The REF assessment framework considers the effects and outcomes of research, and possible evidence and indicators of impact, in relation to different categories of research impact.

For example:

Impact category

Effects/outcomes

Evidence/indicators

Impacts on health and welfare

Outcomes for patients or related groups have improved

Public health and well-being has improved

Public engagement/ involvement in research has improved

The user experience has improved

Measures of improved clinical outcomes, public behaviour or health services

Measures of improved well-being

Documented changes to clinical and public health guidelines

Evidence from audit, changes in guidelines

Evidence of enhancement of patient experience

Impacts on the economy

The costs of treatment or healthcare have changed as a result of research-led changes in practice

Evidence of improved cost-effectiveness

Evidence of service change

Impact on practitioners and services

Professional standards, guidelines or training have been influenced by research

Practitioners/professionals have used research findings in conducting their work

Workforce planning has been influenced by research

Educational or pedagogical practices and methods have changed outside of the submitting unit

Literature/web information from practitioners and advisers, including the research findings and how they are applied in practice

Evidence of adoption of best practice

 

Research Excellence Framework. Panel criteria and working methods. REF 01.2012 (2012)
Part 2A Main Panel A criteria: Table A1 (p28) and Table A2 (p31).

Additional resources

Research Councils UK

The pathways to impact toolkit provides guidance and resources to help demonstrate and maximise the impact of research.

Primary Health Care Research and Information Service

Introduction to research impact on policy

Beacham B, Kalucy L, McIntyre E. Understanding & measuring research impact. Focus on... 2005 Dec;Issue 2: Primary Health Care Research and Information Service, Adelaide, Australia.

Canadian Health Services Research Foundation

Measuring the impact of research: what do we know? (2008) Part I.

RAND Corporation

RAND Europe. Measuring the benefits from research. RAND Corporation (2006).

Links

Back to top