Realist impact evaluation: when, why, and under what circumstances doe it really have impact?

Realist evaluation aims to inform policy decisions, but evaluators are the first to know that the chances of producing results that will actually be used are slim, because evaluation is ‘tortured by time constraints’... [ view full abstract ]

Realist evaluation aims to inform policy decisions, but evaluators are the first to know that the chances of producing results that will actually be used are slim, because evaluation is ‘tortured by time constraints’ (Pawson 2001). How then can we ensure that a realist impact evaluation actually has impact?
This paper describes the challenges of conducting a realist evaluation of the integration of health and social care. A transdisciplinary, participatory approach (Bergmann et al, 2013; ICPHR, 2013) is being used to gain collaboration across statutory and voluntary sectors and shared learning between commissioners, health providers, and community services. The challenges that have arisen during the evaluation include:
• The original ‘vision of change’ has evolved into a different programme theory every six months
• The actors and funding arrangements changed in the first 12 months
• The average net effect was the main interest for the most influential group of stakeholders, although it could not explain why or how the programme worked
• Case study and qualitative data was assigned a lower value in decisions about how to allocate resources
The solutions so far have rested upon
• Establishing cross-sectoral collaborations and productive working relationships
• Using participatory approaches to enable the ‘people who know’ e.g. those who provide and receive services, to generate and analyse data
• Using knowledge brokering to span the ‘semantic distance’ between different user groups, particularly communities and commissioners who have very different cultures and attitudes toward valued knowledge (Cinq-Mars et al, 2010)
• Assessing user expectations and making the different expectations explicit in order to foster dialogue about the different agendas for using the data (Dagenais et al, 2015)
• Supporting the people who know in developing effective strategies for communicating the utility of different types of evidence to policymakers
• Anticipating what policymakers will want to know and ‘chunking’ the research into shorter cycles evaluating process and implementation for different dimensions of the programme
• Using transdisciplinary approaches to define the limits of knowing and highlight the areas of expertise across academics, commissioners, service providers, and service users
• Expecting – and modeling - co-ownership and collective problem solving
The final challenge is to negotiate the meaning of the term ‘impact’. While commissioners interpret this as cost savings and cashability, clinicians describe it as improved health outcomes and providers interpret it as improved wellbeing. The ability of a realist impact evaluation to deliver on all three will be critically reviewed, alongside the approaches that are being used to set and adjust expectations for the project.