Building Capacity With Practitioners To 'Realistically' Monitor and Evaluate Their Own Physical Activity Programmes

This paper aims to critically explore whether Realistic Evaluation principles enable practitioners in the field of physical activity to Monitor and Evaluate their programmes. The paper draws upon the findings of an evaluation... [ view full abstract ]

This paper aims to critically explore whether Realistic Evaluation principles enable practitioners in the field of physical activity to Monitor and Evaluate their programmes. The paper draws upon the findings of an evaluation design utilising Q and realist methods to test a participatory M and E framework delivered at a south coast university.
The topic of M and E has drawn considerable interest in recent years in sport and physical activity programmes (Coalter, 2007, 2010). The interest and subsequent critique has focused on scepticism around whether they are able to facilitate change, how they are monitored and evaluated and who should be involved in these processes. A lack of evidence discourse (Nichols et al, 2010) has emerged which raises a series of issues around the capacity of these programmes. In order to move the field forward it has been suggested that a deeper understanding of what works for whom and why (Pawson and Tilley, 1997) is required to make sense of the mechanisms that may lead to certain outcomes in programmes.
On this backdrop, this study explored the crucial role of practitioner involvement and accountability in the M and E process. It drews upon a participatory approach, training a sample of student sport development practitioners in M and E techniques at a south coast University. The production of an evaluation framework which embeds realistic evaluation techniques (Pawson and Tilley, 1997) was trialled with the practitioners who were monitoring and evaluating their own projects in the local community. The framework consisted of workshops and action learning sets built around a common set of collaborative principles for evaluation (Schula et al, 2016) to build the capacity of practitioners.
The framework was evaluated to test the engagement with realist evaluation and to what extent the collaborative principles facilitated this. Testing consisted of a new and innovative realist evaluation methodology that synthesised realist approaches with Q methodology. The evaluation design consisted of administering a Q sort and factor analysis to understand subjective view points underpinning the practitioners, which led to the production of holistic narratives distinguishing the different groupings of practitioners in relation to what worked for them, why and how. This was then followed and clarified through realist designed interviews and blogs which asserted context mechanism outcome configurations refining the holistic narratives asserted by Q.
The findings of the evaluation uncovered four different groupings (factors) of practitioners based on what worked for whom in what circumstances and why. Findings asserted positive outcomes and mechanisms for those engaging in realist approaches, but based on context, different practitioners reasoned and responded to the framework in different ways. Insight was gained into the significance of the collaborative principles underpinning the framework. These mechanisms are duly discussed.
In conclusion, this paper establishes two key contributions. Firstly, it provides empirical insight around how appropriate realist evaluation is for practitioners working in practice provided appropriate support, training and collaborative mobilisation is achieved. It secondly (on a methodological level) strongly advocates the benefits associated to combining realist evaluation with Q.