Our experience introducing Realist Evaluation within a public service agency: development and early use of an experimental Guidebook to help facilitate consideration of realist approaches in Australian Indigenous policy evaluation contexts

Shifts in policy emphasis can have implications for evaluation. As the Australian government’s Indigenous strategies have become more place-based over the last decade, respectful engagement with Aboriginal and Torres Strait... [ view full abstract ]

Shifts in policy emphasis can have implications for evaluation. As the Australian government’s Indigenous strategies have become more place-based over the last decade, respectful engagement with Aboriginal and Torres Strait Islander communities and regions has become more differentiated and of a higher priority. Evaluation is in part, becoming more focused on the core issues of population diversity and effective policy implementation with particular sub-populations. This ‘messy business’ calls for better theoretical articulation and/or more tailored evaluation approaches.

To support further ‘realist’ discussion within the Indigenous Affairs function of the Australian government a Realist Evaluation Introduction and Guidebook ‘What’s so special about Realist Evaluation?’ was prepared for a non-technical readership, using examples in Indigenous policy settings. A recent workshop with staff from our internal evaluation area highlighted that the circumstances of evaluation practice matter and present particular challenges. Deeply reflective literature reviews for realist purposes ideally tease out key mechanisms of change in particular populations and contexts in configuration with sub-contextual features of agency and known outcomes (the classic realist ‘CMO’ approach) in order to test and refine theory. Given the realpolitik of addressing complexity in high profile, high pressure policy contexts that typically strive for more straight-forward solutions and often with limited data, the time and effort available for reflective practice is limited.

Discussion on these issues led to modifications to the Guidebook (using a notion of ‘bounded complexity’) and the further production of a ‘mini’ realist review of indicative literature (along the lines of a Rapid Realist Review) in a particular program area with possible CMOs signalled to facilitate realist reflection and scheduled early discussion between external evaluators and policy and program civil servants. Both products (the Guidebook and the mini review) remain ‘living documents’ to encourage a broader understanding of realist evaluation paradigms and their relevance to contemporary social policy challenges in Australia, particularly in Indigenous settings.
Additional in-house resources and workshops are being considered which will further address particular challenges such as these, and to boost understanding of the strengths and limitations of realist approaches.

Disclaimer: Views expressed throughout this paper/presentation are and will be those of the authors and do not necessarily reflect those of the Australian government. This paper will be delivered in a private capacity and is not an official government presentation or publication.