Using Big Data to Transform Health: The Importance of Evaluation Frameworks

European governments face a growing number of major health challenges, which are putting unprecedented pressures on public health systems. These include the ageing population, with rising prevalence of chronic disease, and... [ view full abstract ]

European governments face a growing number of major health challenges, which are putting unprecedented pressures on public health systems. These include the ageing population, with rising prevalence of chronic disease, and multi-morbidity, which is largely, but by no means exclusively, associated with unhealthy lifestyles. The expected population rises show that by 2050 the number of individuals aged 65 and over is expected to rise from 129 million to 224 million and the number of individuals aged 85 and over is expected to rise from 14 million to 40 million.

The challenge of providing quality healthcare and more effective allocation of services to an increasing and ageing population requires that health policy makers have the ability to access timely and evidence based insights derived from existing health, social and other data sets.  Enormous volumes of data are currently generated across multiple healthcare contexts, as well as at the national level.  These complex big data sets exist in many formats and structures and are characterized not only by their volume, but also by their variety, velocity and particularly in relation to healthcare, by their veracity (Connolly and Woodledge 2013; Courtney 2013. This data generation has been driven by the need for record keeping, by compliance & regulatory requirements, and by disease surveillance (Raghupathi 2010; Burghard 2012).  It provides the opportunity to manage population health more effectively as correct analysis of individual and population data can provide a richer understanding of the factors that influence outcomes, which can guide physicians regarding the most effective treatment option for a patient at the point of care.  It can equally guide policy makers to make more informed decisions regarding the most effective allocation of resources (Burghard 2012). This has potential to improve individual quality of care in addition to broader health outcomes across society, as well as reduce costs associated with the delivery of such care.  For example, the use of big data analytics in healthcare can provide insights into patient characteristics through profiling, segmentation and predictive modeling.  It can be used to identify patient cohorts who are most likely to benefit from particular interventions including preventative care; it can provide disease profiling to predict events and to proactively ensure that appropriate preventative plans and resources are in place.  It also can assist in identifying fraud detection and health claim authorization as well as many other benefits.  In addition to data generated via healthcare contexts, other data forms including data gathered from social networks is emerging as a key dimension in improving health and wellbeing scenarios. When such data sources are combined with health data, they have potential to provide even more granular insights to assist policy makers.

However, despite awareness of these benefits, big data has played a limited role in the health sector, particularly when compared with its application in other domains. The reasons for this relates to the data characteristics including the fact that the data are heterogeneous, in individual silos; the data are diverse and complex, the fact that such data are collected in different ways using different techniques; and the fact that there are real data protection and data governance issues to overcome.  Most importantly of all, no tools exist to make these data accessible to end-users. There is an urgent need to develop better applications and tools to consume and map data and make it more meaningful, insightful and useful for health policy makers. (Raghupathi and Raghupathi 2014). 

This paper has two objectives: firstly it outlines the development of such a tool: a big data platform that is currently being developed via a partnership involving 5 regional health authorities in 3 member states, and technical big data experts from Universities and the private sector.  A plurality of dynamic data sources linked to the health and social data of over 15 million European citizens is being accessed and analysed. This platform will successfully facilitate the utilization of healthcare data beyond hitherto-isolated systems and makes that data amenable to enrichment with heterogeneous, external data.

The second objective of this paper is to outline the rationale for and development of an evaluation framework that is currently being deployed to support effective utilization of this platform.  Despite their potential benefits, many big data projects fail and the literature is replete with examples of projects that fail due to people issues rather than technology problems. One of the main reasons for this failure relates to wrong decisions regarding the data and in particular the questions that it should be able to answer.  For example, data science is a complex mix of domain knowledge, mathematics and statistics expertise and programming skills.  However, those who develop a system are not the individuals who will ultimately adopt (or resist using) that system. In order to ensure that the user needs guide each stage of the development of the technology, a rigorous and ongoing evaluation of ICT is of great importance for decision makers (policy formulators) and end users of the technology (Kaplan and Shaw, 2002).  Health informatics evaluations can therefore provide an objective measurement of processes and outcomes against expectations, with the intention of identifying strengths and successes, whilst finding means of addressing and improving weaknesses or even failures (Rigby, 2006). These evaluation frameworks do not seek to provide explanations, but “describe empirical phenomena by fitting them into categories. Frameworks usually denote a structure, overview, outline, system, or plan, consisting of various descriptive categories, e.g. concepts, constructs, or variables, and the relations between them that are presumed to account for a phenomenon (Nilson, 2015).  At present, there are no gold standard frameworks of evaluation theory and practice (Rahimi & Vimarlund, 2007). Issues that cause problems for undertaking these evaluations are partly due to health information system complexity with respect to selecting a framework to be applied (Friedman et al, 1997; Ammenwertrh et al, 2003). This may explain why despite an increasing number of health information system that have been developed, the number of published evaluations remains limited (Ammenwertrh et al, 2003; Van der Meijden, 2003).

The current study employs a multiple case study mixed-method evaluation framework (Caracelli et al, 1997).  The objective is to evaluate possible impacts in collaboration with our partners, policy board, and other stakeholders and identify gaps between framework user’s expectations and requirements with their actual needs. An online questionnaire will be developed using Q-methodology to examine continued progress against expected impacts as defined through grant agreements, logic models and initial interviews. This multimethod approach to health ICT evaluation is one which views qualitative and quantitative research methods as complementary (Kaplan 2001; Westbrook et al, 2007). A logic model (Kellogg Foundation, 2004) for the framework was developed with stakeholders to specify expected outcomes, outputs and impacts for the duration of the study. Essentially the model depicts the logic of translating the technical and programmatic inputs of the integrated information system; how these inputs produce quality through liaising with stakeholders. Semi structured interviews will be undertaken with the stakeholders to evaluate the gap in knowledge / expectations between software developers and end users, policy makers) through four case studies, each with one demonstration project on four separate occasions over the projects life cycle. An online questionnaire will be developed using Q-methodology (Watts & Stenner,2005) will be used to explore more deeply the values and understanding of the stakeholders over time, and the tension between various aspects of the project.  Combining these methods allows exploration of the what, why, and how social phenomenon that qualitative methods can address, and the size, extent or duration (how much) of certain phenomena that quantitative methods establish (Stoop et al, 2003).  The longitudinal study design will provide a means of monitoring changes caused by the system within the context it operates (Ammenwerth et al, 2003).

This study contributes to both practice and theory. From a practical perspective, no tool or solution currently exists that utilises the variety of data from the public, patients and healthcare systems to provide actionable insights to public health policy makers. Using a model of data value co-creation, this study will set in motion a new network of knowledge that enables health systems stakeholders to exploit the opportunity of big data for public health strategy and individualized citizen benefit.  The resulting solution will catalyze evidence-based policy decisions and strategic healthcare plans, which will ultimately improve public health and the quality of life amongst European citizens.  The paper also contributes to theory through its evidence-based demonstration of the importance of an evaluation framework in such a context and in particular by identifying the challenges experienced with employment of a multi-method framework in a big health data context.

The study findings provide much needed insights that can guide those involved in development of technology projects and seek to ensure more successful alignment with user needs, thereby ensuring more successful adoption outcomes.  The multi-method evaluation framework employed and the discussion of its benefits and challenges also provide insights for those involved in such framework development