Big data meets small data in that footprint lab in the sky

When he asked the question “Does the Flap of a Butterfly’s Wings in Brazil Set Off a Tornado in Texas?” Edward Lorenz reminded us that everything is connected and that we are all part of the web that makes up life on... [ view full abstract ]

When he asked the question “Does the Flap of a Butterfly’s Wings in Brazil Set Off a Tornado in Texas?” Edward Lorenz reminded us that everything is connected and that we are all part of the web that makes up life on this planet. His question suggested that we must pay attention to small data as well as to big - the micro and the macro. Lorenz's work depended on computer modeling of atmospheric dynamics backed by observation. Writing in 1972 he concluded that with the ever-increasing power of computers and improvement of knowledge, one day his question would be answered.
Our work on bringing together big picture economic and social data and small-scale observations has also depended on increased computer power and advancements in knowledge. However with today's speed of change we have not had to wait so long. When we first began our work at the University of Sydney around the year 2000 we could build models of a single nation's economy and examine, at a macro level, a limited number of social and environmental effects of doing business. By 2013, pushing at the edge of computing capacity and stretching the research agenda, we built a model of the world's economy and examined, albeit still at a macro level, a much wider range of social and environmental indicators. For example we quantified the effects of global trade on issues such as poverty, inequality, well-being or worker safety.
Meanwhile qualitative social researchers pursued micro level research documenting, for example, the practices of local organizations or addressing difficult social issues. The quantitative and the qualitative shared little common ground; big data and small data, the macro and the micro seemed destined to continue in their separate spheres of existence.
However 2015 saw exponential growth in computing capacity. And thanks to a small community of researchers across the globe, the know-how to include in a big data model of the world economy the particulars of social and environmental on-the-ground research became possible. This meant that for the first time big and small data could be handled within the same framework.
This paper discusses the bringing together of qualitative social research and quantitative input-output analysis. It illustrates the need to work together to realise the power of combining big and small data within a single framework that enables trade offs to be examined and well-being to become a recognised part of progress.