Used with specific permission from Polyp, please do ask before reproducing.
Used with specific permission from Polyp, please do ask before reproducing.
This is a guest post by David Hales, a fellow associate of the new complexity think-tank, Synthesis. David specialises in computational social science and here he provides a thought-provoking response to the rise in big data, and some of the more outlandish claims made about it. For a good example of the latter, see Chris Anderson’s piece ‘The Data Deluge Makes the Scientific Method Obsolete‘. In this piece, David makes some very relevant points for development big data initiatives.
Almost everything we do these days leaves some kind of data trace in some computer system somewhere. When such data is aggregated into huge databases it is called “Big Data”. It is claimed social science will be transformed by the application of computer processing and Big Data. The argument is that social science has, historically, been “theory rich” and “data poor” and now we will be able to apply the methods of “real science” to “social science” producing new validated and predictive theories which we can use to improve the world.
What’s wrong with this? On one level nothing. We know so little about the social world that anything is worth a try. Mining these huge databases will almost certainly lead to new ideas and insights. However, before we run headlong into this new world of big data, promoted as it is by corporations such as IBM and the large consultancies, perhaps we might benefit from a little critical reflection.
Firstly what is this “data” we are talking about? In it’s broadest sense it is some representation usually in a symbolic form that is machine readable and processable. And how will this data be processed? Using some form of machine learning or statistical analysis. But what will we find? Regularities or patterns (for a useful discussion of patterns within complex systems, see Greg Fisher’s post, Patterns Amid Complexity). What do such patterns mean? Well that will depend on who is interpreting them.
Given this level of generality, if someone tells you they are working on “big data” it tells you almost nothing. One way to approach the issue if confronted with a “big data” project is to ask the following question based on a thought experiment:
Imagine you had a massive computer database that contained all possible measurements that could ever be made over the entire span of all space and time. You could query it with any question and it would deliver the result instantaneously. All big data is merely a subset of this ‘the biggest data that could ever exist’. What would your project ask it?”
If no coherent answer can be produced to this question then any such project is at best directionless and at worst not conscious of its aims.
One answer might be “looking for patterns or regularities in the data”. Looking for “patterns or regularities” presupposes a definition of what a pattern is and that presupposes a hypothesis or model, i.e. a theory. Hence big data does not “get us away from theory” but rather requires theory before any project can commence.
What is the problem here? The problem is that a certain kind of approach is being propagated within the “big data” movement that claims to not be a priori committed to any theory or view of the world. The idea is that data is real and theory is not real. That theory should be induced from the data in a “scientific” way.
I think this is wrong and dangerous. Why? Because it is not clear or honest while appearing to be so. Any statistical test or machine learning algorithm expresses a view of what a pattern or regularity is and any data has been collected for a reason based on what is considered appropriate to measure. One algorithm will find one kind of pattern and another will find something else. One data set will evidence some patterns and not others. Selecting an appropriate test depends on what you are looking for. So the question posed by the thought experiment remains “what are you looking for, what is your question, what is your hypothesis?”
It seems to me that one must at least try to answer this question if one is to pursue social science. Not just because it is good science but also because it has ethical and political implications. The view one takes of social phenomena, either consciously or through algorithms and data, frames what is and is not conceivable for past and future social reality. If you doubt the importance of such ideas one should look that the history of the 20th century. Ideas matter. Theory matters. Big data is not a theory-neutral way of circumventing the hard questions. In fact it brings these questions into sharp focus and it’s time we discuss them openly.
Right now we are “data rich” and “theory poor”. We need new theory for the 21st century. That requires critical discussion, reflection, honestly and humility. It is not clear to me that such concerns are prominent within much of the “big data” movement.
Here is a more eloquent and playful take on these issues, by a colleague of mine, in the genre of that wonderful Orwell fable: https://scensci.wordpress.com/2012/12/14/big-data-or-pig-data/
Cross-posted from the Synthesis blog.
Innovation is popular in aid at the moment, so much so that there is a steady spate of articles which range from trashing its potential contribution to development through to challenging Western, donor, countries’ assumed roles as the ‘providers’ of innovation.
In this post I want to argue that there is a middle ground between the unthinking mantras that are increasingly peddled by agencies and the growing number of entirely justifiable critiques.
An Economist article made the point succinctly over a decade ago, ‘what precisely constitutes innovation is hard to say, let alone measure.’ Some concluded, as a result, that innovation was a ‘new theology’.
However, with a growing chunk of economic growth being driven by industries and products that in fact didn’t exist ten years previously, such dismissals seem increasingly Luddite.
While clarity and precision in thinking about innovation is all-important, it is far from easy. We are not helped by the fact that many innovation stories are in fact apocryphal – retrospectively woven to lend the star protagonists much more agency and awareness than in fact they possessed. This is true of even the best known innovation stories. Take Alexander Fleming’s infamous and much-lauded discovery of penicillin – Fleming himself used to describe the conventional account of his contribution as the ‘Fleming Myth’.
Typically in business the market is the ultimate arbiter of innovation, and as we know, most products fail. In aid, however, the market does not provide an adequate indication of what is successful and what is not. This is largely done by the aid system itself. Bill Easterly made much of the fact that the market could get Harry Potter books anywhere there was a demand, which he compared unfavourably to the inability of the aid system to get simple treatments like vaccines to where they were most needed:
There was no Marshall Plan for Harry Potter, no International financing Facility for books about underage wizards. It is heartbreaking that global society has evolved a highly efficient way to get entertainment to rich adults and children, while it can’t get twelve-cent medicine to dying poor children.”
But as Amartya Sen subsequently argued:
the disparity in the results is indeed heartbreaking… [but] there is a radical difference… between the enterprise of supplying “what is in demand” — which is integrally linked to the buyers’ ability to pay — and that of supplying needed goods and services to people whose income and wealth do not allow a need to be converted into a market demand.
While Sen’s point applies more broadly to aid delivery, it is also relevant to new ideas and innovations within aid.
In any case, whether because of market failure or the wilful self-interest of aid agencies, innovation – which is an ambiguous enough concept in the business realm – becomes very murky territory indeed in development. It is hard to say what innovation actually is, what it generates, and for whom. Like the famous ruling about pornography, many are of the view that ‘I know it when I see it.’
Such vagueness is the ideal seeding ground for development fads, and indeed, innovation is fast becoming the latest ‘fuzz-word’. Everything is being labelled ‘innovation’: as one blogger memorably put it, we seem to be suffering from Innovation Tourette’s.
Little wonder that growing numbers of thinkers and writers see the need to beat innovation with a big snarky stick. These criticisms play a vital role in highlighting the risks and downsides of all shiny new aid agendas – and innovation is no exception.
Having observed such trends in the past, I think there is a danger that between the rise of the fad and the indignant reaction to it, we lose sight and sense of why the issue is question is actually important. Specifically, we risk learning the wrong lessons about what innovation actually is, and the potential it has to add to our work.
What we need is a more precise and accurate way by which to separate the innovation wheat from the faddish chaff. This was in fact one of the key motivations of a study I co-authored with Kim Scriven and Conor Foley while at ALNAP back in 2008-9.
So what did our study suggest in terms of getting more precision in innovation? We found it useful to ask some key questions to identify whether a particular idea or approach was in fact innovative.
There are no doubt many more questions that could be asked, but the above provide a good starting point for what might be termed ‘innovation due diligence’. The key, in my opinion, is to use these and others questions to develop more honest, rigorous stories about ongoing and historical innovations: about how they came about, why, and with what benefits. Such questions are useful because they help us look at innovation from a more systemic perspective: looking not just a idea, but the overall social, technological and institutional context from which it has emerged.
These questions should be relevant whether you are a donor bombarded with new proposals and ideas, an operational aid worker seeking to get funding for your exciting new idea, a blogger wanting to shine a light on the depressing excesses of innovation-speak, or a researcher wanting to investigate an supposed innovation in a systematic fashion (in fact I think we need far more of the last category, but that’s another story).
We will need to keep wielding the big stick as necessary, to curb against such excesses of aid ‘innovation-speak’. But we may also at times need a magnifying glass and ruler – metaphorically speaking – and asking these kinds of questions could help with this. My $0.02 is that if a would-be innovator can’t take a reasonable stab at these questions, they aren’t working hard enough, or they are over-selling something. A lot is spoken about creativity in innovation, but recent work suggests – in echo of the old 1% inspiration, 99% perspiration line – that the larger part of innovation lies in the proper execution of the idea.
I think there is a special role for the aid blogging community in asking such questions and demanding answers. We have seen in the past few years how bloggers have mobilised in a largely self-organised fashion to push back against various poorly considered ideas.
I know there are many bloggers who want to engage with innovation in a serious fashion, and who are dismayed by the current hype surrounding it. We should be able to highlight the good and bad of what we see emerging from the aid innovation agenda. And aid agencies should be willing to open their ideas up to the views and scrutiny of this emerging, globally networked, community of thinkers and analysts. This kind of effort has, in other distributed sectors, developed into new crowd-sourced marketplaces for innovation such as Innocentive. There’s no reason why the same couldn’t happen in our sector.
Our ultimate goal, I’d argue, should be to work to bringing the perspectives of aid recipients into the mix as part of our standard operating procedures. Now that in itself could be seen as a real innovation.
The need for such engagement goes beyond mere niceties. The most effective ideas we uncovered in our 2009 study were precisely those that re-thought and re-formulated this core aid relationship: cash, community approaches to malnutrition, transitional shelter. Put simply, these were the innovations that we found to be most worthy of the term.
Posted in Accountability, Economics, Evaluation, Evolution, Influence, Innovation, Knowledge and learning, Leadership, Networks, Organisations, Public Policy, Reports and Studies, Research, Technology | 5 Comments »
From that Aid on the Edge of Chaos favourite, Saturday Morning Breakfast Cereal.
Resilience has become popular in foreign aid, but it is far from easy. Previous posts on this blog have discussed challenges of applying the ideas – which emerged from numerous places including ecological sciences – to development and humanitarian aid (see here and here).
A New Scientist editorial piece on the aftermath of hurricane Sandy highlights the fact that the challenges are not unique to our sector. It usefully outlines what I call ‘resilience risks’ – shortcomings which may arise from narrow or simplistic applications of resilience concepts. This post shares the three key resilience risks – which I hope may prove relevant for aid practitioners, researchers and policy makers working on these issues.
Resilience risk 1: Resilience is analysed in highly linear ways
A key critique made in the NS piece was that the potential risks from Sandy were thought about, anticipated and planned for in a linear and simplistic fashion:
Danger of fire? Equip fire departments. Possible electricity failure? Turn off transformers and give hospitals generators. Risk of floods? Build barriers. But in New York City all three risks hit at once, and then some. Houses burned because firefighters couldn’t get to them or operate equipment. Electricity substations exploded as record floods hit. Two hospitals were evacuated as backup generators failed.”
This analysis resonates with a previous post on this blog focusing on complexity and disasters – to quote directly:
Consider the following three ingredients: a mega-city in a poor, Pacific rim nation; seasonal monsoon rains; a huge garbage dump. Mix these ingredients in the following way: move impoverished people to the dump, where they build shanty towns and scavenge for a living in the mountain of garbage; saturate the dump with changing monsoon rain patterns; collapse the weakened slopes of garbage and send debris flows to inundate the shanty towns. That particular disaster, which took place outside of Manila in July 2000… was not inherent in any of the three ingredients of that tragedy; it emerged from their interaction’
Taken to extremes, the linear approach diminishes the potential relevance of resilience efforts. Work by the Stockholm Resilience Centre shows that narrow approaches to resilience can actually heighten rather than reduce vulnerability.
Despite the widely discussed notion that resilience should be a ‘unifying concept’ to bring together different disciplines and approaches, the reality is that the old silos may simply be too rigid to enable such cross-fertilisation.
Resilience risk 2: Resilience is only thought about after crises
The editorial also drew comparisons which will no doubt be familiar to regular readers of this blog:
…the lesson of Sandy is the same as the lesson of the Eurozone crisis and other recent events such as the Egyptian revolution: complex systems play by their own rules. You can’t manage them in a linear way. We live in a web of systems: if one falls, it takes others with it… as climate change bites, there will be more and bigger storms, and other mega-events such as crop failure, political instability and financial crises. The knock-on effects will accumulate. All complex networks are susceptible to collapse. How many body blows can ours take before it can no longer stand back up? (emphasis added)
While this suggests that there is more need for anticipation in our resilience work, and more continuous and ‘joined-up’ resilience thinking, the reality is that we only tend to think of resilience when it is too late. This is what I have elsewhere called the ‘catastrophe-first school of lesson learning’.
Moreover, even when we do try, anticipation is often limited by the too-common tendency to fight the last war. The reality is that the crises of the future are likely to be very different to those we experienced in the past.
Resilience risk 3: Resilience is seen as ‘the money saving option’
Finally, the piece shared some sobering thoughts for initiatives that push hard in the ‘resilience equals value for money’ direction.
[networked] systems can be made more resilient [but] to do so will be expensive. Money-saving efficiency would have to be sacrificed for more redundancy… Resilience may be expensive, but as Sandy showed… we need a lot more of it. [emphasis added]
Clearly, the value for money approach is not going to go away any time soon. But in pushing resilience forward, we have to be careful not to oversell it, raise unrealistic expectations, and thereby diminish its actual contribution. The key I think is not to take too limited a view: what is needed is less of a bean-counting approach, and more of a long-term perspective of the value of resilience.
The overall lesson for those of us working on resilience in development and humanitarian aid may be that we need to find ways of moving away from an overly reductionist approach. Multi-dimensional analytical frameworks (such as the disaster resilience model I developed for DFID in 2011) are an important starting point. But these need to feed into policy and practice, and this is much harder – as suggested by a recent thoughtful (and at times delightfully grumpy) ODI think piece.
My sense is that, to date, efforts which attempt to take a more ‘systemic’ approach to resilience are a bit thin on the ground in foreign aid. As a result, I would argue that we are currently leaving our efforts wide open to all three of the ‘resilience risks’ raised here.
Do others agree, and if so, what might be done? Or am I over-stating the potential impact of these risks on resilience efforts in development and humanitarian work? I’d be interested to know what readers think.
So after billions of dollars and several years of hard campaigning, the US elections are finally over. The typical map of the 2012 US election results looks like this:
Which is clearly not a million miles away from the 2008 equivalent.
In these maps, of which there are thousands online, on TV shows and in newspaper reports, the US states are coloured red or blue according to whether the majority of their voters were Republican or Democrat.
These maps, are, of course an illusion. They suggests that the ‘reds’ might have won because there is more red on the map, and that the reds and blues are sharply divided. Typical comments about such maps run along the lines of “what a huge sea of red”, “there you go, the liberal-conservative divide”, “it really is two different countries, isn’t it?”, and so on.
However, these maps fail to take account of some basic realities. First of all, there is no representation of population. The reality is that the population of the red states is on average significantly lower than that of the blue ones. So while the blue are small in area, they represent large numbers of voters. Second, more importantly for the results of elections, the maps take no account of the distribution of electoral college votes. Third, they take no account of the often fine-grained distribution of voter preferences within states.
Mark Newman, a noted complexity researcher, has done a lot of work on how we can get more realistic, less simplistic maps of complex, real-world phenomena. By drawing such cartograms, which enable maps to be re-scaled according to key variables like population, maps of the electoral spread can be made more realistic and detailed. They can also tell different, more subtle, stories about political allegiance.
By the sounds of things, he is busy working right now on maps of the 2012 election. Here is his depiction of the 2008 election using a population cartogram.
In this, the states have been squashed and stretched to give relative sizes while preserving the overall US structure. A similar thing can be done with the electoral college results. In the map below, the map scales the sizes of states to be proportional to their number of electoral votes in 2008.
As Newman writes:
The areas of red and blue on the cartogram are now proportional to the actual numbers of electoral votes won by each candidate. Thus this map shows at a glance both which states went to which candidate and which candidate won more electoral college votes – something that you cannot tell easily from the normal election-night red and blue map.
Newman and his colleagues went further to map the election results by county, the resulting images are even more striking. This is the equivalent of the first map above, with each county coloured red or blue according to the majority vote in 2008.
Again, the red appears to be in the majority. Using a cartogram of population gives this:
All of these maps are however also somewhat fictional as they pay no attention to the fact that no single state is in fact a sea of red or blue. Instead, as this election showed, every county and state contains quite closely balanced numbers of Republican and Democratic supporters. By using only two colours we lose any sense of this balance, and feed the myth of red states and blue states, and of sharp country-wide divides.
Newman and his colleagues have got around this by using red, blue, and shades of purple in between to indicate the nuance in voting patterns: different shades of purple indicate different splits of votes.
This is the county level map with this applied:
And this is the population cartogram:
As Newman explains:
As this map makes clear, large portions of the country are quite evenly divided, appearing in various shades of purple, although a number of strongly Democratic (blue) areas are visible too, mostly in the larger cities. There are also some strongly Republican areas, but most of them have relatively small populations and hence appear quite small on this map.
What I love about this work is that it clearly demonstrates the power of maps and visualisations to shape our thinking. These depictions pose direct and clear challenges to those lazy, pervasive but ultimately unhelpful narratives (“sea of red”, “lib-con divides”, “country of two parts”, etc, etc).
I think that these more realistic, sophisticated representations should become much more commonplace in politics and indeed in development. Mark Newman set up the World Mapper project back in 2006, which has a whole host of similar maps, many of which have been widely used in presentations and reports.
Much of this work owes a debt – of sorts – to the infamous and controversial Gall-Peters projection, which provided a new visualisation of the earth using a more egalitarian and precise calculation of the relative landmass of developing countries.
Along broadly similar lines, a recent guest post on this blog looked at how we might use tools like fitness landscapes to more accurately represent non-linear development progress.
Perhaps such tools could slowly help change the way we think about a whole range of complex, routinely over-simplified, phenomena.
Who knows, one day they may even help inform some less divisive narratives about the US political landscape. As President Obama put it this morning in his acceptance speech:
We are not as divided as our politics suggest. We remain more than a collection of red states and blue states.”
Postscript on 8th November 2012: the 2012 election maps are now done, and here is the 2012 county cartogram.
This guest post by Andy Sumner and Sergio Tezanos Vázquez explores new approaches to classifying developing countries, based on a new IDS Working Paper published last week. In the paper, they develop a more precise and accurate classification system for low and middle income countries, and suggest that this can support a more complex, non-linear appreciation of development trajectories. This post concludes with the intriguing notion of ‘development landscapes’ (building on a recent CGD blog post written by Owen Barder and myself).
[t]he typical case is a largely unindustrialised economy, the foreign trade of which consists essentially in selling primary products for manufactures. There are about 100 identifiable economies of this sort, covering the great majority of the world’s population (1963, p. 80).
And perhaps most famously, Seers wrote in The Meaning of Development:
The questions to ask about a country’s development are therefore: What has been happening to poverty? What has been happening to unemployment? What has been happening to inequality? If all of these three have become less severe, then beyond doubt this has been a period of development for the country concerned […] If one or two of these central problems have been growing worse, especially if all three have, it would be strange to call the result ‘development’, even if per capita income has soared (1969:24).
Since then many have challenged the use of income per capita as the primary proxy for development. Of course, in addition to low and middle income countries there are many classifications – notably those by Human Development and the Least Developed Countries classifications and the new pioneering work of the Oxford Poverty and Human Development Initiative.
Our new paper continues this tradition with what might be described as a complex systems twist. The paper challenges the continuing use of income per capita to classify developing countries as low income countries (LICs) or middle income countries (MICs), given that most of the world’s poor live in the later group (see here,here, here, here).
Further, it highlights the ambiguity over the usefulness of the MIC classification given the diversity in the group of over 100 countries that includes Ghana and Zambia, as well as India, China and Brazil.
Finally, it points to a new way of framing development, which moves beyond the linear league table approach which tends to imply a certain shared pathway (cf critiques of HDI as ranking countries on a scale of 0 to Denmark). In closing, we touch on one example of what this might look like below: there are no doubt others.
We used a cluster analysis to identify five types of developing country using a set of indicators for 2005-2010 covering definitions of development based on the history of thinking about ‘development‘ over the last 50 years from four conceptual frames:
This is what we found:
Our development taxonomy differs notably from the usual income classification of GNI per capita (Atlas method) used to classify LICs and MICs. Notably many countries commonly labelled “emerging economies” are not in the emerging economies clusters because they retain characteristics of poorer countries.
Our analysis generated 5 clusters as follows:
Cluster 1: High poverty rate countries with largely traditional economies. Those countries with the highest poverty and malnutrition headcounts, who are also countries with low productivity and innovation and mainly agricultural economies, with severely constrained political freedoms. This cluster includes 31 countries, some of them might be surprising: Pakistan, Zambia, Nigeria, and India.
Cluster 2: Natural resource dependent countries with little political freedom. Those countries with high dependency on natural resources, who are also countries with severely constrained political freedom and moderate inequality (relative to the average for all developing countries). This cluster includes 9 countries, such as Mauritania, Vietnam, Yemen, Cameroon, Congo, Swaziland and Angola.
Cluster 3: External flow dependent countries with high inequality. Those countries with high dependency on external flows, who are also countries with high inequality, and moderate poverty incidence (relative to the average for all developing countries). This cluster has 32 countries, such as Senegal, Ghana, Indonesia, Thailand, Peru, Colombia, and Panama.
Cluster 4: Economically egalitarian emerging economies with serious challenges of environmental sustainability and limited political freedoms. Those countries with most equal societies, who are also countries with moderate poverty and malnutrition but serious challenges of environmental sustainability and –perhaps surprisingly– limited political freedoms. This cluster has 15 countries, including China, Azerbaijan, Belarus, and Kazakhstan.
Cluster 5: Unequal emerging economies with low dependence on external finance. Those countries with the lowest dependency on external finance and who are also countries with the highest inequality. This cluster includes 14 countries, such as South Africa, Botswana, Costa Rica, Malaysia, Argentina, Mexico, Brazil, Turkey, Chile and Uruguay.
Two-thirds of the world’s poor – not surprisingly given the characteristics noted above – live in Cluster 1 countries though this is largely due to the inclusion of four populous countries (Bangladesh, India, Pakistan and Nigeria –and one should remember a third of world poverty is accounted for by India). About a quarter of world poverty is situated in Cluster 3 and Cluster 4 countries and the remaining 5% live in Cluster 2 and Cluster 5.
What is most striking – and of particular relevance for readers of this blog – is that we find that there is no simple “linear” representation of development levels (from low to high development countries), as found in the Human Development Index or its variants.
We find instead that each development cluster has its own and characteristic development issues. Building a development classification is not a simple task: once we overcome the over-simplistic income-based classification of the developing world, we find that there is no group of countries with the best (or worst) indicators in all development dimensions.
It thus would be more appropriate to build “complex” development taxonomies on a five-year basis than ranking and grouping countries in terms of per capita incomes or other indicators, as this will offer a more nuanced image of the diversity of challenges of the developing world and policy responses appropriate to different kinds of countries.
In keeping with a growing consensus in development thinking, this way of thinking points to an understanding which is less about linear progression, and where change is path dependent, and development policy is best seen as an evolutionary process of learning.
What might such a complex taxonomy look like? A fruitful avenue to explore (following the recent post by Ben Ramalingam and Owen Barder on complexity and results) might be the notion of fitness landscapes from evolutionary biology, which illustrate how different species attain biological fitness through processes of variation, selection and amplification.
Erik Beinhocker and Tim Harford have argued that growth and innovation can be likened to a evolving search of a dynamic fitness landscape. “Development Landscapes” might provide an analogy and a model for explaining the distinct but related positions of countries in different development clusters.
This would enable analysts and policy makers to examine the different pathways the cluster members have been on, and to explore the space of possibilities for future change.
About the guest post authors:
Sergio Tezanos Vázquez is an associate professor at the Economics Department and a research fellow at the Iberoamerican Research Office on International Development & Co-operation at the University of Cantabria.
Andy Sumner is Co-Director of the newly established, King’s International Development Institute, King’s College London.