Your post really adds some thinking material to our recently started initiative- we would like to see whether torrents of operational data that global development organizations create each day through implementation of projects (e.g. contracts, amounts, suppliers, etc) could be better used to improve operational effectiveness of UNDPs and WBs of this world.

Our idea is to start exactly where you suggest- refine questions. We will organize a series of data dives (http://europeandcis.undp.org/blog/2013/01/11/can-big-data-help-deliver-better-operational-results/) where we would bring our organizational data sets to programmers and ask them whether what questions we could answer give the available data (can we predict likelihood of corruption in projects, what leads to under-delivery of certain projects, are there any companies that tend to win majority of contracts in any given sector and what are their linkages to other suppliers, etc), are we not asking questions we should be asking, and are we not collecting data we should be collecting.

We tried to take a stab at few of these issues two weekends ago, looking at the World Bank data set of major awarded contracts from 2007-2013 and came up with some really interesting results (http://europeandcis.undp.org/blog/2013/01/31/big-data-and-development-organizations-what-happens-when-you-move-from-theory-to-practice/).

So in a nut shell, I think we’re just at a tip of the iceberg, but any steps forward need to be grounded in asking proper question. And just as a side note no your comment on theory and data- I’m reading ‘Antifragility’ by Nassim Taleb, where he makes an argument that we first DO and then BUILD theory (as opposed to having theory then building something out of it) and I think in the context of big data that’s all the more relevant. We aren’t even aware of all the value we can generate, in our case operational data- we know some but not all. However, as we go along, as we start tinkering with the data, chances are we’ll know more and we’ll be able to contribute to social science.

Again thanks so much for a super interesting post, really enjoyed reading it!!
@ElaMi5

I haven’t encountered anyone who says we should just look for patterns in data and not root causes.

On the other hand, I’ve had to endure quite a few people who come up with complex theories and models from first principles, and feel no obligation to test them against real world data, nor could they pass the test.

Thanks for all the thoughtful comments on the post. Basically I think I ought to clarify. I am not against data or machine learning. What I was aiming at was the concept of “big data” as something that is somehow qualitatively different. That somehow just having a lot of data will make new social theories and models appear. I’ve worked extensively with many machine learning methods and they are great for finding certain patterns in data. These might be decision trees formed by recursively splitting the input attribute space based on information gain over some assigned class of interest. They may be genetic algorithms for which some fitness function or method has been assigned. You can even evolve, say, genetic programs that code dynamic behaviors.

But what these algorithms can do does not support the great claims of “big data”. Big data is a research project in machine learning not a new way to dispense with theory. Theory directs what to look at, what data to collect and how to view it – interpret it.

My background is in Artificial Intelligence and as a grad student working on machine learning and neural networks, at the time, I remember the big neural network hype. People said, we just need to put a massive neural network in a robot and give it all the human senses (all that data) and it would learn to do intelligent things – like interpreting it’s environment, walking and talking. Yes really. There was no new theory there, it was argued that the system would magically emerge these results. The ideas was “intelligence without representation”.

Well, it didn’t work. Don’t get me wrong, very interesting things were done. But the hype turned out to be that.

I’m not against people trying things – as I said in the post we know very little about how social systems work – but I don’t think we should suspend critical evaluation. Those promoting big data should try (at least try) to describe what kind of things they are looking for. I don’t feel saying “big data” and “machine learning” is really enough.

What big data will do – as it always has – is provide useful statistical summaries for technical policy work. Actually, IBM itself grew out of the need for machines to process the US census data when it got too big for people. All that census data and machines that process it have been around for a long time. Essential for running a modern state – but it’s not producing new social theories.

My remark about IBM and large consultancies is not about them trying to sell software. It’s about them getting large government contracts – and possibly control of data. Modern states have a lot of data and want people who can offer them solutions to their problems. Those who make big promises are likely to get a favorable hearing. One should be aware of these issues because they can distort debate.

Overall, I’m all for data and machine learning. But let’s not forget to think. Thankfully, given the quality of the responses on this blog I feel a bit better about that.

Thanks again!

Less laconically: Genetic/evolutionary machine learning is deductive reasoning – if in a psychologically alien context. Yes you must have an initial aim to your project (otherwise its not a project), but you do not need a model. That will evolve. An expressed genome is a theory. Billions may get discounted. Just as billions of ideas form in our minds, but fail to ‘take hold’ when they fail to hold water/prove useful. Many, however, should eventually prove usefully predictive, in regard to your project’s aim. Rarely, but on occasion, one will remain the most potent for many generations, but eventually it will most likely get superseded. Very rarely some never will, and they become enshrined by dint of simply being ‘better’ than all other competing theories/expressed genomes. Just as they do in ‘science’. The problem with this is that humans want to be able to communicate their ‘theories’, and co-comprehend them. Currently, only the test of time can afford any credence to such black boxes, as today’s evolutionary algorithms. I predict a major goal of such machine learning techniques, will be to automatically describe the models that are generated by ML, in such a way as to afford critical analysis by humans, or possibly other independently created systems.

The ethical quandary left at which point is then: “Aren’t we holding back progress, by imposing human requirements on these systems?” Do we needlessly anthropomorphize machines learning systems, and thereby hinder them? Or is scientific method or some analogous, machines based, critical analysis a necessary part of insight generation, and our trust thereof?

I guess this is also saying that we need both theory and data. However, one important point: The position taken in Anderson’s post might be seem to be a straw-man; in fact Anderson later himself admitted that he was being provocative. But it would amuse you to know that this seemingly ridiculous, straw-man of a position is actually followed in practice! Its been going on in many fields. For example consider what the Nobel Prize winning biologist Sydney Brenner has to say about such an approach in his field: “The orgy of fact extraction in which everybody is currently engaged has, like most consumer economies, accumulated a vast debt. This is a debt of theory, and some of us are soon going to have an exciting time paying it back – with interest, I hope”

And I feel in essence, this is what the post tried to bring out.