Update 01/02/2011: the first evaluation cited is specific to UN agencies, the second to donors. Have also clarified the specific references in the UN report. Thanks to Michael Keizer for pointing this out.
One of the recurring themes of this blog is the idea that aid agencies need to become more flexible and responsive, both to changing contexts and the needs of developing countries. Unfortunately the ongoing focus on ‘results’ – as it is currently being shaped and implemented – seems to be in direct conflict with this idea. Moreover, the repeated criticisms that have been made of the ‘results agenda’ doesn’t seem to affect its influence or the broad approach. How are we to move beyond this situation?
There is a engineering theory that underlies many ‘modern management’ approaches, which involves taking a “reductionist approach to problem solving.” The aim is to decompose a complicated problem into more manageable and well-defined sub-problems that can be separated and dealt with as discrete issues, like so:
The benefits of this reductionist approach are that it is methodical, conceptually intuitive, and it allows any given operating environment to be defined with precision and accuracy. It supports transparency, it benefits hierarchical decision-making, it aids trouble-shooting, and it facilitates planning and control.
It is also, in quite a few cases, complete nonsense.
Even in the best case scenario, such approaches can only be followed loosely because real-world systems cannot be divided up and controlled in neat and tidy ways. The mismatches between reality and the managerial assumptions manifest themselves in numerous ways. Two are especially relevant to current debates on improving aid:
- First, when failure occurs, where it is acknowledged, it becomes common and logical to highlight the precise points in the linear chain where the failure occurred. The implicit assumption is that through such precise narratives and their subsequent take-up, future failures can be avoided. Those with access to these narratives just need to make the necessary modifications during design or planning stages. Failures that indicate wider systemic issues may be highlighted from time to time. But they are usually found to be too challenging to address in the context of ongoing work, and are therefore dismissed as ‘just not practical’.
- Second, when faced with uncertainty, the value of standard operating procedures and plans tends to diminish considerably. But the need for pre-defined purposes and actions makes design principles such as ‘be adaptable to novel conditions’ seem ambiguous and vague. The common way around this it to develop contingencies and build these into strategies and plans. This involves the paradoxical attempt to pre-define the kinds of responses that will be followed when faced with unanticipated changes. Such contingencies are next to useless because they usually do not allow for re-orientation of resources around these changed realities. They are also increasingly meaningless in a world characterised by dynamic, turbulent change.
Such reductionist approaches are hard-wired into the international aid system. We see them manifested in the form of logical frameworks, results-based management approaches, and the more recent interest in value-for-money. The Paris Declaration for aid effectiveness and related documents repeatedly state the importance of ‘modern management’ techniques, essentially bringing New Public Management (with all its ails) to bear on issues of global poverty and crises.
There is a lot of anecdotal evidence about these approaches – ranging from vaguely negative to downright hysterical. Even some of the major donors who support them have admitted that: ‘we don’t pretend these things describe reality’. But what systematic evidence is there on how well are these approaches are doing? Do they show aid agencies navigating the challenges outlined above, finding ways to make these approaches relevant to a dynamic and interconnected world? Are these results-based approaches succeeding on their own terms?
Sadly, it appears not. An evaluation of results-based management approaches among UN agencies identifies a range of concerns with the approach at both conceptual and practical levels. It found that:
- the formalistic approach to codifying how to achieve outcomes, inherent to RBM, can stifle the innovation and flexibility required to achieve those outcomes
- results-based management as implemented across the UN takes no account of the fact that outcomes are influenced by multiple actors and external risk factors
- results-based management processes has been found to have made virtually no contribution to strategic decisions in any of the reviewed organisations
- the determination of development success does not lend itself to impartial, transparent and precise measurement
- Many of the results planned for have been expressed in a self-serving manner, lack credible methods for verification and involve reporting based on subjective judgement
- although aspirational results are used to justify approval of budgets, the actual attainment or non-attainment of results is of no discernible consequence to subsequent resource allocation or other decision-making.
It concluded damningly that “…RBM in the United Nations has been an administrative chore of little value to accountability and decision-making…”
It doesn’t stop there. Andrew Natsios, former head of USAID has argued convincingly that measurability of the kind propagated by existing RBM systems is inversely proportional to development relevance. And an earlier report by the OECD-DAC on RBM in donors found that:
there are dangers in designing performance measurement systems too much from the top-down. Unless there is a sense of ownership or “buy-in” by project/program management and partners, the performance data are unlikely to be used in operational decision-making. Moreover, imposed, top-down systems may lack relevance to actual project/program results, may not sufficiently capture their diversity, and may even lead to program distortions as managers try to do what is measurable rather than what is best. Field managers need some autonomy if they are going to manage-for-results. Some operational level flexibility is needed for defining, measuring, reporting, and using results data that are appropriate to the specific project/program and to its country setting.
The abject failures outlined here do not appear to have been a major cause for concern for those promoting and supporting such approaches. As the UN review cited above glumly concluded, despite these conceptual and practical shortcomings, the RBM agenda is ‘here to stay’. Ironically, results-based management and accompanying top-down control processes do not themselves appear to need results to be championed and implemented with ever-greater enthusiasm.
This supports the findings of social complexity thinkers, who have found that that when faced with ‘wicked problem’-style challenges to the engineering mindset, the predominate tendency is to try to apply the existing mindset more firmly: “to stay safe within the existing cognitive framework and try to find a solution within it.”
The danger is that being wedded to a particular approach in the face of repeated failure risks the overall legitimacy and relevance of what is being done. This problem extends beyond management to scientific thought.
In his infamous work on new paradigms in natural sciences, Thomas Kuhn noted dryly: “the research scientist is not an innovator but a solver of puzzles, and the puzzles upon which he concentrates are just those which he believes can be both stated and solved within the existing scientific tradition.”
How to move beyond this problem? Kuhn argues that it is the slow accumulation of anomalies that leads to breakthroughs in scientific thinking. When weaknesses in the old paradigm are revealed and resolved (or not), this highlights the problems with the existing way of doing things. This slowly builds up to the point that the old paradigm reaches crisis point and is supplanted.
So the key is not to throw the results baby out with the reductionist bathwater. Instead, if the results agenda is here to stay, maybe the key is to reform the results agenda to make it more relevant to the complex, ambiguous world we live and work in.
Nancy Birdsall put her thoughts forward succinctly in a blog last year:
For a country to get results might not require more money but a reconfiguration of local politics, the cleaning up of bureaucratic red tape, local leadership in setting priorities or simply more exposure to the force of local public opinion.
Let aid be more closely tied to well-defined results that recipient countries are aiming for; let donors and recipients start reporting those results to their own citizens;
let there be continuous evaluation and learning about the mechanics of how recipient countries and societies get those results…
[let's focus on] their institutional shifts, their system reforms, their shifting politics and priorities…
And to add one more point in closing: let’s also make sure we demand the same results of the ‘results agenda’ as it demands of every other agenda.