Answering the question: “Was this a successful project?”
February 12th, 2009 | Published in Google.org
A guest post from Howard White, Executive Director, International Initiative for Impact Evaluation:
In the foreign aid business, it's very important to answer the question, “Was this a successful project?” because donors need to know where to target their scarce resources. Unfortunately, the answer to this question has frequently been, “Well it must be, we spent loads of money and employed lots of consultants." This problem has been most recently highlighted by the Centre for Global Development’s report, "When Will We Ever Learn?". Billions of dollars are spent on development interventions each year with little evidence on whether they work or not.
I started my own academic career looking at aid impact at the macroeconomic level. In my own defense, I can say that, starting with my PhD, I criticized the over-aggregated cross-country regression approach, arguing instead for careful country-level analysis of macro impact that could pick up on both context but also the many channels – both money and ideas – through which aid affects development outcomes. Having published a couple of widely ignored books pursuing this approach, I became increasingly convinced that aid impact had to be examined from the bottom up. That is a systematic analysis of, if not all interventions, then enough to be able to make authoritative statements about whether the aid program is working or not. This might seem like a lot of work; but then so is measuring GDP and we do that.
But there are two problems to implementing a bottom up approach. The first is that evaluation design is often poor when it comes to measuring impact. The second problem is that most agencies do not systematically compile evidence across interventions. It is symptomatic of a culture of producing evaluations as a bureaucratic requirement, not as a management or learning tool.
It is to tackle these two problems - the lack of evidence and the failure to synthesize what evidence there is into policy-relevant lessons – that the International Initiative for Impact Evaluation (3ie) has been created. Google.org has been one of our early funders.
The bulk of 3ie funding is going to help expand our knowledge about what works by supporting new impact studies, focusing on what we call the enduring questions of development. We are in the process of a consultative exercise to identify these questions – to which you can contribute by visiting our website. We will also have an open window for studies on other topics, for which the first request for proposals went out recently. Please visit our web site and join the conversation!
In the foreign aid business, it's very important to answer the question, “Was this a successful project?” because donors need to know where to target their scarce resources. Unfortunately, the answer to this question has frequently been, “Well it must be, we spent loads of money and employed lots of consultants." This problem has been most recently highlighted by the Centre for Global Development’s report, "When Will We Ever Learn?". Billions of dollars are spent on development interventions each year with little evidence on whether they work or not.
I started my own academic career looking at aid impact at the macroeconomic level. In my own defense, I can say that, starting with my PhD, I criticized the over-aggregated cross-country regression approach, arguing instead for careful country-level analysis of macro impact that could pick up on both context but also the many channels – both money and ideas – through which aid affects development outcomes. Having published a couple of widely ignored books pursuing this approach, I became increasingly convinced that aid impact had to be examined from the bottom up. That is a systematic analysis of, if not all interventions, then enough to be able to make authoritative statements about whether the aid program is working or not. This might seem like a lot of work; but then so is measuring GDP and we do that.
But there are two problems to implementing a bottom up approach. The first is that evaluation design is often poor when it comes to measuring impact. The second problem is that most agencies do not systematically compile evidence across interventions. It is symptomatic of a culture of producing evaluations as a bureaucratic requirement, not as a management or learning tool.
It is to tackle these two problems - the lack of evidence and the failure to synthesize what evidence there is into policy-relevant lessons – that the International Initiative for Impact Evaluation (3ie) has been created. Google.org has been one of our early funders.
The bulk of 3ie funding is going to help expand our knowledge about what works by supporting new impact studies, focusing on what we call the enduring questions of development. We are in the process of a consultative exercise to identify these questions – to which you can contribute by visiting our website. We will also have an open window for studies on other topics, for which the first request for proposals went out recently. Please visit our web site and join the conversation!