Print
Share
Don’t Do Impact Evaluations Because…
May 31, 2011
Recently, I was called for advice by someone who will be running a workshop attended by people who implement and evaluate programs. She asked me to help her anticipate the main objections raised against doing impact evaluations—evaluations that measure how much of an outcome can be attributed to a specific intervention–and to suggest possible responses. The FAQs on our Evaluation Gap site offer some guidance, but in answering her question, I realized that five particular objections come up over and over again. They are:
Objection #1: “We already spend a lot on evaluation”
This may be true. Most organizations do spend money on evaluations and many of these evaluations are very useful for improving operations and monitoring outputs. The problem is that most of those studies don’t measure impact. So an organization may know that it implements its programs well, but it doesn’t know if its programs have the desired outcomes. A second common problem is that even when impact evaluations are undertaken, their rigor and quality may be poor. When we reviewed impact evaluations back in 2004, it was depressingly common to find a paragraph in the methodological section explaining that conclusions were compromised by the lack of baseline data. So, yes, your organization spends on evaluation, but it may not be commissioning impact evaluations and it may not be getting good ones.
Objection #2: “Impact evaluation methods can’t be applied in our field”
Debates over the best methodology are interminable when they are argued in the abstract. The only time you can decide if an impact evaluation can or cannot be conducted is when you have identified a particular question in a specific situation. (Thanks to Stef Bertozzi for pointing this out to me years ago and to Michael Clemens who illustrates this point with respect to a specific prominent initiative, the Millennium Villages Project).
Researchers have demonstrated considerable creativity in developing ways to study programs that address such diverse issues as corruption, teacher absenteeism, women’s empowerment and ethnic fragmentation. The first step is to identify the question. The second step is to figure out which method will give you the most convincing answer. For impact evaluations, that usually involves making a serious effort to compare outcomes from your program with some alternative – a counterfactual that you can either directly observe or plausibly construct.
Objection #3: “Impact evaluations cost too much”
Relative to what? The cost of an impact evaluation should be judged relative to the value of the information it will produce. So, a $2 million evaluation of a $500,000 program might be extremely cost-effective – a bargain – if the study helps policymakers decide whether or not to scale up into a billion dollar national program. This also means that impact evaluations should not be required of every program – rather they should be commissioned strategically to assess those programs that are unproven and are either widely used or are new and promising. Being selective in this way makes the overall budget for impact evaluation manageable relative to the overall budget for operations.
Objection #4: “We know that our programs work so it would be a waste of money”
If you really know that your program works, then it would be wasteful to conduct impact evaluations. We don’t need a study to know that feeding starving people will keep them alive. But most social programs aren’t this obvious. For years, job training programs in the U.S. were thought to be highly successful with good placement rates–until impact evaluations showed that placement rates were higher for comparable people who didn’t participate. Initiatives like conditional cash transfer programs that are widely supported today were initially the subject of considerable worry, that they would lead, for example to dependence, increased alcohol consumption or violence against women. Only because of good evaluations has it been possible to allay these fears and document the broad benefits. As with medical treatments, we need to have more humility about social programs and responsibly assess whether they are truly beneficial or cause harm relative to alternatives.
Objection #5: “Impact evaluations don’t affect policy decisions”
Evidence from impact evaluations is only one of many factors that influence policy decisions. If you think that the passage from evidence to action is linear and direct, you will almost always be disappointed. But the influence of impact evaluations is more complex than that. The questions they answer shape public debates over appropriate and effective policies. They also provide a base of information that is available when critical moments or opportunities arise to influence key choices. Rigorous impact evaluations have a much longer shelf-life than you’d think, while studies that are less rigorous or less conclusive quickly fade. Certainly more can be done to encourage the use of evidence in policymaking but this is a case where more and better supply can, I believe, have an indirect and ultimately decisive influence over the course of time.
My last suggestion for the workshop? Foster a collaborative atmosphere by posing problems as a shared challenge to which you are seeking solutions and be honest about your own weaknesses and failures. People usually respond to such admissions by feeling safer about sharing their own experiences, doubts and mistakes. After all, we’re human beings and that’s how we learn.
Possibly Related Posts
7 Responses to “Don’t Do Impact Evaluations Because…”
Post a Comment
We value frank and constructive exchanges and encourage you to use your real name in your comments.
June 1st, 2011 at 5:25 am
There’s however a problem with impact evaluation that you don’t cover : they are a poor replacement for the best impact evaluation tool that has ever been found, the market.
The example of job evaluation program shows that : I doubt the people were *really* comparable. Because if it was really easier for them to get a job directly, they wouldn’t have applied for job training. There was a problem of negative selection, people who took the job training had for some reason a problem at getting a job directly and evaluating precisely what that problem was is really difficult. But that’s the joy of a free market : if it was easier to get a job directly they would have done that, so they took the job training because they saw some advantage in it. So I think the market says the job training did bring them something, we just don’t know exactly what, and that the evaluation impact failed at measuring exactly what.
June 1st, 2011 at 9:17 am
Great post. You might like the recent ODI paper ‘Learning how to learn’, which I blogged about here:
http://blogs.odi.org.uk/blogs/.....dence.aspx
Ben
June 2nd, 2011 at 7:36 am
Useful – glad that questions, and not methods, are up front as they should be in any evaluation process.
The answer to objection 5 and objection 2 seem incompatible. Objection 2 assumes people will use the information to scale up, hence justifying costly evaluations. Yet objection 5 says there is not necessarily a linear relationship between evaluation findings and decisions.
I miss definitions of ‘impact evaluation’ and ‘rigorous’ in the answers. These are contested terms, there is no global consensus on these.
June 6th, 2011 at 4:13 pm
[...] findings and are often ignored (a point stressed by Carlsson, by the way, in her speech). Savedoff comments on five recurring objections to expensive impact evaluations, including the non-linearity [...]
June 9th, 2011 at 2:34 pm
Thanks everyone for the comments.
To jmdesp: Markets give good feedback on things that consumers buy regularly and whose qualities are either readily apparent or can be learned at little cost. When I’m talking about impact evaluation, I’m addressing things that don’t have a market or whose qualities are not readily observed.
The job training programs I refer to are for people who are not served well by existing private for-profit training programs. The studies did not have negative selection because they started with a pool of people who were eligible for the program and then randomly selected the ones that would or would not receive the training. You can find a good overview of the literature by Friedlander, D., D. H. Greenberg, and P. K. Robins. 1997. “Evaluating Government
Training Programs for the Economically Disadvantaged.” Journal of Economic Literature 35 (4): 1809–55.
To Ben: thanks for the reference to “Learning How To Learn”
To img: In Objection 5, I’m arguing that the use of evidence is not automatic. I’m not arguing that it is always ignored. I believe it does influence policy, just not directly.
As for definitions, you can find some answers on our FAQs at http://bit.ly/iptjQZ and in “When Will We Ever Learn?” at http://bit.ly/b4DYyy.
Also, 3ie has a glossary of terms related to Impact Evaluation at: http://bit.ly/m0jNvl
June 15th, 2011 at 9:55 am
There could be 6th….
Program is having some impacts that become visible after many years even if the program gets over………how impact evaluation captures those long term effects on the population when demography gets changed over the time??? So, how effective is the result of impact evaluation in policy planning —- a continous dilemma amonst the policy makers???
Paul P.
June 16th, 2011 at 10:34 am
thanks, good reflection… it makes me wonder also about the fears around evaluation, things like “is there a hidden agenda”, ” finger pointing”, etc. sometimes these are the real issues but people rather use one of the arguments you have mentioned.