October 30, 2009
Will the New White House Initiative for Rigorous Evaluation Elicit a Response from the U.S. Foreign Assistance Agencies?
By Mead Over
This October 7 memo from Peter Orszag is interesting not only for its emphasis on evaluation, but also for its use of a carrot approach instead of (in addition to?) a stick approach to getting the participation of the various agencies and bureaus of the US government. (Thanks to Mattias Lundberg for flagging this memo for me.)
Beginning during the second world war, with a periodic surge of interest every few decades, the US government has attempted to improve governmental decision-making through the use of evidence-based evaluation. The current initiative announced by Orszag in this memo constitutes a resuscitation of the this laudable public sector objective. Strong features of the memo are its promotion of studies that evaluate multiple alternative interventions approaches against one another and its exhortation for improved “rigor”.
Indeed, the memo uses the word “rigor” or “rigorous” a total of nineteen times. For me, an essential ingredient of a “rigorous” evaluation design is an explicit strategy for identifying a counterfactual to the program being evaluated – what would have occurred without the program. In order of increasing rigor, such a strategy might use a before-after comparison, a comparison of a group that did not receive the intervention to those that did, or a randomized assignment of people to receive or not receive the program. Yet the memo never uses any of the words “counterfactual,” “matched,” “random” etc., instead leaving it up to each bureau to define the characteristics of a “rigorous” evaluation.
Furthermore, the memo omits any reference to defining a set of priority objectives that can be measured by agencies and programs of similar mission (e.g., the more than 20 U.S. government entities involved in development and foreign assistance) in order to aggregate and compare impact. And it neglects to urge that programs compare alternatives in terms of their costs as well as their effects. The memo only mentions the word “cost” as an attribute of the evaluation studies to be proposed, not as an object of evaluation in and of itself. Again it will be up to the responding bureaus to decide whether the cost of a program should enter into judgments of its relative “worth” compared to alternative programs.
I wonder how the foreign assistance agencies mentioned above will respond. The Millennium Challenge Corporation has a great web-site full of interesting evidence available to the public. In contrast, PEPFAR is rumored to have completed studies of the cost of delivering antiretroviral therapy in six countries, but the studies have not been released to the public. This is particularly strange in light of the Congressional mandate that they report the costs of their programs to Congress by September 30, 2009, as David Wendt and I blogged here. Did OGAC make that deadline? If so, why has the information been kept under wraps?
And how about the parts of DoD that deliver foreign assistance? Will they also be producing evaluations? If so, what criteria will they use? Hearts and minds won? Or perhaps something more concrete like wells dug per thousand dollars? And back to that aggregation and comparability point – if we can’t report out an aggregate impact of our collective efforts, how do we continue to sustain Americans’ support for foreign assistance; and if we can’t compare across like-missioned agencies, how can we appropriately reduce the fragmentation?.
November 6, 2009 at 12:31 am
Dear Mead,
I think you are right to point out the lack of precision and clear guidance given in the OMB memo. Nevertheless, it is somewhat promising that the White House is starting to think about evaluation and accountability to the tax payers. The US government could learn much from the Latin-American experience of institutionalizing evaluation, in particular from the leading evaluation entity in Mexico, CONEVAL. The fact that the social sector agencies are required by law to have an annual evaluation program agreed-upon with CONEVAL, the Ministry of Finance, and the public comptroller’s office as a prerequisite for inclusion in the national budget gives the institution a powerful mandate. Please refer to 3ie’s recent report: ‘Institutionalizing Evaluation: a Review of International Experience’ (http://tinyurl.com/ygvrbm8)
and a policy brief ‘Impact Evaluation 01: How to institutionalize evaluation?’(http://tinyurl.com/yljf7ps).
Best regards,
Marie Gaarder
Deputy Director, 3ie
November 6, 2009 at 8:48 pm
Evaluation has to relate to a goal, and in development unfortunately the USG doesn’t have a theory that isolates the factors that can serve as goals for evaluation of its efforts.
The New Growth Theory provides its answer, in terms of human capacity to absorb, adapt, generate, and use technology to create change, but it seems to be too technical and not political enough to get development agencies’ attention.
As a result they still don’t know what to evaluate.
OMB is not well adapted to leading this effort.