Evaluation, Evaluation Everywhere: IOM Progress on Evaluating the Impact of PEPFAR

January 11, 2010

Within the beltway and across newspaper headlines, PEPFAR is widely perceived as a very successful US foreign assistance program. Without fail, mentions of PEPFAR’s success all reference the same single measure: currently PEPFAR supports 2.4 million people on ARV treatment.

Interestingly, these broad perceptions of success exist despite the fact that, or maybe because, there has never been a systematic evaluation of PEPFAR’s impact. However, that is in the process of changing.

In the 2008 re-authorization of PEPFAR, the US Institute of Medicine (IOM) was mandated to evaluate the impact of PEPFAR. This is a monumental task. PEPFAR will have spent around US$60 billion over 10 years, through multiple US government agencies and hundreds of partnering organizations, and across dozens of countries and equally many program areas. The IOM has been tasked with figuring out how this immense initiative has changed the world.

The IOM approach has two phases. In November 2009 it started a planning committee, made up of 20 members with a wide range of expertise, which is currently designing an evaluation plan. By fall 2010, after five meetings, information gathering sessions, and discussion, they hope to have an evaluation plan. After that an evaluation committee will be formed, with some overlapping membership from the planning committee, and asked to produced an impact evaluation report by fall 2012.

So far the planning committee has met three times (out of the five they have scheduled): a November meeting in Washington, a December meeting in Geneva, and most recently a meeting last week again in Washington. Last Thursday I attended the “public information gathering session”, which was the second day of the three day meeting and open to the public. At this meeting the planning committee received presentations from staff responsible for monitoring and evaluation (M&E) at various levels of the PEPFAR, including: staff in the strategic information department at OGAC headquarters, M&E managers within CDC and US AID, and program managers and funding recipients based in developing countries. Surprisingly, six hours of these presentations continued to capture my interest. It struck me that this process, perhaps even more than the final report, offers great opportunities for learning and building understanding of what PEPFAR is and how it has performed. It also struck me that the room was half empty, despite being open to the public.

There have been literally hundreds or perhaps even thousands of evaluation studies of PEPFAR programs. While not all of these studies investigated impact, they do indicate an environment potentially rich with program data and analysis. At the committee meeting I was disappointed to learn that OGAC does not receive or even track all of these evaluations, receiving only very general aggregated data for their own analysis upon which to base decisions.

A further concern is that there is very little sharing of data outside the programmatic silos. In particular, national stakeholders are often left out of the loop, neither benefiting from the M&E learning nor being armed with the evidence for effective decision-making about programs. Country ownership and accountability to national stakeholders will always be limited if they are not involved in the design, implementation and learning that comes from monitoring and evaluation.

Given the critical importance and opportunity of this activity, I hope that we start to see far more engagement around this process. Perhaps more could be accomplished if the committees share their intermediate products as they take stock of the PEPFAR program in its entirety and try to figure out how to evaluate its impact. For instance, it would be invaluable if the committee would conduct and publish a systematic synthetic review of the evaluation studies that have already been done of PEPFAR programs and projects. Such a review would benefit both the committee as they plan their evaluation and everyone concerned with the effectiveness of PEPFAR programs.

It could help:

• Provide a clear picture of which PEPFAR programs have and have not been evaluated and how programs have been evaluated;
• Map what has already been learned about impact, even if existing results are limited in scale; and
• Identify gaps and opportunities for evaluating PEPFAR on a broader scale.

You can follow any responses to this entry through the RSS 2.0 feed. You can leave a response, or trackback from your own site.

1. Ruth Levine Says:
   January 13th, 2010 at 1:09 am

   Very useful blog post, David. I had the opportunity to speak to the committee on Thursday, conveying the key messages from the IOM workshop that I facilitated in April 30-May 1, 2009, on the design of a PEPFAR impact evaluation [see the report here: http://www.iom.edu/Reports/2008/Design-Considerations-for-Evaluating-the-Impact-of-PEPFAR-Workshop-Summary.aspx, and I also was surprised to see a half-empty room. It is almost always the case that a public event in Washington, D.C., related to HIV/AIDS, and particularly PEPFAR, draws a crowd. But in this case there were only a handful of die-hard spectators, several from the Office of the Global AIDS Coordinator and others from different parts of the US Government involved in global health issues. Where were the HIV/AIDS advocates and PEPFAR implementers who were filling up all the meeting rooms around the time of the PEPFAR reauthorization, and who are watching with great interest how the Administration approaches the budget requests to Congress?

   It’s too bad that more weren’t at the IOM on Thursday, because I think the eventual credibility and value of the evaluation rests, in part, on key audiences in the policy and advocacy community understanding the difficult challenges that the Committee faces in defining the full scope of the evaluation and the methods to be used. For example, during the panel session in which I participated, Martin Vaessen of DHS Macro described the conduct of the Five-Year Evaluation of the Global Fund to Fight AIDS, TB and Malaria, which reported out in April 2009. That effort, which cost something north of $16 million, encountered tremendous challenges in obtaining high quality data and estimating the impact on incidence and prevalence. In the end, despite an almost unimaginable amount of original data collection at the household, community and facility levels, the estimates of impact were subject to so many caveats that it’s hard to know what to make of them. The challenge facing the PEPFAR evaluation design Committee is how to arrive at more definitive conclusions, given similar problems of lack of baseline information and uncertainties about the data quality.

   Fortunately, the composition of the design Committee is excellent, with an impressive range of substantive and methodological expertise (see the list here: http://www8.nationalacademies.org/cp/CommitteeView.aspx?key=49140). I’m sure they’re equal to the task, and will take on board both the input from the earlier workshop and the lessons from both the first PEPFAR assessment and the Global Fund Five-Year Evaluation.

   And, David, I very much like your suggestion that a careful review of prior evaluations and studies be undertaken so that the Committee can accurately see what has already been learned and what the gaps are. This sort of review would be particularly helpful if it highlighted the types of methods used for the different evaluations, and the corresponding strength of the evidence generated.

2. Steve Rosenzweig Says:
   January 14th, 2010 at 8:08 am

   Great blog, David. While I’m disappointed that the evaluation is not getting as much attention from stakeholders as it deserves, I’m not surprised. It’s an incredibly complex task, and engagement in the process is not as sexy or visible as lobbying for increased funds.

   I think the idea for a synthetic review is a great one. When the Monitor started just a few years ago, there was very little evidence from evaluations of PEPFAR programs. Now that’s no longer the case, but I think it’s safe to say that even the most avid PEPFAR watchers aren’t fully versed in the evidence produced to date and could benefit from a systematic review of what’s been learned. And the benefit of this type of review for those designing the impact evaluation is obvious.

   As Ruth noted, those of us who closely followed the Global Fund 5-year evaluation know how difficult it will be to evaluate the impact of such a large and diffuse program with limited baseline data. Hopefully, lessons learned from that exercise will help the committee to produce an evaluation that’s able to, at least partially, tell us how PEPFAR has \changed the world,\ as you put it. I don’t envy them the task.

3. Wim Van Damme Says:
   January 18th, 2010 at 5:51 am

   Dear CGD bloggers,
   From the other side of the Atlantic, I agree with your analysis. This week’s Lancet articles on the evaluation of UNICEF’s Accelerated Child Survival and Development (ACSD) programme in west Africa, certainly illustrates several of the points you make. It really underlines the importance of “penetrating the black box of implementation”, as the online Comment by Stefan Peterson calls it.
   I think that the PEPFAR evaluation should indeed build on the Global Fund’s 5 year evaluation; but also that evaluation should involve some scenario building; asking the question: ‘where is this leading to in 10 – 20 years from now?’ and with “this”, I mean the AIDS pandemic, and the AIDS programmes. I am sure Mead Over also has some ideas on this; but there are other perspectives too.
   AIDS 2031 is trying to do that, but seems to be struggling, at least for certain aspects.
   Wim Van Damme, Institute of Tropical Medicine, Antwerp, Belgium

We value frank and constructive exchanges and encourage you to use your real name in your comments.