Print
Share
PEPFAR Might Be Saving Millions of Lives – But We Don’t Have Evidence Yet
April 28, 2009
It’s often said that the perfect is the enemy of the good. But in the area of program evaluation, the enthusiasm for a humanitarian program can lead to wide dissemination of optimistic results even if they are based on a flawed evaluation technique. In this case, the bad is the enemy of the good – because poor quality evaluation can deflect interest from good evaluation. Members of Congress will want to consider this cautionary note when it comes time for confirmation hearings for Eric Goosby as Global AIDS Coordinator and head of the President’s Emergency Program for AIDS Relief, or PEPFAR, and ask about his plans for supporting better evaluation of this important program.
On April 9 the Washington Post published an editorial on the success of PEPFAR not only in placing people on treatment but in reducing mortality in the PEPFAR countries. In support of the latter assertion they cite an article that is due to appear in the May 19 issue of Annals of Internal Medicine and is already available online. In the words of the Washington Post’s editorial staff, “[T]he authors of the study, Eran Bendavid of Stanford University and Jayanta Bhattacharya of the National Bureau of Economic Research, concluded that “about 1.2 million deaths were averted because of PEPFAR’s activities.” Unfortunately, this conclusion is far from warranted by either the original authors or by the Post.
Bendavid and Bhattacharya arrived at their conclusion by comparing the mortality trends since the start of antiretroviral therapy in the 12 African PEPFAR focus countries with 29 other African countries. But how did they get data on mortality in those 41 countries? Perhaps readers thought that the authors had compared vital registration data in the 41 countries, but unfortunately none of these countries systematically collects data on the births and deaths of its residents. Readers might have guessed that the authors had used census data to measure mortality, but unfortunately only a few have conducted censuses in the last five years. Or readers might have reasonably conjectured that the authors had compared mortality estimates from systematic sample surveys such as the Demographic and Health Surveys that exist for many countries, but the authors did not attempt this exercise. A careful reading of their “methods” section reveals that they simply used the latest version of the UNAIDS/WHO data on AIDS mortality, available here.
The authors don’t raise the question of where UNAIDS might have gotten its data on mortality. The answer is that UNAIDS projected its mortality estimates. This is a perfectly respectable activity. I do it myself for the future (here and here), and UNAIDS’ job was to use known treatment enrollment data to project mortality into the past. The trick is to assume one can generalize from the various small studies on treatment success (summarized in Table 1 of Stover et al. here) and then to infer the trend in national AIDS mortality from that assumption and from national estimates of the number of patients enrolled.
But it is tautological for Bendavid and Bhattacharya to use the projected mortality data as proof that treatment is working when the mortality data itself has been generated by ASSUMING that treatment is working.
As I mentioned in my Q&A of November 2006, it was fairly surprising that the 2006 issue of the UNAIDS annual data compendium showed an increase in the number of AIDS deaths in Sub-Saharan Africa despite the fact that one million persons had been placed on anti-retroviral therapy. The most likely explanation is that in the rush to get out the first annual report since treatment had become so widespread, the UNAIDS team responsible for mortality projections had not yet entered into their model the number of people on treatment. Neither I nor anyone else would have concluded that AIDS treatment rollout had caused an increase in AIDS mortality.
Since 2006, UNAIDS has worked hard to make the country data on AIDS mortality consistent with national estimates of treatment enrollment and the best guesses available on treatment’s benefits. The 2007 update, which I blogged, revised all the data to incorporate treatment uptake and the 2008 update, available here, provides retrospective estimates of mortality and prevalence which the UNAIDS team has “backcast” to be consistent with treatment numbers over the last few years as well as with current lower prevalence estimates.
The UNAIDS data team is to be congratulated on producing a comprehensive revision of their data on HIV prevalence, treatment enrollment and mortality. They have done their best to make these data internally consistent using the latest estimates of survival with and without antiretroviral therapy. But no one should mistake their mortality projections as evidence that treatment is working. We need to have REAL data on the success of treatment in order to judge how well it’s working and to make the case for its continued support.
My colleague Nandini Oomman has recently published a memo to President Obama urging the release of PEPFAR data on contracting. I join her in that request – and further ask that PEPFAR’s data on effectiveness and unit costs also be released. This data has been collected with billions of US taxpayer funding. What is the justification for keeping it from the public?
8 Responses to “PEPFAR Might Be Saving Millions of Lives – But We Don’t Have Evidence Yet”
Post a Comment
We value frank and constructive exchanges and encourage you to use your real name in your comments.
April 29th, 2009 at 4:12 pm
You hit on one of the biggest problems of determining program success… how can you calculate the effects of a program (PEPFAR) that has prevented another act (death) from happening. Having worked with some of the folks that do/did the UNAIDS estimates I know how much goes into the estimates. But the question is, what else do we have to go on? Is there a better way? DHS surveys have their flaws, just like census data. It is a tough challenge for demographers, policy analysts, and policy makers.
May 4th, 2009 at 5:47 pm
It is true that estimates of AIDS deaths are modeled not measured directly. The estimates are based on model calculations of the number neededing ART, the number receiving ART and assumptions about survival with and without ART. Thus the finding that PEPFAR contributed to a reduction in AIDS deaths is based on the assumption that ART reduces mortality. The UNAIDS approach uses recent analysis of cohort data to develop these assumptions so they do rest on the best available information. Most people would be willing to accept that ART reduces mortality, although the exact amount is subject to great uncertainty. The model do produce uncertainty estimates that could be used to test whether there is a significant difference in mortality between PEPFAR and non-PEPFAR countries. Perhaps it would also be useful to show whether the PEPFAR-supported countries had higher ART coverage than non-PEPFAR focus countries. If so, then it would strengthen the conclusion that PEPFAR contributed to a reduction in mortality.
The trends in prevalence and number of infected people are based on fitting a simple epidemiological model to surveillance and survey data. But it is not clear how prevalence relates to program success. High coverage of effective ART will increase prevalence by keeping people alive. High coverage of effective prevention would reduce prevalence by prevening new infections. Lower prevalence could mean a prevention success or a treatment failure. Higher prevaelnce could mean a prevention failure or a treatment success. The UNAIDS approach allows the estimation of incidence, which is the indicator we would really like to compare to evaluate prevention success, but it depends on many assumptions including ART effectiveness. As yet we have no good direct measure of HIV incidence.
May 6th, 2009 at 5:49 am
Thanks for your comment Ben. I agree that the evaluation of a scaled up treatment program on mortality is challenging, but I don’t think it’s as hard as you suggest. Implemeting vital registration in a few communities would be sufficient to measure mortality impacts at the population level. This has been done successfully on a small scale e.g. in Rakai, Uganda and could be replicated on a larger scale by PEPFAR. Also, I disagree that there is anything inherently difficult about measuring averted health problems. After all, this is what all successful drug trials have accomplished.
May 8th, 2009 at 4:15 am
Most interesting blog, thank you. Should we not also care about cost-effectiveness? I do not follow this work closely, and am unfamiliar with the numbers — but with PEPFAR (and the Global Fund for that matter) when you spend so many billions, there’s bound to be some good. In addition to establishing the evidence for what difference PEPFAR has made, we need to assess whether the difference is good enough, and whether there’s evidence that suggests that other approaches may have made a bigger difference.
May 8th, 2009 at 9:01 am
And thanks for your comment too, John. As a major contributor to the UNAIDS data production process, your contribution to this discussion will be particularly helpful to me and the readers of this blog. Thanks also for directing our our critical attention to the Bendavid and Bhattacharya findings on prevalence. Apparently the Washington Post has not been reading this blog or your comment on it, since their editorial on Thursday again cites this flawed analysis not only in praise of PEPFAR’s mortality benefits, but also to suggest that PEPFAR has not been doing enough to reduce prevalence. The WP might be correct that PEPFAR has done too little on prevention (as I have elsewhere argued), but the Bendavid and Bhattacharya paper has nothing to contribute on this question. http://www.washingtonpost.com/.....s_opinions
May 9th, 2009 at 2:13 am
We thank you and others who wrote in response for highlighting aspects of our work which we did not in our manuscript.
We disagree with Prof. Over’s critique on several grounds. While the UNAIDS data is not perfect, it is the best available data on mortality. That the estimates are projected does not invalidate them and does not mean that they are not based on direct observations. On the contrary, the UNAIDS team has gone to great lengths to collect the best available epidemiologic data which guides their projections. “Implemeting vital registration in a few communities” would hardly solve the problem of inference, as those also require projections of one sort or another to infer death rates. Our estimates are, if anything, an underestimate of PEPFAR’s true impact. The true counterfactual is the course of the epidemic without PEPFAR. While the comparison countries received relatively little direct assistance, they probably enjoyed some of PEPFAR’s spillover benefits such as improved regional supply chains and lowered unit costs for antiretrovirals and monitoring equipment. Thus, we used a counterfactual of the course of the epidemic in the presence of spillover benefits, which underestimates the true benefit. Finally, John Stover raises the excellent point that the uncertainty estimates could help in examining the effect of PEPFAR. We have done this analysis in the manuscript’s (under Sensitivity Analysis).
We also disagree that this will “deflect interest” from additional evaluations of PEPFAR. Our evaluation has already stirred interest in further evaluations, such as Rakesh Rajani’s comment suggesting a cost-effectiveness evaluation.
John Stover’s second main point is also insightful, and one I wish we had made more directly in the manuscript. To rephrase, parallel prevalence trends in the presence of relatively lower mortality imply relatively lower incidence, given a relatively stable population size. In the context of our analysis, the reduction in deaths in the presence of similar prevalence suggests a reduction in new infections. Direct estimates of incidence would be extremely valuable for future evaluations of PEPFAR and other prevention programs.
We hope that Prof. Over will agree with us that having only subjective statements on PEPFAR’s successes or failures would have been a poor guide for policy makers at this time when the global health budget is up for debate.
May 28th, 2009 at 12:56 pm
Thanks to Eran Bendavid, one of the co-authors of the AIM paper I critiqued, for responding to my comments on this blog. He is correct to point out that implementing vital registration in a few communities would not be sufficient to enable him to produce reliable national mortality estimates for 41 countries that could then be used to test whether PEPFAR is reducing mortality. Nor did I suggest that it would. This fact underlines the difficulty of actually measuring mortality rates in poor countries. There is an entire literature in demography on this issue. I agree with his statement that \the UNAIDS team has gone to great lengths to collect the best available epidemiologic data which guides their projections\ – provided we limit that characterization to the HIV prevalence and AIDS treatment data which form the foundation of their backcasted estimates of mortality. While the UNAIDS team may have checked their model’s mortality projections against other independent estimates of mortality in a few countries, they do not document this process or even refer to it in any publication or web location I have seen. (They do document their methods for inferring HIV prevalence from the available data.) John Stover of the Futures Institute provides close support to the UNAIDS statistical team. I think we should accept his statement in his comment above that the authors’ “finding that PEPFAR contributed to a reduction in AIDS deaths is based on the assumption that ART reduces mortality”. None of this means that PEPFAR did not reduce mortality. But it will be at least another decade before the demographers are able to estimate 21st century mortality using population censuses across all the 41 countries used in this regression analysis. Once demographers have done that arduous work, Bendavid and others can use the resulting independently estimated mortality numbers to estimate the impact of PEPFAR without fear that they are simply recovering the underlying assumptions.
May 29th, 2009 at 4:32 pm
Certainly there is a wealth of studies showing that ART reduces mortality. We do not have rigorous evaluations of the impact of ART at the national level because of the difficultly of monitoring adult mortality. The data on the number of people started on ART with PEPFAR funding are reliable. So it is reasonable to assume that PEPFAR has had an important impact on mortality. This paper provides an estimate of the magnitude of that impact and is useful for that purpose. It does not prove that PEPFAR-supported ART programs have reduced AIDS mortality since these are modeled estimates, but I feel that the justification is strong for claiming an impact of this approximately this magnitude.