June 19, 2009
The “Who, What, Where, When, How and How Much” of International Health Aid – But Not the “Why”
By Mead Over
A study by Nirmala Ravishankar and colleagues at Chris Murray’s Institute for Health Metrics and Evaluation (IHME) published today in the Lancet and reported at Forbes and MSNBC gives the most comprehensive estimates yet available of the total amount that people in higher income countries have been spending to try to improve the health of those living in low and middle income countries. For you data junkies, that number was $19 billion in 2006 and $22 billion in 2007.
The many reasons why improved information on resource flows would contribute to better health assistance policy are laid out in the 2007 CGD study entitled Following the Money: Toward Better Tracking of Global Health Resources. The new IHME study is valuable for providing the first comprehensive point of reference for the amount actually disbursed on health assistance worldwide. The figures are higher than those previously available from the OECD, which recently estimated that health assistance by bilaterals and multilaterals totaled only $12.6 billion in 2006. The total is also substantially smaller than the $45 billion that was recently suggested by the Council on Foreign Relations for that same year.
Do we have reason to believe that the new estimates are more accurate than previous efforts? The starting point for IHME work was the OECD-DAC database. But unlike the OECD’s own statisticians, who confined themselves to analysis of aid commitments, the IHME team attempted to estimate total disbursements for health by all of the donors. While we would prefer to know disbursements than commitments, the task is challenging, because donors under-report aid disbursements to the DAC database. To compensate for this underreporting, and sometimes to fill in other gaps in the data, the IHME team were forced to fall back on data imputation, an approach which requires heroic assumptions. (We at the CGD have poked some fun at IHME’s penchant for data imputation before here and here, but we also practice a bit of this arcane art ourselves, e.g. here and here.) Although neither the published paper not the web appendix provides full details on how the authors imputed the missing numbers, on balance I prefer their imputed disbursement numbers to raw commitment numbers. I just wish the authors had told us what percentage of their totals depends upon imputed numbers.
The authors go beyond their imputed inflation of the DAC database to add estimates of flows from philanthropic organizations and from US-based NGOs. This is a valuable contribution to our understanding of total health assistance flows, since by 2007 they account for more than 20% of all estimated disbursements. One wishes they had been able to include major international NGOs based outside the US, like Medecins sans Frontier and Oxfam, but hopefully they will issue updates of this data every year and will succeed in soliciting input from the missing organizations for those updates.
The authors provide a large number of colorful and informative bar charts to track assistance funding over time, from the various sources to the various destinations. However, causal analysis based on these numbers is fraught with peril. Unfortunately, the authors themselves engage in some problematic causal analysis when they point with some unstated emotion (surprise? alarm? ) to the positive but imperfect correlation between the total disease burden in a country and the total amount of health assistance it receives. If we believe that health assistance should reduce disease burden then we might expect this correlation to be negative. If we believe that health assistance should be directed to places with higher burdens, then the correlation should be positive. If we think the main criterion for allocating foreign assistance should be the cost-effectiveness of spending opportunities, not the size of the burden, and that many other factors influence health beside health assistance from abroad, then we should expect that the correlation will be weak. If we believe all of these things at once, as most of us do, then we have no prior belief whatsoever about the correlation between burden and assistance and will find it to be uninteresting. So why the focus on this correlation at the end of the paper?
Which brings me to the biggest gap in the paper – one that the authors would not have been able to fill because of the way this data is reported. The DAC breaks down health assistance in many ways, including a distinction between budget support and project support. But there is no distinction between results-based health assistance like GAVI and the rest of health assistance that one way or another only funds inputs without reference to outputs. Since the use of performance based incentives is becoming more popular, I expect that when this study is replicated ten years from now, the future authors will be able to show that an increasing proportion of total assistance is extended in the context of some kind of results framework. And over time, I would expect that countries which receive a larger share of their assistance in this way will experience more rapidly falling disease burdens. Now that will be interesting.
June 20, 2009 at 3:31 pm
I would like to thank Mead Over for highlighting IHME’s research, Financing of global health: tracking development assistance for health from 1990 to 2007, published this week in The Lancet, and for emphasizing its value as a “first comprehensive point of reference” in tracking how much money has been spent on development assistance for health. In doing so, he raises several important questions about the research methods, the differences between our estimates and previous estimates in this area, and also some questions regarding the findings. We appreciate the questions and the opportunity to help clarify the research.
First, Dr. Over states that the main difference between OECD estimates and IHME estimates is that the OECD focuses on commitments, while we focus on disbursements. This is indeed one difference, but we would argue it is not the main one. The OECD’s coverage is much more restrictive than ours and the OECD is the first to acknowledge that it does not track private foundations and non-governmental organizations, which together constitute nearly 30% of total DAH in 2007, as per our estimates. Additionally, the OECD only counts some types of expenditures and not others. For example, the document referred to in the blog states clearly that at present the OECD data do not include core-funded activities of WHO or any GAVI expenditure.
Second, the observation that we “imputed” disbursements using methods with “heroic assumptions” is incorrect. In this research, we used observed project-level disbursement data to calculate a mean disbursement schedule and then used applied that to observed yearly commitments to estimate annual disbursements. There are no underlying parametric assumptions involved. As Dr. Over notes, it is preferable to know annual disbursements – actual dollars transferred – as opposed to commitments or promises of future payments. The ideal scenario would be for donors to provide complete disbursement sequences for past years as well as current disbursement data, which would obviate the need for any kind of estimation. As we indicate in the study, we are committed to providing measures of uncertainty for any quantities we estimate in future years when we update these numbers. It is also suggested that we did not provide sufficient documentation of our methods. We have provided over 40 pages of documentation for the study in a linked web appendix; I refer interested readers to page 19, where a full description of the procedure used to estimate disbursements is explained.
Third, Dr. Over raises the question as to why we focus particular attention on the correlation between aid and burden, when the distribution of aid across countries is a complex function of burden, cost-effectiveness, and a myriad of other factors. We acknowledged in the paper, and acknowledge it again, here, that many factors determine how much aid countries get. This is an area we plan to explore further in the future. As we are trying to carve out a “point of reference”, we found this a natural place to start. Regardless of other considerations, we feel that at some basic level, aid should follow need. Burden is one measure of need, albeit a crude one. Since in this particular case, we used burden measures from 2002, and aid received from 2002 to 2007, we had no reason to expect the relationship between the two to be negative.
Finally, it is suggested that GAVI’s aid to countries is pegged to outputs, unlike other aid which only funds inputs. Studies have documented how countries misreport vaccination rates in response to performance-based incentives and the editorial in The Lancet this week comments on the failure of global health initiative like the Global Fund and GAVI to rigorously evaluate their programs. It seems that Dr. Over is assuming that incentive-based funding mechanisms are having a positive impact on reducing burden, when such salutary effects have yet to be proven in a scientific way.
Again, we appreciate – and encourage – open dialogue on our methods and our findings in order to stimulate the opening of new avenues for consultation and collaboration, which will in turn serve to improve and strengthen the evidence base in the long run.
June 25, 2009 at 2:51 pm
Very useful to see this update — and great additional information re NGOs and philanthropic sources.
As to what we may learn in a future 10 year update, lets hope that by then practitioners of either ‘new’ or ‘old’ models have gotten around to making serious investments in building country information systems which enable local decision makers to better measure (and therefore improve accountabilty for) health outcomes and system performance. It’s way past time for the donors (of all stripes) to stop financing fragmented and duplicative project by project monitoring systems – an extractive industry all on its own –and started helping countries build the kind of population based information systems (yes, including civil registration and vital statistics) and analytic capacities that would enable them to stop trying to steer blind. Without these investments — all the RBF schemes in the world are going to stumble — as strongly attested by recent evaluations of GFATM and the Bank’s work in HNP (echoing findings from 10 years ago…themselves echos of findings of 20 years ago….)…
Both ‘old’ and ‘new’ models obsess over ‘fiduciary systems’ for financial management and procurement — why not put some more attention on a country’s systems for defining and routinely measuring performance and outcomes? Isn’t knowing whether the benefits you are aiming for are occuring a part of ‘fiduciary oversight’?
July 1, 2009 at 2:03 pm
Thanks to Nirmala Ravishankar, the first author of the study under discussion, and to Susan Stout, a friend and former colleague from my days at the World Bank, for commenting. Readers interested in this paper might also want to look at a comment by Peter Heller.
Ms. Ravishankar takes issue with my use of the term “heroic” to describe their imputation techniques. I agree that the methods they describe quite properly eschew the use of truly hazardous techniques, such as predicting a donor’s disbursements based on a cross-country regression on variables such as GNP per capita. But the estimates they do use involve estimating a number of parameters, including “mean disbursement rates” and “average under-reporting,” etc. I don’t fault them for this, but like Peter Heller I do wonder how their numbers compare with what would be computed by sticking strictly to observed disbursements.
Ms. Ravishankar also says that I assume “that incentive-based funding mechanisms are having a positive impact on reducing burden.” While the studies at the interface between provider and patient reported in the CGD’s recent book on performance based initiatives lend strong support to this claim, I meant to say that this hypothesis needs further testing at the country level. That’s exactly why this study’s inability to distinguish aid by this feature is disappointing.
Susan is suggesting another aid category that would contribute to future analysis: aid devoted to statistical data collection and management, and to monitoring and evaluation. Since all such measurement instruments are tools for performance based aid, and may have proved fallible in the GAVI example, such a category would indeed be interesting to track. If other donors’ experiences are like that of the World Bank, one finding I would expect would be that the ratio of disbursements to commitments is lower for this M&E category than for any type of aid. Reasons why donors and recipients commit to M&E in their plans but then change their minds and fail to disburse for it should be the subject of another discussion.
July 7, 2009 at 11:49 am
the missing “why” is always hard to disentangle, since country-targeted funding is often a mix of the positive (best performance), the negative (greatest need), and the geo-political (foreign policy, geopolitical and organizational interests). All three need to be monitored and understood, but as Susan notes, the M&E system to allow that to happen are poorly funded themselves!