Print
Share
Malaria Estimate Sausages by WHO and IHME
February 9, 2012
This is a joint post with Felix Lam.
Published last week in The Lancet, a new study by the Institute for Health Metrics and Evaluation (IHME) finds that there were 1.2 million deaths from malaria in 2010, not 655 thousand as estimated by the WHO. Following its release, headlines began splashing uncritically: “Malaria kills twice as many as previously thought, study finds” (The Guardian) “Malaria deaths hugely underestimated” (BBC). Which set of estimates is correct? Or at least which is less biased? Given the 1.2 billion dollars by donors to malaria in 2010, is it unreasonable to demand to know with more certainty, how many people are dying from malaria?
In an effort to gain clarity on these questions, we’ve ventured into the sausage-making of both malaria estimates (see below, if you dare venture into the methods of the sausage machine). In essence, both the IHME and WHO start with ‘raw ingredients’ (some data), then apply different ‘cooking’ techniques (some methods). From both of their ‘recipes’, it is not obvious to us that IHME’s estimates are necessarily less biased (or more correct) than WHO’s. Both IHME and WHO are doing the best they can, given shockingly unavailable and bad data. In Africa where 90% of malaria deaths occur, almost all mortality data come from verbal autopsy (VA) studies, where family are asked to report symptoms and diagnose the cause of a person’s death well after the death occurred. Malaria mortality estimates are taken from sub-national samples of this VA data and then extrapolated to the broader population. For example, one study in a Nigerian village with 227 malaria deaths was used as the main basis to extrapolate all malaria deaths in all of Nigeria.
One distinguishing feature of the IHME study is that it nobly tries to correct these VA studies, but even the validity of these corrections for estimates of Africa is questionable (see below). The other main factor contributing to the differences between IHME and WHO estimates is the extent to which the burden of malaria falls on children under age 5 compared to those older (see below). These two differences probably explain much of the discrepancy between IHME and WHO in counting malaria deaths in Africa. Beyond that, both IHME and WHO take fairly similar methods in Africa to extrapolating and modeling bad and non-existent data (although we’re not too certain since it’s a bit harder to find out what WHO did).
It would help if both IHME and WHO (WHO probably more so) were more transparent and made their data and methods available for interested parties to replicate their results. In any case, if we want to say anything reasonably straightforward without having to make sausages every year from VA data, better quality vital registration systems and data are needed. Data-production and sausage-making enterprises, fancier or simpler, are ultimately temporary coping mechanisms until countries themselves have better information and can correct it on their own. The long-term goal should be to not rely on people sitting in Seattle and Geneva for the ‘correct’ numbers. To realize this goal, we will need to value this investment of knowledge and country capacity as essential to public health. But until then, will donors continue to spend million or billions of dollars on malaria without really knowing how many people malaria kills each year?
The Method of the Sausage Machine: How to Count Malaria Deaths
IHME’s sausage recipe: what did IHME do?
IHME researchers compiled vital registration (VR) and verbal autopsy (VA) studies on malaria deaths of all ages globally. In Africa alone, the researchers found 106 VR and VA studies since 1980, which were almost entirely subnational VA studies. As an example of such ‘subnational’ studies, look at Nigeria (which according to IHME had 380,632 malaria deaths and the highest in Africa). For all of Nigeria the IHME paper used 1 subnational VA study in 1 Nigerian village which had 227 malaria/fever deaths among infants in 1995 (see Appendix, p. 100).
Using this VR/VA data as the main ‘ingredient’, IHME processed the data in two main steps. First, they attempted to correct the VA estimates given that malaria was likely misclassified. Why correct for misclassification? Malaria is notoriously known for being incorrectly diagnosed: a fever may be incorrectly classified as malaria when it is simply a fever, for example. Here is where the sausage-making starts. In the VA studies, IHME used a “validation” study from a “multi-site” study – and by multi-site they mean 6 sites in Mexico, Tanzania, India, and the Philippines which had a total of only 217 malaria deaths – to correct their VR/VA database to correct the proportion of malaria deaths. From this exercise, they shift estimates up by 21%. At this point you may be asking, “So, for all of Africa which accounted 90% of all malaria deaths – 1.1 million (IHME) or 596 thousand (WHO), they used 1 validation study in Tanzania (and perhaps the other regions) to correct the array of VA/VR data points?” It is tough to imagine that these sites can be considered representative of all Africa.
In the second main ‘processing’ step IHME ran statistical models to determine associations on malaria death rate and cause of death fraction and an array of predictor variables including malaria transmission rates, rainfall, bed net coverage, etc. Running literally hundreds of these models, they test the “predictive validity” in order to choose the best-fitting model and extrapolated in time and within region where they did not have data.
WHO’s sausage recipe: what did WHO do?
The WHO relies on two main sources of data. The first set are administrative data on the number of cases multiplied by a case fatality rate, estimated from previous studies, for each country and voila, the numbers of malaria deaths (p. 73 in WMR 2011); WHO used this data for low-transmission countries in Africa plus countries outside of Africa.
The second set of data involved compiling vital registration (VR), verbal autopsy (VA), and other mortality studies to estimate the percentage of deaths under age 5 attributable to malaria; WHO used this data for the remaining countries in Africa plus Somalia and Sudan. Sound familiar? Like IHME, WHO had no other choice but to mainly rely on VA data because of the lack of VR data in Africa. In addition, it’s not clear whether, like IHME, WHO also adjusted for VA misclassification (we suspect not). Next using the VR/VA data, WHO processed them by using statistical models to find associations between the fraction of child deaths from malaria and various country predictors (not clear what variables they use). As for those aged 5 and older, it seems that WHO deduced mortality rates using previous studies on age-specific malaria mortality rates and the intensity of malaria transmission (p. 140 in WMR 2008).
How do the results between WHO and IHME differ?
We compared the numbers by IHME and WHO by WHO Region (see Table 1 below). Both IHME and WHO agree that Africa accounts for 90% of all malaria deaths. There is agreement between IHME and WHO in the numbers for the Americas, Western Pacific, and Europe (regions which have better vital registration data). However, the numbers differ greatly for Africa, Eastern Mediterranean, and South-East Asia regions. This difference in South-East Asia is likely explained by WHO non-use of the latest VA estimates for the South-East Asia region (and similarly for the Eastern Mediterranean). But why do estimates in Africa vary so much if both IHME and WHO compiled and processed VA/VR data?
Source: Compiled by authors from IHME and WHO
One distinguishing feature is that IHME seemed to correct for misclassification. The other major question is how much of the malaria burden falls on children under 5 compared to people aged 5 and older. Will the IHME study overturn years of clinical observation that, by far, most of malaria’s victims are children? IHME says that in 2010 children under 5 accounted for 58% of the world’s malaria deaths (our calculations from IHME data), whereas WHO says it is 86% (p. 74). Yet when we looked by region at the percentage of malaria deaths aged 5 and older (see Table 2), we found that indeed both IHME and WHO agree that age distribution of malaria deaths differ by region and that at least in the Americas, South-East Asia, and Western Pacific, most malaria deaths occur in people older than age 5.
Source: Compiled by authors from IHME and WHO
Where IHME and WHO differ the most, in both the numbers and the age profile of death, is for Africa. How valid is either group’s estimates in Africa? It would depend on how much you believe each stage of the sausage-making process:
- Are WHO’s VA/VR/mortality study for child deaths estimates combined with review of other mortality studies for adult deaths sufficient compared to IHME’s VA/VR studies of deaths for all ages? (And how exactly did WHO calculate its adult deaths? Why is it hard to figure out what WHO did?)
- How valid is IHME’s “multi-site” validation study (with its 214 malaria deaths in 4 countries, of which only 1 is in Africa) to “correct” the VA data? Did WHO correct for misclassification?
- How sensitive are results to the choice of predictors in the models by both WHO and IHME?
- More importantly, how much does each stage of the sausage-making process contribute to the estimates?
Possibly Related Posts
- Saving Lives by Counting Deaths
- Promising Malaria Vaccine Is a Rare Bright Spot in Clinical Trials Labyrinth
3 Responses to “Malaria Estimate Sausages by WHO and IHME”
Post a Comment
We value frank and constructive exchanges and encourage you to use your real name in your comments.
February 10th, 2012 at 4:05 pm
Dear Victoria (and Felix),
Thanks for another excellent post.
I think you’re absolutely right to point out that the main takeaway from this paper is the urgent need to improve national malaria surveillance and health information systems in countries affected by the disease.
There also appear to be valuable data sources that are already available and were not used. I’m thinking in particular of published and unpublished studies by the INDEPTH network of Demographic Surveillance Systems, which has a wealth of information on causes of death in SSA and South Asia. It’s unclear why INDEPTH data wasn’t incorporated, but the extent to which the network should be expected to share data has been a subject of some controversy in the past. http://www.plosmedicine.org/ar.....ed.1001005
The potential for INDEPTH data to add to improve assessment of malaria risk is addressed in this paper.
library.wur.nl/frontis/environmental_change/09_sankoh.pdf
Keep up the great work!
February 11th, 2012 at 6:08 pm
You posed some great questions with this post and raised some important issues. We really appreciate this kind of exchange as it inspires researchers to improve on one another’s methods and policymakers to lobby for more reliable data collection. It also gives us the opportunity to explain some of the questions raised about IHME’s approach to mortality estimation.
1. Preparing the data and adjusting for potential misclassification
In your overview of what you describe as IHME’s “recipe,” you suggest that we use the results of a six-site validation study to adjust our verbal autopsy (VA) data for possible cause of death misclassification. You appear to be blending together elements of our main analysis and our sensitivity analysis.
In our main analysis, we corrected both VA and vital registration (VR) data for potential cause of death misclassification by applying the method Naghavi and colleagues developed in their paper, “Algorithms for enhancing public health utility of national causes-of-death data.” For selected studies, e.g. in Tanzania, we used information from a re-investigation by medical personnel in those sites to redistribute fever of unknown origin.
We also conducted a sensitivity analysis to test how our results would have changed if we had used different input data in our models. In this analysis, which is entirely separate from our main analysis, we corrected our VA data for the physician bias observed in the six-site validation study you referred to by Lozano and colleagues. The results of this side analysis suggest that our use of VA data actually underestimated the total number of malaria deaths, meaning that the true burden of malaria might be higher than what we reported.
2. The truth about Verbal Autopsy data
A couple of reports have inaccurately described what VA is, so here is a clear explanation. Data are collected by trained interviewers who visit families and ask standardized questions about the signs and symptoms leading up to an individual’s death. Subsequently, a team of physicians reviews the survey responses and determines the likeliest cause of death for each individual in the sample. So while VA diagnoses are not perfect, it is important to recognize that they are generated by a group of physicians and not family members. In the absence of death registration systems with medical certification of cause of death – which is the case for the vast majority of malaria endemic countries – VA is the best alternative. This is why VA data have become increasingly important for mortality studies and other research, including WHO’s World Malaria Report 2011.
3. Representativeness of the data
You also raise valid concerns regarding the representativeness of our data. You ask whether using subnational data to extrapolate to a broader population might bias our results. You are correct to note that using subnational data is risky. For instance, most of South Africa is malaria-free, but there are still selected regions with malaria transmission. We addressed this issue in the following way. We identified the precise location, i.e. the latitude and longitude, of every subnational study in our database. Then, using data from the Malaria Atlas Project, we estimated the location-specific level of malaria transmission at each of these sites. When we estimated the relationship between our predictor variables and our death data, we used subnational rates of malaria transmission. In this way, we linked high malaria mortality rates to high malaria transmission rates, even in countries with low overall transmission rates. But when we estimated malaria mortality for the entire country, we used a nationally representative value of malaria transmission. This approach properly adjusts for the subnational variation in malaria transmission and mortality.
4. The scarcity of data
You point out that our results are based on relatively sparse data (even after a systematic search for all available data sources) and highlight that for Nigeria, we used “1 subnational VA study in 1 village which had 227 malaria/fever deaths among infants in 1995 (see Appendix, p. 100).” A critical output of our study was to provide readers with not only the mean number of malaria deaths but also the uncertainty range around the mean estimates. In our paper we provide this for all countries and for four different time periods (1980, 1990, 2000, and 2010). These uncertainty estimates reflect the amount of data that is available for each country, i.e. they will be larger for countries like Nigeria. As we note in the paper the uncertainty around our estimates of malaria mortality is larger than they are for other causes, e.g. maternal mortality, which highlights the need for additional data collection on causes of mortality so that we can understand more precisely the extent of the health burden of malaria.
We wholeheartedly agree that one take-home message from this study should be that better quality data are needed. The second point is that even with the data challenges, there is strong evidence that malaria intervention programs are working. No matter which “recipe” was used, the result is a downward trend in deaths.
Thanks again for your interest in our work and giving us the opportunity to discuss further what we found to be an exciting study.
Sincerely,
Lisa Rosenfeld and Steve Lim on behalf of the Lancet malaria mortality authors
February 16th, 2012 at 10:07 pm
Victoria & Felix:
Thanks for a nice article!
As an aside on the need demonstrated by your article that sausage really shouldn’t be made in Geneva or Seattle with (a carefully cleaned-up version of) whatever we have on hand: Who (WHO?) has been responsible for strengthening disease surveillance systems in the past? Countries? Development donors?
I feel like ‘we’ often call for strengthened (surveillance) systems as something that *should* be done or that we bundle in with health systems strengthening or horizontal/diagonal approaches to addressing health care needs. But I actually don’t have a sense of whether there are any successful cases in recent years (perhaps outside of small island nations) of dramatically improved surveillance. Moreover, and perhaps I should be really embarrassed by this, I am not entirely sure I know what working parts go into a nationwide surveillance and vital registration system.
Did I miss the how-to manual and success story? If so, help. If not, it’d be really great to better understand what needs to be done, how much it costs, and who are some of the likely funding sources for such an endeavor.
And, to somewhat second John’s suggestion, it would be good to understand the extent to which lessons learned from INDEPTH could be scaled-up.