O, snap.

… sorry for the econ jargon.

What I find most interesting is that recipients of this “aid” had to give up more lucrative jobs in order to implement their “gift.” Just how was that supposed to help these Kenyans, exactly? Good thing Kenyan economists weren’t shy about publishing their research. Thank you for sharing!

Agreed on all your points. However, I do think that it would be better (and maybe this will be done since it’s only a working paper) to be explicit about what reductions the study was powered to detect. It is true that the study shows that the income effect predicted by the MVP itself did not happen, which — if the main concern is comparing their promotional materials with independently measured outcomes. But saying ‘this study shows no effect on income’ is less informative than saying ‘this study shows there was not a 10% increase in income’ or something like that; ie, the study would be strengthened if it were more explicit in this regard, as it would head off one line of potential criticism. But this is a very minor point compared to the broader conclusions of the study.

First of all, the study looked at mean income, not poverty rates. These are related, of course, but not the same. Second, the authors used a lack of significance from a low-powered study to draw a conclusion about lack of effectiveness. This is never valid.

Second, as I commented at Marginal Revolution, the total income in control villages have a standard deviation that’s much larger than the mean, which indicates highly skewed data. (Both data sets are probably skewed, but the problem is particularly acute in the control group.) But t-tests are only appropriate for normal distributions. This data isn’t even remotely normal, so t-tests on untransformed data are out. The thing to do is to describe the datasets using medians, not means, and do a bootstrap test. Surely an economist should know about skewed distributions!

Since this is only a working paper, I really hope the authors reanalyze their data before publishing.

Thank you for this report. It seems highly unethical to carry out research and then refuse to publish the data collected, yet to use purported data for a continuation and promotion of the MVP strategy. Aside from the wisdom of allowing development to be dominated by economists, money is in short supply. Why continue to pour it down the toilet.

What do you think are the biggest errors of the project and how should similar amounts of money be spent, to better effect?

Thanks for your comment, Jane. The burden of proof lies on the Millennium Villages project to show that its project can accomplish its goals, variously described by the project’s leaders as a 50% decline in poverty rates or a 100% decline in poverty rates. Obviously, if treated and untreated households have indistinguishable incomes, the burden of proof shifts even further onto the project to prove that it can cause African families to “lift themselves out of poverty in five years time”, which is what it promised to its funders.

On your second point, yes the data used by Wanjala and Muradian are noisy, and one study does not prove zero impact. The failure to detect any meaningful income effect in a project that claims massive and lasting income effects is notable, but of course it requires replication with larger samples and at other sites. I agree, and that’s why I wrote that Wanjala and Muradian’s findings call into question the project’s short-term impacts, not that they prove one way or the other what those impacts are. More research is needed. Unfortunately the project, which is collecting income data at the sites, and has released copious data over the last two years about changes in non-income indicators, has not released any analysis of changes in income. (Why not?)

Note also that the MVP itself does not hold itself to the excellent standard of evidence you suggest. For example, when they tested for changes in three different indicators of child malnutrition across nine of their sites, in the paper we describe here, they only found a statistically significant decline in one of the three indicators. In the other two indicators, standard errors are huge and the indicator increased. But you won’t find any mention of that in the paper’s abstract, nor in the description of the results contained in the project’s fundraising reports. Those materials describe large, quantitatively precise effects of the project on child malnutrition. And that article is already published in a scientific journal.

Simon, thanks for your comment. I don’t think that the project’s refusal to share any of its data, even those that form the basis of published journal articles, reflects an ethical failing. It certainly prevents independent and objective analysis of the project’s impacts, which can lead to costly errors in the allocation of resources, but I don’t now believe that any of the project’s leaders are unethical researchers. I do think that by evaluating this costly project’s impacts in ways that are not rigorous, and impeding independent analysis, they are wasting a massive opportunity for the development community to learn what does and does not work in development policy.

I want to clarify that I don’t know whether the money spent on the Millennium Villages Project was wasted or not. No one can know that without careful and independent impact evaluation. I do have skeptical prior beliefs about the effectiveness of such money, given that many billions of dollars have been spent on similar village-level package interventions over several decades, with no lasting effects, as we describe here and less detail here. But that per se is not a reason not to try an intervention; sometimes it’s a reason to try an intervention and then carefully evaluate its impacts and cost-effectiveness. It’s the latter part on which the MVP is falling short. A careful and scientific impact evaluation may yet find that the MVP is a wonderfully effective and inexpensive way to eliminate poverty, if the project chooses to go in a more scientific direction at a later date. The problem is that the project has already asserted that it can and does end poverty, and asserted that such claims are backed by “peer-reviewed science”, when this is not the case.

So what would a better use be for the tens of millions that have gone to the MVP? Here I subscribe to Tim Harford’s view in his sharp book Adapt (which also discusses the Millennium Villages). A wide variety of things should be tried, and rigorously evaluated with regard to their impact and cost-effectiveness. Many completely different interventions are happening in the same countries, such as John Githongo’s Ni Sisi project in Kenya and Rakesh Rajani’s Twaweza in Tanzania. The leaders of both of those organizations are interested in rigorous impact evaluation. The only way forward is to try many different things, from a humble standpoint of profound ignorance about the causes of poverty, and evaluate our efforts at each step. One of the best descriptions of this approach is to be found in Poor Economics.

Thank you Michael for your reply and thank you also for the reading list. I can follow up the online ones. Perhaps ‘unethical’ is the wrong word but I thought failing to provide data that would allow an independent evaluation and making claims without supplying supporting data when these claims could be used to raise funding for this and similar projects would be unethical. I wouldn’t wish to claim that increasing productivity leads to increased income when this doesn’t appear to be the case. Yet many funding proposals do make this claim, I’ve made it myself. But I don’t wish to do so in the future. I read the draft and the authors do supply some good suggestions so overall, this has been a very interesting and informative exchange. Like yourself, I am also skeptical of such projects as MVP. I find claims made about it so far to be highly questionable and wonder how much money is being spent on publicity, damage limitation and sheer vanity. Perhaps that’s not unethical either, but something to be avoided.

Ron, your anonymous smear that I’m \cheering\ is inappropriate and incorrect. I’m profoundly sad that so many millions of dollars have been spent on an antipoverty project that may or may not have increased anyone’s average income, and I’ve never expressed any happiness about it. I suggest that you spend more time reflecting on the profound sadness of this situation, and less time sending anonymous insults to people whose thoughts and feelings you know nothing about.

Propensity score matching is not as rigorous a measure of impact as a randomized controlled trial, a fact of which I’m well aware, which is why we recommend randomized evaluation in our paper. Propensity score matching is, however, vastly more rigorous than the impact evaluation methods that the project itself has used in several \scientific\ publications, which are so far limited to before-and-after analysis without a meaningful comparison group (such as the just-published paper we describe here), and comparison of treated and untreated farm plots without any matching method at all, such as in the fertilizer study I discuss in the post. The project has systematically negated and denied that any better evaluation methods are needed or even desirable. Wanjala and Muradian took things into their own hands, and did better. I do cheer that.

Matt, thanks very much for your reply. Wanjala and Muradian’s study obviously does not prove beyond any doubt that the project had zero poverty impact at any of its sites. It is also not a huge, expensive, gold-plated randomized trial, and parts of its method are debatable, as in any impact evaluation study. But it is carefully done, and gives a more careful consideration to what would’ve happened without the project than any evaluation report released so far by the project’s internal evaluation.

It’s notable for that reason, and several others as well. It’s the first quantitative information we have about incomes at the sites (since the project, weirdly, has released none). It’s the first independent impact evaluation using primary survey data. And it’s notable because it finds no difference in incomes between treated and untreated households, a counterinuitive result. None of those things make it perfect or definitive.

But because the project’s internal evaluation has produced nothing like Wanjala and Muradian’s paper so far, W&M’s work highlights the important role that independent evaluation can play in all development work. That is the principal lesson of the paper at this point. We may learn later that the paper’s findings are held up by subsequent analysis, in which case there’ll be more lessons. But that remains to be seen.

Michael:

The bias from PSM is really worse than its putative cure for selection effects. A particular attractive feature of both RCTs and its closest counterpart RDDs is that the rules governing who gets into the program and who doesn’t is actually set out in the design. Effectiveness of the interventions can then be explored by examining i) eligibility criteria was actually honored int he actual program ii) examining using both quantitative and qualitative instruments and techniques how the intervention did in fact affect outcomes. Not just if things work, but why things work at all (or as a developing country based national, who actually shows up to work and what have they got to show for it).

To give the authors credit they actual posit some channels by which the intervention might affect outcomes (page 5). However the PSM, specfically the construction of the PSM speaks to none of that since we do not know the selection rules into the program. Hence the selection has to be simulated. This in turn has led to a great deal of rough and ready rules about how to carry out the first step, the selection into the program. As the authors observe (pg. 16) there is considerable uncertainty, let alone best practices to guide this first step. The authors use a logit but do not discuss why their selection equation in terms of its functional form was appropriate. It would have been useful at least to discuss a range of possible criterion such as maximiziing log likelihoods or minimizing AIC. Alternatively the authors could have discussed discrimination and calibration techniques to examine if their selection equation is a good predictor of those in and those out of the program (model validation techniques for discrete modeling).

Assuming CIA and SUTVA holds (not testable by definition) from a programming perspective if do we obtain a ATET based on the estimated PSM, what does that have to do with actual program implementation. Restricting comparisons to the common support to obtaining net impact has no literal meaning for program staff in making course corrections. As such if the authors did do a qualitative investigation among treatment and comparison groups the information gathered on program design would neither validate or invalidate or explain the quantitative results. Takeway if the statistical model for quantitative analysis is not based on what is actually known about the social program’s implementation i.e. assignment into treatment for allocating groups into treated and non-treated groups, not much for the program staff to do except gawp at the applied hacks!

Ron

PS:
Restricting themselves to STATA and canned estimations the authors should at the very least present some robustness checks such downloadable tools such as SENSATT or MBOUNDS to check to see if their results would hold in the possible presence of unobserved heterogeneity.

The state of the art is to use R, and genetic matching.

Better yet to talk with program staff to understand their targetting criteria for the actual treatment (Methinks a fuzzy RDD would be the appropriate design) villages and carry out a less glamorous and more useful process evaluation. By the way one point of good practice by the MVP folks, publishing their protocol for their prospective evaluation. A registry of prospective and on-going evaluations with full protocols would be an important public good.

Thanks for your reply, Jane. Null hypothesis choice is not just a part of statistical method in Bayesian statistics, it is a part of statistical method in all statistics. Every frequentist test tests a hypothesis that was chosen somehow, and that choice reflects people’s opinion of where the burden of proof lies. There is no exception to this. In the case at hand, the project has claimed vast economic effects, and the burden of proof lies on them to support those assertions, and therefore vast effects should be the null hypothesis. If the goal is to test claims of vast effects, it’s irrelevant to run tests where the null hypothesis is zero effect.

In this this blog post at the World Bank’s Africa blog, Gabriel and I explain a few of the profound errors in the child stunting paper. The most glaring is that they choose their “comparison group” as nationwide trends between 1986 and 2006 (Figure 3), a long period when—on average over the whole period—child stunting was not going down nationwide, and compare that to trends at the village sites between 2005/6 and 2008/9, a period when child stunting was going down nationwide in all but one of these countries. That’s a completely inexplicable decision, since nationwide data during the project were available, and they authors of the paper knew that they were available.

Note that their decision to use a comparison group during an irrelevant time period substantially exaggerates the effect of the project. This is great for fundraising purposes, but reflects poor scientific practice, and that article is published in a peer-reviewed scientific journal. It is much more egregious a decision than anything done by Wanjala and Muradian, who are independent analysts with no incentive whatsoever to make the project look good or bad. Wanjala and Muradian are engaged in an independent and technically competent exercise that does the best they could with the available data. Their findings are not final and definitive, but they are noteworthy and worrisome.

There are numerous other problems with that child stunting study, some of which we mention in that blog post.

Aha … I see a flaw in the study: Wanjala interviewed randomly selected households instead of ones that MVP picked.