Thanks for this thoughtful comment, Matt. I agree, and that’s a useful clarification: It’s not necessarily true that flu declines in the neighborhood of the clinic can’t be the result of the clinic, just because flu infections are down everywhere. Rather, the fact that flu infections are declining everywhere should shift the burden of proof. If I count up the decline in flu infections on my block and confidently declare that all of them are an “achievement” of my clinic, the burden of proof should lie on me.

Hypothetically, there might be some reason why the people on my block wouldn’t have gotten shots if not for my little clinic, but it falls to me to demonstrate that. If I simply assert that the full change was caused by my clinic, without further critical thought—which is what the MVP has done for many of its claimed impacts—I would not come close to meeting that burden of proof.

And if I assert that city-wide trends cannot even conceptually provide any useful information whatsoever about what was likely to have happened without my clinic—which, again, is what Sachs and Singh do—that’s when I would be entering the realm of the ‘indefensible’, ’baffling’, and ‘absurd’.

Michael, I almost always agree with you but not here. You’re not giving Sachs the benefit of the doubt.

I read Jeff Sach’s book a long time ago and back in 2005 he was basically saying that if you implement this laundry list of simple interventions, things are going to get better. I’m not sure he thinks MV does all those things better than other NGOs. Did he ever say it? I’m almost certain he doesn’t think Africans have nothing to do with the success Africa is having with malaria.

That said, I do believe that MVP’s reports probably are over the top in their claims. Aren’t all NGOs? Where’s the harm? It’d be obvious if the aid pie is static and MV is stealing from other, better uses of money but I think the political economy of aid is a little more complicated. Shouldn’t there be an endogenous budget constraint and model for how funds get allocated? Sachs has certainly shifed the budget constraint outward with his lobbying/writing and his advocacy for bed nets has, through dump luck or shrewd judgment, shifted resources away from relatively bad uses (pretty much everything that’s not health) to good ones (nets).

The MVP and Prof. Sachs in particular, have to be commended on his ability to raise funds. An additional $72 million …

The MVP has to do more to demonstrate that over the years, it has indeed made some difference to the target communities. I agree 100% that a simplistic before-and-after analysis falls miserably short for a project that has drained several million dollars over the years. Any efforts seeking to expand the project into new locations should be guided by a strong, solid evaluation and not the broad brush evaluation approach.

I have read, with some degree of trepidation I must admit, Jeff Sach’s End of Poverty. The MVP seeks to implement some of the ideas contained in the book. What I found miserably lacking in End of Poverty was an appreciation of the local political, social and cultural dynamics of any community in which a direct intervention is being proposed. Perhaps it would be worthwhile to have targeted evaluations (or reviews) of specific components and approaches that underpin the MVP. I would be interested if anyone can direct me to any past evaluation of the MVP that has a political economy angle in it. Any attempt to introduce the project in it’s current form in highly complex societies (e.g in Papua New Guinea) would potentially be met with miserable failure. It’s time for a serious, robust, unbiased and professionally designed evaluation of the MVP

Erin Trowbridge, who comments above, is Director of Communications at the Earth Institute at Columbia University where the Millennium Villages Project is housed.

I would like to call attention to the fact that CGD’s blog is a forum for open discussion, which is why we clear and publish comments by Erin Trowbridge and all others offering substantive commentary as she and others do above.

It’s ironic that we’re willing to publish Trowbridge’s comments, because the Millennium Villages Project blog, in contrast, blocks substantive comments on their blog. I know firsthand that thoughtful and substantive comments have been submitted on this MVP blog post over the last few days, comments that happen to be critical. But those comments have not been cleared or published by the staff who run that blog, staff who presumably answer to Ms. Trowbridge.

As of Wednesday evening October 26th, no comments whatsoever appear below that blog post. This is not a sign that readers have had nothing to say about Sachs and Singh’s notions about evaluation, it’s just a sign that the MVP does not want the public to see what readers have had to say.

This irony is especially acute because the MVP is housed at Columbia University. If universities are unwilling to promote open discussion, who is?

CGD’s choice to actually clear and publish the comments submitted to its blogs are part of what distinguishes a genuine blog from a propaganda apparatus. I’m proud of CGD’s choice.

I have followed the MVP project for some time now. A frank and open discussion is long overdue and I have always got a sense that the project are not very receptive to criticism. I am keen to hear from others on their experiences with how the project has been marketed into new countries. With the increased resources and the project’s fundraising prowess, I imagine they are seeking to expand the initiative into other countries and are engaging with both donors and host governments alike.

Sachs has painted himself into a corner with his claims for the MVP. It is nevertheless appalling that he would go on record with these defenses. I am actually not against the MVP approach. It does help, but it is not sustainable and the MVP approach is almost certainly no better than similar approaches taken by CARE, Save the Children and many other worthy NGO’s. Moreover, I suspect that the cost of the MVP’s has been higher than similar approaches. Does the evaluation address a cost per beneficiary and factor in the expensive technical assistance from Columbia? The real question is who is putting another $72m into this approach?

Regarding Jeff Barnes’ above point about the full costs of the project, an update: George Russell has put together some incomplete but useful data on the full costs of the Millennium Villages Project, by going through financial filings of U.N. agencies.

I take no position on whether or not those data reveal a conflict of interest, as Russell claims. But apart from that claim, the raw numbers that Russell uncovers provide a fuller picture of the true costs of the project than has been available before.

[...] Why would I do this, and do it this way?  Well, I believe that in a general way those of us working in development have very poor information about what is actually happening in the Global South, in the places where the challenges to human well-being are most acute.  We have a lot of assumptions about what is happening and why, but these are very often wrong.  I wrote a whole book making this point – rather convincingly, if some of the reviews are to be believed.  Because we don’t know what is happening, and our assumptions are wide of the mark, a lot of the interventions we design and implement are irrelevant (at best) or inappropriate (at worst) to the intended beneficiaries.  Basically, the claim (a la Sachs and the Millennium Villages Project) that there are proven development interventions is crap.  If we had known, proven interventions WE WOULD BE USING THEM.  To assume otherwise is to basically slander the bulk of people working on development as either insufficiently motivated (if we weren’t so damn lazy, and we really cared about poor people, we could fix all of the problems in the world with these proven interventions) or to argue that there simply needs to be more money spent on these interventions to fix everything (except in many cases there is little evidence that funding is the principal cause of project failure).  Of course, this is exactly what Sachs argues when asking for more support for the MVP, or when he is attacking anyone who dares critique the project. [...]

Thanks for your comment Rob. I’ll ignore the personal swipe against me with which you close your comment. I’ll say nothing about you personally, as I’ve said nothing about the leaders of the MVP personally, and I’ll focus my response as usual on the substantive issue of what you and the MVP are *claiming*.

The comment that Sachs requested from Charity Ngilu occurred in the context of a year-long discussion of whether before-and-after analysis is an appropriate method of quantifying the impact of the MVP. The MVP has claimed again and again for the past year that before-and-after analysis at the sites is an accurate measure of the impact of the project. They have never conceded that the full quantitative impact of the project differs even slightly from the before/after difference in outcomes observed at the sites. Instead, they have rejected that comparing the change at the sites to the change in any other area is even a legitimate thing to do.

They have not just done this for the malaria scale-up, as I wrote in my piece above. They have done it for primary schooling, vaccination rates, cell phone ownership, crop yields, skilled birth attendance, and several other outcomes. They claim the full change at the sites as their “impact” and they have never retracted or modified those baseless statements. We show that most of those indicators, at most sites, have substantially improved across the large regions and countries where the villages are located.

The project feels that it and its leaders are substantially responsible for all those other changes as well. Not just malaria control, but all of them, because the question being discussed is whether or not changes in *all* of those indicators at other sites is a legitimate comparison group. And not just in part, because the question being discussed is whether or not before-vs-after measurements yield anything close to an accurate measure of the full “impact” of the project. In other words, it was in the context of a specific, year-long discussion that Sachs requested Ngilu’s statement and posted it without comment on his official blog.

If you read what I wrote again, you’ll see that I did not claim — and I would not claim — that Professor Sachs individually had *no* impact on Kenya’s anti-malaria scale up. I think it’s likely he had some impact on that, but I don’t have the expertise to know whether or not that’s true, and your assertion that I make claims I’m not qualified to judge is ridiculous. What I did and do claim is that in order for the full before-vs-after change at the sites to measure the “impact” of the project, the project and its leaders would have had to cause the *full* change in *all* of the indicators where they’ve claimed this impact, *away* from the sites.

That claim is ludicrous, it deserves ridicule, and I will continue to ridicule it, whether I sit in Sauri or Antarctica. It is furthermore offensive, since the implication that the *full* change off the sites, in *all* of those indicators, wouldn’t have happened without the project or its leaders is a wholesale devaluation of the efforts that all Africans make toward fostering the development of their own communities and countries.

Erin Trowbridge, above, accuses me of misrepresenting MVP researchers’ statements. She does not identify a single incorrect statement I’ve made, preferring instead to make a blanket, unsupported accusation. In my post above I identify exactly which statements by MVP researchers have no basis in scientific evidence, and I offer thoughtful evidence on why those statements are baseless.