New Challenge to Studies Saying Microcredit Cuts Poverty

June 18, 2009

No post on this blog will I write with less objectivity than this one. CGD and the Financial Access Initiative have just released a working paper by Jonathan Morduch and myself that critiques what have been the leading studies of whether microcredit reduces poverty. We recreate three well-known analyses of survey data collected in Bangladesh in the 1990s. They seem to show that microcredit either increased household spending or reduced its volatility (meaning there is enough to eat more of the time). In our view, they fail to show either benefit. I’ve already blogged the general problem: without introducing an artificial random element in what you study—without experimenting—it is really hard to prove that A causes B instead of B causing A. If prosperity and borrowing go hand in hand, which causes which? Economists do what they can to mathematically distinguish competing theories but I have found that they succeed less than they realize.

Another highlight: when we re-run the complicated regression that is the source of a statistic Muhammad Yunus has cited, that “5 percent of the Grameen borrowers get out of poverty every year,” we get the opposite sign. Seemingly, lending to women makes families poorer…but I just told you how much credence we put on such claims about cause and effect.

Bottom line: the academic evidence that microcredit reduces poverty is really weak.

From my point of view, the story goes like this:

1991–92. With funding from the World Bank, and in cooperation with the Bangladesh Institute for Development Studies, economists Mark Pitt and Shahidur Khandker field a survey of some 1,800 households in Bangladeshi villages, visiting each three times, in three successive seasons.

1996. Pitt and Khandker (PK) circulate a World Bank working paper analyzing this data using complex mathematics and concluding that microcredit increases household spending, especially when given to women.

1998. The study appears in the prestigious Journal of Political Economy and becomes the leading analysis of the impact of microcredit. “[A]nnual household consumption expenditure increases 18 taka for every 100 additional taka borrowed by women…compared with 11 taka for men.” But a young economist named Jonathan Morduch circulates a draft paper that applies much simpler methods to the data and reaches different conclusions. Microcredit does not seem to increase spending, but it does appear to smooth it out from season to season. Morduch questions key assumptions in PK.

1999. Pitt retorts, seeming to rebut Morduch’s criticisms one by one. Neither Pitt nor Morduch uses the other’s methods, so no direct confrontation between the seemingly contradictory results occurs. For interested bystanders, the exchange is as enlightening as two nuclear engineers arguing over obscure properties of plutonium isotopes. Meanwhile in Bangladesh, surveyors revisit the households of 1991–92 to collect more data.

2003. Khandker attempts to rise above the old fray by studying the superior, augmented data set.

2005. Khandker’s paper appears in the World Bank Economic Review.

2007. In my drive to understand what we know about the impacts of microfinance, I determine to get to the bottom of the unresolved, confusing methodological debates by recreating the old studies. I write a computer program that makes it much easier for people to perform regressions (statistical analyses) like those in PK. in time, I collaborate with Jonathan Morduch. Rerunning old regressions and applying new statistical tests, we show why economists should doubt the positive conclusions in the previous Pitt, Khandker, and Morduch papers. The arguments are inherently technical. Most of the number crunching is new, but much of the conceptual critique traces to Jonathan’s earlier paper.

2009. I write, “in my view it was for decades essentially correct to say that we have zero solid studies of whether microfinance makes clients better off on average.” Now you see why.

Note well: I am not saying that microcredit, much less microfinance as a whole, is bad for poor people. For me, the take-home lesson is that social scientists and promoters of social programs respond to incentives to overestimate and exaggerate the power of mathematics to enlighten us about causality in social systems. Math does not substitute for wiser reflections on the nature of development and how financial services can contribute to it.

Tweet  

1. Zvi Galor Says:
   June 20th, 2009 at 2:11 am

   You forgot one thing: This industry of MC and of MF give the chance to so many people to find their livings, by being employed in the organizations which supply the finance, as well as for those who are flying from continent to continent to participate in international conference on the benefits of the microcredit and poverty alleviation it supposed to bring.

   Even the false myth that Dr. Yunus invented the mutual guarantee has no sound legs in the reality.

   Thanks for putting lights on the issue.

   Zvi

2. shafiur Says:
   June 21st, 2009 at 4:08 pm

   Professor David Hulme’s recent paper goes into the history of Grameen and explains some “silent” parts of it and aspects of its restructuring and move away from the “poorest of the poor”…. and how the rhetoric does not match the reality.

   http://papers.ssrn.com/sol3/pa.....id=1300930

3. David Roodman Says:
   June 21st, 2009 at 6:14 pm

   Thank you for pointing me to this. I just read it and highly recommend it. It’s a short, readable, insightful history of the Grameen Bank.

4. Asif Dowla Says:
   June 23rd, 2009 at 11:44 am

   David:

   The takeaway message (I do not agree completely with the first part) of Hulme’s paper reinforces your message about microfinance that it provides agency to the poor (your interview in the CGD blog):

   While the international message associated with the Bank – microenterprise credit for extremely poor women lifts them out of poverty – is now inaccurate, the broader thrust of this message – of hard working poor people using their personal agency to overcome the problems they face – is highly appropriate for the publics and politicians of rich countries.

   The major problem with microfinance is that it has little spilling over effect. It helps the borrower and the family members, but it does not affect other people. This limits the poverty reduction impact of microfinance.

   David Hulme further adds (I agree with this wholeheartedly):

   The Grameen Bank today is a very different organisation from what it was 20 years ago, but it still serves as an inspiration for those trying to help poor and low- income people in their ownefforts to improve their lives.

5. David Roodman Says:
   June 23rd, 2009 at 1:49 pm

   Asif, I’d be very interested to hear where you disagree with Hulme in his first part, since you were an eye-witness to and participant in the history. Perhaps you have already written it somewhere?

6. Rob Says:
   June 29th, 2009 at 5:02 pm

   David, this is really topical, as Manuel Bueno over at NextBillion.net (which – full disclosure here – I am the editor of) just wrote a post arguing that microfinance DOES in fact reduce poverty.

   I wonder if you have a second to skim Manuel’s arguments and see where they differ from yours, whether from a pedagogical or data or other perspective.

   Thanks for a great blog here.

7. Mac Says:
   August 6th, 2009 at 12:52 pm

   Very interesting review.

   I am sure you are aware, but since you don’t make note of it in your article, let me point out that recently the sector has finally started addressing these concerns with randomized control trials and several groups are making strong advancements in the area. There is a lot more work to be done, but at least the evidence and studies are starting to come in.

   Centres that do a lot of work include The Centre for Microfinance (CMF), a collusion between Indian and US research teams (http://ifmr.ac.in/cmf/research.html). CMF has many smaller studies out, but has also more recently released one of the first large-scale random trials of the Indian MFI Spandana. Other centres include the Poverty Action Lab at MIT (http://www.povertyactionlab.org/) and Innovations for Poverty Action (http://poverty-action.org/)

   So we’ll see if we get better evidence. It will not be solved tomorrow, it is a very complex field with a lot of model variations.

8. Shahid Khandker Says:
   March 14th, 2011 at 6:25 pm

   I believe that Roodman’s assertions about serious problems with our “well known” studies are without merit. His co-authored (Roodman and Morduch) paper referred to was rejected for publication for cause by the Editors of Journal of Politcal Economy after it was anonymously refereed. This is the prestigious journal in which the most widely cited of the studies was published. My coauthor (Mark Pitt) and I stand behind the results of our published study and believe that the Roodman and Morduch paper is faulty.

9. David Roodman Says:
   March 14th, 2011 at 8:07 pm

   Shahid,

   We are each entitled to our own interpretation of the data. But I would point out one major asymmetry in this debate. Jonathan Morduch and I have been completely transparent about how we reached our published results. We have posted our full data set and computer code showing exactly how we go from the survey data to the results tables in the papers. Pitt and Khandker have not done this. Shahid, you have in fact resisted disclosure. One can define “science” in many ways. According to one definition replicability is essential for an analysis to be science. Right now our results are replicable while the originals are not.

   The lack of complete disclosure is one major reason it has been so hard for us to replicate the originals. For a set of reasons laid out in the paper, our confidence in our results remains high. For the Pitt and Khandker (1998) paper at least, our best guess at this point is that the discrepancy is caused by a programming error in the original. For Khandker (2005), we are are less doubtful of the results than of the causal interpretation attached to them, based on additional statistical tests we run.

   It seems to me that the most constructive step toward resolving this debate would be for Pitt and/or Khandker to share the exact data and code that generated their results. Otherwise, seemingly, people should trust our results since they can watch them being made. Right now your stance strikes me as sort of like “My results are right but how I got them is secret.”

   It is correct that the prestigious JPE rejected our paper. However, the more prestigious the journal, the less worrisome the rejection. Many good papers don’t get into the JPE. We are still hopeful of publication in another journal.

   –David

10. Shahid Khandker Says:
    March 14th, 2011 at 8:31 pm

    David,

    As you know, the data are already in the public domain. We are reviewing your program and will let everyone know why you have produced different results. This would also be a test of the credibility of our results vis-a-vis yours.

    Shahid

11. David Roodman Says:
    March 14th, 2011 at 8:53 pm

    Shahid, yes, the raw survey data are in the public domain (despite your best efforts to block the sharing of the second half of it). But there is a huge amount of complexity in the processing of that raw data into published results, all of which should ideally be public: first transforming the raw survey data into the data rectangle for analysis, then the actual analysis. This is why the literature on replicability emphasizes sharing data and code. See for instance:

    Richard Anderson, William H. Greene, B. D. McCullough and H. D. Vinod, “The Role of Data/Code Archives in the Future of Economic Research,” Journal of Economic Methodology 15(1), 99–119, 2008.

    We have shared our processing but you have not.

    I am pleased that you are now examining our code and look forward to your feedback. It would seem to me that if the goal is to get to bottom of the disagreement as expeditiously as possible in order to serve public understanding, then it would help for Jonathan and I to be able to do the same in return. That would require more transparency on your part.