Meeting of the Minds? Researchers, Microfinance Leaders at CGD

March 1, 2010

On Friday afternoon, the Center for Global Development and ACCION International’s Center for Financial Inclusion convened a “roundtable” meeting of U.S.-based microfinance leaders and researchers to talk about the interactions between their two professional domains. What does the latest research tell us or not tell us about how microfinance works? What can can each group do to improve how it communicates about impacts? How can each best support the work of the other? The event was the brainchild of CFI’s Elisabeth Rhyne, and hosted at CGD. Comments were not for attribution without permission.

The CEOs of ACCION, the Grameen Foundation, and Opportunity International, namely Michael Schlein, Alex Counts, and Adrian Merryman, attended, as did people from Kiva, FINCA and CHF International. All these organizations run or support microfinance programs in many countries. Proxying for the research community were Jonathan Morduch of NYU and the Financial Access Initiative, Dean Karlan of Yale (he’s also a CGD non-resident fellow), and Iqbal Dhaliwal, Director of Policy at MIT’s Poverty Action Lab. Also there were Rich Rosenberg of CGAP, who has been both a practitioner and an analyst, and my colleague Liliana Rojas-Suarez.

Personally, I see the meeting as responding to the rupture created by the release last year of randomized studies of the impact of microfinance. Putting the question of whether microfinance reduces poverty on average to such a rigorous test was healthy—and impertinent. The studies found no clear effect on bottom-line indicators of household welfare, and the press unsurprisingly accentuated the negative. “Perhaps microfinance isn’t such a big deal after all” and “it looks like ‘microlending’ doesn’t actually do much to fight poverty,” ran real headlines. They posed a public relations challenge for microfinance groups, or at least for the network organizations based in the west. And I think they also forced some tough thinking within the groups about what we know about impacts and even how we should define success. The events also raised questions about whether researchers can write and talk about their work in ways that will reduce misleading coverage. For their part, researchers seem particularly interested in which questions they should study to produce the most useful knowledge for practitioners. The practical questions are what services to offer and how to target the people most likely to benefit.

After such a contretemps, it seemed therapeutic, shall we say, to bring some of the key players together to talk. Working at the intersection of rigorous research and practical ideas, CGD was pleased to host. Here are highlights. I offer these more as a participant than a perfectly even-handed rapporteur:

• Jonathan, Dean, and Iqbal presented the headline-grabbing work of 2009: the Hyderabad study, the Manila study, and Portfolios of the Poor.
• One person commented that it seemed strange that they hardly referred to the older, non-experimental studies. It was as if 20 years of research on the impacts of microfinance—much of it more positive—never happened. I agree that academia has some explaining to do: first the most prestigious study says microcredit reduces poverty, then it is overturned even as ambitious, young researchers arrive on the scene with new studies, hardly mentioning the past. Still, as I said in the meeting, I’ve delved into the older studies more than anyone, except possibly Jonathan, and that exploration has convinced me that such studies have serious problems. I told the story of how a randomized study contradicted previously influential non-randomized studies and cut breast cancer. For more of my take, see these posts.
• Rich (cited with permission) cut to the chase: “You gored our ox!” He was being friendly, not complaining. For some 25 years, microfinance organizations have put out the message that microfinance reduces poverty. Now his best reading of the evidence is: “we don’t know.” He asked whether the bad headlines had affected anyone’s funding but got little response.
• Someone pointed out that two big stories buffeted microfinance in 2009. The second was about microcredit bubbles. (See also.) I wondered aloud whether this meeting was fighting the last war. If yet another study emerges showing no average bottom-line impact, will the press even notice? Maybe the bigger threat is indeed the bubble story, especially if something happens in Bangladesh.
• Some argued that academics should take more care to write their papers to prevent non-experts from overestimating their scope. I think a typical research paper states a broad question for motivation (”does microcredit improve welfare?”), then zeroes in on a particular context such as the slums of Hyderabad. So obvious is it to the primary audience of such papers—academics—that you have to be careful about generalizing from one case, that this often goes unsaid. Then a non-expert can read the paper and think it is testing globally whether microfinance “works.” The question is, can researchers do better at writing for both audiences at once? Judge the recent papers yourself.
• Something that interested (only?) me was what the financial diaries made famous by Portfolios of the Poor tell us about impact. The agenda asked, “How to translate findings from the diaries into testable hypothesis?” To me, this almost implied that tracking people’s money management sheds light on the impacts of microfinance only by generating hypotheses for randomized testing, such as: “Does microcredit help people smooth consumption, so that they go hungry less often?” I think diaries are good for that, but to suggest only that is to undersell them. I doubt that Stuart Rutherford sees his research as merely generating hypotheses for randomized trials. In a world without limits, perhaps we could put every question to randomized tests in every context. In reality, the knowledge from such studies will always be fragmentary. Indeed, every approach to knowledge has distinctive limits. So the portfolios of the poor fools like me who invest energy in understanding impacts should be diversified. (Would that be epistemological pluralism?) That is why in the meeting I riffed on the three definitions of success at heart of my book (chapters 6, 7, and 8). Without question, microfinance has given many people more control over their lives (freedom) and built dynamic businesses and businesslike institutions. Both gains are central to the processes of economic transformation we call development.

You can follow any responses to this entry through the RSS 2.0 feed. You can leave a response, or trackback from your own site.

1. Rich Rosenberg Says:
   March 1st, 2010 at 12:06 pm

   I agree with David that financial diary studies can tell us things–important things, indeed–about impact. True, they can’t provide a rigorous demonstration of causality. But to take a single example, in presenting the findings from the Portfolios-of-the-Poor diaries, Jonathan Morduch made the sensible observation that learning about the wide range of informal loans and other financial arrangements available to poor people should temper our expectations that adding microfinance to the mix would make a huge transformative difference. The same, I would think, is true when we find that many of the users apply their financial proceeds to non-income-producing purposes.

   I’m enthused about having RCTs coming on stream. (Actually, I was disappointed to find that only a few more are in progress right now.) But the notion that RCTs are the only way to develop meaningful knowledge bearing on the effects of microfinance seems to me to be pretty hard to defend. Actually, I’m not sure that the RCT researchers themselves are making such a claim.

2. David Roodman Says:
   March 1st, 2010 at 12:14 pm

   You’re right, Rich. No one present made such a claim. Reading your comment, I realized that because I was generally avoiding attribution, my mention that a a line in the agenda “came close to implying…” could be read as innuendo that someone had made the claim. That implication would be wrong.

3. Ryan Calkins Says:
   March 2nd, 2010 at 10:13 am

   Thanks for letting us peek in on a meeting of the minds. Did the conversation extend beyond the discussion of microcredit?

4. David Roodman Says:
   March 2nd, 2010 at 10:38 am

   Ryan, it was indeed mostly about credit, for better or worse. All the studies presented were about microcredit, for example. The Dupas and Robinson paper on the impacts of savings got passing mention. I suppose it was the credit studies that generated the negative publicity, the public relations challenge, and the searching questions, so they got the attention.

5. Alex Says:
   March 2nd, 2010 at 12:13 pm

   I’m excited to hear about this meeting of the minds (and wish I could have been a fly on the wall!). There’s one more angle I’d like to add. I think that the studies that ‘gored the ox,’not only demonstrate impact, but are also excellent vehicles for improving impact. We’d be selling the research short if we just stopped at the findings and said, “OK, microcredit in Hyderabad didn’t increase income for households, case closed.” Rather, we should sift through the results (or perhaps the research community should make this more explicit) and see how the results could be used by practitioners to tweak products, practices and probably, impact. I’ve been reading the book Nudge lately (was out of the country when it came out so I missed all the hype) and so have started thinking about choice architecture as it applies to microfinance. For example, the Hyderabad study found that households that were entrepreneurially inclined used credit to purchase durables, cut down on temptation/non-durables, while households that were not entrepreneurially inclined tended to just take the loan and spend on consumption. The findings in Kenya by Dupas and Robinson showed that microsavings had a positive impact, but only for the small percentage of those households who actually used their accounts. I think these findings are ripe to be used by microfinance institutions, the choice architects, to tweak product design, marketing or other dynamics of their operations. Some impact evaluations are even explicitly designed with this in mind. For example a study done by Rohini Pande, Erica Field and the Centre for Micro Finance looks at the impact on investment decisions and repayment through a RCT that varies repayment schedules for clients of the Village Welfare Society (VWS) in West Bengal.

   Also, I would counter Rich’s comment and say that there are actually many RCTs on microfinance going on around the world, with at least 15 housed within the Centre for Micro Finance in India. A friend at the Microinsurance Network recently put out a website that details all the impact evaluations (some RCTs, other not) going on globally for microinsurance. In short, there’s a lot going on out there, probably a lot beneath the radar.
   http://www.microinsurancenetwork.org/workinggroup/impact/stocktaking.php

6. 51Give Says:
   March 4th, 2010 at 4:44 am

   It is so exciting to hear about leaders in the industry recognize that microfinance is a community and involved parties must share, collaborate, and learn from one another. It is also important to remember as these bridges develop that there are many MFIs that are based outside of the U.S. In fact, 51Give is a platform for MFIs that are based in China. So, while it is wonderful to see the microfinance community develop we hope it continues to develop as an international community.

   Learn more about Chinese MFI at http://www.51Give.oom

7. Alex Counts Says:
   March 6th, 2010 at 1:23 am

   I was the one in the meeting who questioned whether it is wise and appropriate to act as if two decades worth of scholarship about microfinance should be dismissed because there are a new wave of studies using different and perhaps superior techniques, and because one study has been cited for possible methodological flaws. We commissioned a rising star in the research field, Nathanael Goldberg, to assess studies published through 2005 that did not have serious methodological flaws and he found more than 90 and they are summarized in his paper, part of Grameen Foundation’s white paper series, titled “Measuring the Impact of Microfinance: Taking Stock of What We Know.” We are coming out with an updated version of this paper in 2010 that will incorporate the RCT studies and other scholarship. Let’s not let the (possibly) perfect be the enemy of the good. Goldberg now works for Innovations for Poverty Action, and by all accounts is doing great work there. His GF paper is available for free download from our website http://www.grameenfoundation.org

8. David Roodman Says:
   March 6th, 2010 at 9:15 am

   Alex, the report actually seems to have gone missing from grameenfoundation.org. But if you Google the title, you can find it elsewhere. Toward the end, it says:

   On the whole, the evidence points in two directions: 1) there is much to be enthusiastic about; and 2) there is much to discover about the many ways in which microfinance works and does not work for different types of clients. This brief examination of a significant swath of the literature makes it quite clear that no study is perfect, and many suffer from serious methodological flaws. However, it would be hard to read through all of the many positive findings in these dozens of studies—noting how rarely the comparison groups showed better outcomes than clients—and not feel that microfinance is an effective tool for poverty eradication.

   On the other hand, considering all the ways we have seen in which subtle differences between clients and comparison groups can affect the conclusions we draw, the evidence, as convincing as it is, is not quite good enough. It will be an enormous benefit to the entire industry when the first “incontrovertible” study is published. The only way to achieve this is through randomized control trials. Fortunately, the first of these studies is already underway.

   This seems to me to be saying, in mid-2005, that the evidence to date is very encouraging but randomized trials are in class of their own.

   I assume that when you write “one study has been cited for possible methodological flaws,” you are referring to my paper with Jonathan Morduch. I feel that this characterization is misleading in four ways. First, our paper actually challenges three studies, not one. Second, these include the one the Grameen Foundation report says “may…be the most reliable impact evaluation of a microfinance program to date”—not merely “one study” but the best. Third, I think it is not accurate in spirit to describe our paper as citing previous work “for possible methodological flaws.” In fact, the Grameen Foundation report points out that all the studies reviewed have “possible methodological flaws”—that is the thrust of the quote above. You could say that the Grameen Foundation report cites the whole microfinance literature for possible methodological flaws. More to the point, I think my paper with Jonathan is the academic equivalent not of a citation but an indictment (not to impugn the professionalism of the authors of the original papers). It is a long document packed with logic and evidence that the flaws are not merely possible but provable in academic court and important enough to generate wrong results. Fourth, the problems we demonstrate are not unique to the three studies. It’s not as if the authors made idiosyncratic mistakes like adding up the wrong numbers. Rather, the problems are of the type the Grameen Foundation report says are present in all non-randomized studies. Showing, as we did, that the problems are serious even in the best studies calls the whole non-randomized literature further into doubt.

9. Nathanael Goldberg Says:
   March 6th, 2010 at 2:25 pm

   Thanks, Alex, for the mention, and thanks, David, for the quote. Reading it, four years on, I can’t help but think it still summarizes my position quite nicely. I wasn’t able to attend the meeting we’re discussing here, but if I may offer my own perspective, four years on, it would be this:

   The paper I did with GFUSA summarizes a LOT of research that shows microfinance is welfare enhancing. Three randomized trials now show mixed results, which are in some ways disappointing (especially to those who have overhyped microfinance’s potential) and in other ways dramatically successful (especially for savings, but more work on that is underway).

   The GFUSA paper, even as it was written, noted that while some studies were better than others, none of them could provide the quality of evidence of a randomized trial. The paper was meant to capture what we knew about the impact of microfinance in the pre-RCT era, knowing that these studies would slowly be replaced by ones of better quality, the kind of quality appropriate to the massive investments made globally by donors, investors, and clients.

   So Alex is right in that we only have these few studies, which cannot yet replace all historical evidence on impact. But these few studies carry a lot more weight, because they are the best we have, by far. Personally I am very interested to see what we find from randomized trials of rural microcredit, and in my own work with IPA I am using RCTs to evaluate grant-based “ultra poor” programs.

   Best,
   Nathanael

10. Alex Counts Says:
    March 17th, 2010 at 6:21 am

    David,

    Thanks for bringing to my attention that the Goldberg paper dropped off our website (probably during a site revamp late last year) — we are reposting. Thanks also for all the information/links that are going on my summer reading list about impact, RCTs, etc.

    As I believe the Goldberg paper speaks for itself (as will our update to it later this year), I would like to say the following: I still think it is unwise to dismiss more two decades of scholarship because a handful of the best known (but not necessarily best) studies during that period have been called into question. (By the way, have the authors of these studies responded to your critique and if so is that exchange public?).

    I still think there are things we can learn from many of these prior studies despite their limitations (and I believe the RCT studies of microfinance have their own limitations). It seems to me that you are saying that the flaws you have found in the three studies you looked at are not based on flawed implementation of valid (non-RCT) research approaches, but that non-RCT approaches are inherently flawed (which allows you can dismiss dozens of studies without reviewing them). If this is true, are we saying that all social science research on impact and the like (in any field) is not valid unless RCT approaches are used? Or are you saying that there are other valid approaches that if implemented effectively can give us useful information and that RCTs (if well implemented) have some unique strengths and can add significantly to existing scholarship?

    -Alex Counts

11. David Roodman Says:
    March 17th, 2010 at 9:50 am

    Alex, thank you for your thoughtful questions. No, Jonathan Morduch and I have received no response from the authors of the studies that we replicated and questioned. (Mark Pitt, however, was quite helpful in the early stages of the work.)

    Yes, speaking only for myself, my critique of quantitative non-experimental studies of social phenomena is quite broad. In previous work, I have probed cross-country studies of the effects of foreign aid and financial system development on economic growth and almost invariably I have found problems. I’d point to my post a while back on Rossi’s Rules. Paul Rossi’s Stainless Steel Law of evaluation is: “The better designed the impact assessment of a social program, the more likely is the resulting estimate of net impact to be zero.” He wrote this and other laws to distill decades of experience evaluating social policies.

12. Alex Counts Says:
    March 17th, 2010 at 10:57 am

    So, leaving aside qualitative studies which have their own limitations, you are basically saying that apart from what we may know from RCT studies, we either know nothing about the impact of social programs/phenomena, or we know that they don’t work at all (i.e., nothing really works). I am not sure which you are saying (or perhaps you are saying both?) but it is quite a significant statement with major implications (including big support for RCTs in the future).

13. David Roodman Says:
    March 17th, 2010 at 11:23 am

    Yes it is quite a significant statement. I would not characterize it quite that way though. I would say that by and large what we can rely on comes from the RCTs of the last few years. We know that short-term consumer credit helped employed people in South Africa keep their jobs. We know that savings accounts helped women in a Kenyan market town save more and boost income. We know that microcredit in Hyderabad helped people start businesses and boosted profits at existing businesses. What we don’t have clear evidence of is a microcredit program affecting bottom-line poverty indicators

14. kate lauer Says:
    May 6th, 2010 at 3:07 pm

    David

    I just read your March 1 2010 blog post and look forward to reading your and J Morduch’s paper as well as the two papers on the RCTs.

    You may be interested to read the following paper regarding indicators and problems tied to the use of indicators: http://www.iilj.org/publications/2010-2.Davis-Kingsbury-Merry.asp