Microfinance Groups, Feeling Misunderstood, Misunderstand Research

April 9, 2010

In Meeting of the Minds? Researchers, Microfinance Leaders at CGD, I told you about a meeting I co-hosted at the Center for Global Development with Elisabeth Rhyne:

Personally, I see the meeting as responding to the rupture created by the release last year of randomized studies of the impact of microfinance. Putting the question of whether microfinance reduces poverty on average to such a rigorous test was healthy—and impertinent. The studies found no clear effect on bottom-line indicators of household welfare, and the press unsurprisingly accentuated the negative. “Perhaps microfinance isn’t such a big deal after all” and “it looks like ‘microlending’ doesn’t actually do much to fight poverty,” ran real headlines. They posed a public relations challenge for microfinance groups, or at least for the network organizations based in the west. And I think they also forced some tough thinking within the groups about what we know about impacts and even how we should define success. The events also raised questions about whether researchers can write and talk about their work in ways that will reduce misleading coverage.

Yesterday, six of those network organizations, including four represented at the meeting, went public with another response to the public relations challenge from the new studies: a four-page document making the case that microfinance, while not a cure for poverty, is still a very good thing. Me not blogging it would be like the Podunk Daily News not reporting the election of a U.S. president, except that the Podunk Daily News is something I just made up.

Seriously, substantial parts of this document elicit inward “Amen!”s from me. Here are significant and reasonable statements from the American microfinance community about the role of microfinance in economic development. The document emphasizes that microfinance is about more than microcredit. It lists seven ways that poor people use financial services, thereby making a clean break from the tradition of emphasizing just one, microcredit for microenterprise. And the closing shows a laudable humility:

We need to be reasonable and measured in our claims for what microfinance can accomplish. To thrive, microfinance requires a host of favorable conditions, including a sympathetic regulatory environment, political stability, mission-driven practitioners, cost-effective delivery, capital adequacy and—most of all—clients ready and able to leverage its services.

Nor can microfinance be reduced to a sound bite. A host of critical and complex issues must be consistently considered and thoughtfully debated, including consumer protection, fair lending, transparency, acceptable profits, measureable impact, and capital-market investment, to name but a few. ‘Financial access’ and ‘building inclusive societies’ are more forthright terms to describe what microfinance delivers, without risking either simplification or exaggeration.

Microfinance takes effort to understand, and patience to exercise. It is but one mechanism in the toolkit of global poverty alleviation—although one that has clearly demonstrated scale, sustainability, results, and enormous further potential.

My major concern is about the way the document handles the uncomfortable challenge from researchers, more fundamentally, the way it manages the tricky relationship between evidence and knowledge. I see that while minds did meet at our February meeting—and I do believe it was a constructive afternoon—there was no true meeting of the minds. This document does not really understand what good research is, and how good researchers must think in order to do it. (Perhaps if my work made me more of a practitioner, I would detect a similar lack of understanding in the other direction?) Though the document speaks of researchers politely, it subtly betrays an inability to understand, therefore respect, their professional competence. As I write in chapter 1:

Some practitioners who work daily with appreciative customers roll their eyes at the researchers who insist we cannot be sure if microfinance reduces poverty on average. But pursuit of empirical truth is what researchers are trained for, and just as they should hesitate to tell practitioners how to do their jobs, so should practitioners acknowledge researchers’ competence in measuring microfinance’s social return.

The document responds first to the challenge from academics by citing those academics to the effect that reporters have mischaracterized their work. Fair enough.

Next the document frames the new studies:

…these studies, conducted over a short timeframe and with small sample sizes, are part of a much larger body of research conducted over the last 20 years that have explored how microfinance affects the lives and well-being of clients.

Implied here: These aren’t the only impact studies of microfinance; the older ones reflected better on microfinance and deserve weight. Unacknowledged is that the new studies are qualitatively more credible, just like the ones that upended previous literature and cut the U.S. breast cancer rate. “Short timeframe” is fair since the new studies look at impacts over just 15–18 months. But “small sample sizes” is right only if interpreted liberally. The studies’ samples, covering thousands of people, are not too small to support their conclusions about the short-term impacts of individual lending in Manila and group lending in Hyderabad, India. But certainly the samples are not representative of all poor people in the world, so we don’t know about impacts in other contexts. Perhaps that is what is meant by “small sample sizes.”

Then come the stories—one from each of the six signatories. Going by word count, this is the groups’ biggest counter-volley. “Numerous first-hand accounts of how microfinance can improve lives are available on our websites.” Indeed. Considering the number of clients these six groups represent, this may be the most filtered, unrepresentative collection of microfinance stories ever. (And surely it is a “small sample” next to the Hyderabad study’s 6,800 subjects?) For counterpoint, here are a couple of stories of the kind you won’t find on those web sites.

From BusinessWeek:

Hernández, 29, the daughter of a small farmer, says her mother purchased several knitting machines in 1992, but lacked cash for yarn. The equipment remained idle for years. A loan, Hernández thought, would enable her to buy nylon and more machines. She aimed to lure home her brother and two sisters, who she says are undocumented workers with restaurant and hotel jobs in the U.S. Over four years, beginning in 2001, she, her mother, and a sister took out a series of loans ranging from $200 to $1,800, at an [annualized interest rate] of 105%. They rolled one into the next and used the money to increase their weekly output from 800 dozen pairs of socks to 1,500 dozen. At their peak, they say they brought in $800 a week, more than enough to sustain an extended family of six.

Then things unraveled. Wholesale customers fell behind on payments. Compartamos’ steep inter-est rates took an unremitting toll, as Hernández and her relatives each missed several $130 payments to the lender. That was a lot for the rest of the 23-member borrowing circle to make up. Resentment sur-faced. Soon after Compartamos trumpeted her story in 2005, Hernández and her family were banished from the group.

Lacking capital, she has seen her production and earnings plunge to 500 dozen pairs of socks and $270 a week. Her siblings remain north of the border. Stoic about her tarnished accomplishments, she is uncertain about the future. “It’s been a huge effort,” she says, “and we’re barely afloat now.”

From Lamia Karim, “Demystifying Micro-Credit: The Grameen Bank, NGOs, and Neoliberalism,” Cultural Dynamics 20(5): 5–29 (2008), copying from an earlier post:

I saw that credit-related strife amongst members and their families were routine occurrences. Women would march off together to scold the defaulting woman, shame her or her husband in a public place, and when she could not pay the full amount of the installment, go through her possessions and take away whatever they could sell off to recover the defaulted sum. In circumstances when the woman failed to pay the sum, which happened several times a month in the NGOs I studied, the group members would repossess the capital that the woman had built with her loans. This ranged from taking away her gold nose-ring (a symbol of marital status for rural women, and removing it symbolically marks the ‘divorcing/widowing’ of a woman) to cows and chicks to trees that had been planted to be sold as timber to collecting rice and grains that the family had accumulated as food, very often leaving the family with no food whatsoever. The women who committed these acts did so at the exhortations of NGO officers, but they also considered these acts to be ‘protecting their investments’, and the defaulting woman as someone who had ‘broken faith with the community’. These acts were committed with the full knowledge of NGO officers, but the officers did not participate in these collective acts of aggression. Instead, they threatened to withhold future loans unless the defaulted money was recovered.

In instances where everything had been repossessed because of a large default, members would sell off the defaulting member’s house. This is known as house-breaking (ghar bhanga) and has a long history in rural society.

Of the recitation of success stories in the document, I see two possible readings. One is as evidence that microfinance does work. But by the same logic, my stories prove that microfinance doesn’t work. The other reading is as evidence that microfinance can work. But we can deduce that from common sense: surely microcredit in particular has made some people better off and some worse off. Credit always does that. I don’t think that microfinance groups can rest their case at proving opportunities if the opportunities sometimes harm. The question stands: what is the overall pattern of impacts?

Think about how you might answer that question with stories. How would you go about collecting the stories by which to judge? Probably you’d start to think about how to build a representative sample; how big you could practically make the sample; and how to distill the core elements of each story into a few common terms in order to allow summary statements, such as the percentage that are success stories. This kind of thinking—not fancy statistical arguments—is what these new microfinance impact studies are really about. It’s hard to see how to avoid it if you want to understand the impacts of microfinance. Yet this document does avoid it. Telling a few success stories not only hides that question of overall impacts; it denies the importance of generalizing carefully from instances. It denies, in other words, the value of serious research.

The signatories tack a final contentious caveat onto the academic research: being quantitative, it cannot “capture and analyze all the benefits of microfinance” (emphasis in original). Yes, life cannot be reduced to numbers. But social scientists have ways of measuring many things: happiness, trust in neighbors, role in household decision-making, income, spending. The methods are highly imperfect, but more reliable than story-filtering. Surely most defining aspects of poverty—earnings, type of roof, whether kids are in school—can be reasonably represented in numbers. Check out the Grameen Foundation’s Progress out of Poverty Index, whose web page contains no apology for quantifying.

Quantitative research seeks statistical power and objectivity by abstracting from particulars and studying many instances. That is both its strength and its weakness. It actually can measure benefits—outcomes—pretty well. Where it struggles—and where the disciplined story collecting we call qualitative research can shine—is in elucidating causal chains, the subtle and unguessed pathways along which life moves. That said, qualitative research like that of Lamia Karim quoted above can help us understand outcomes too. Helen Todd’s book is a great example; but even it does some serious quantitative analysis (on a small sample). I strongly doubt that the drafters of this document have access to insight about the impacts of microfinance on clients that is beyond quantification.

Having in a sense judged the statement on my terms, by scrutinizing its fidelity to the spirit of genuine inquiry, I should also try to judge it on its own terms. Will it serve the apparent goal of changing how reporters, donors, and the general public perceive microfinance? It is hard to know. More randomized studies are coming. If the next one also shows average poverty impacts statistically indistinguishable from zero, then maybe this document and message-honing exercise will prevent simplistic headlines about the “failure of microfinance.” The document contains good statements about the way microfinance enhances freedom and builds industries. Or maybe no one will pay attention to the third randomized study of microcredit anyway.

Arguably, what bothers me about the document will undermine the drafters’ goals. The American microfinance industry has not shown in this document that it has learned one lesson from last year’s publicity black eye: the danger of getting ahead of the academic evidence. While calling for more evaluations, the industry seems uncomfortable with the ones just completed, grudgingly dwelling on their limitations rather than what they teach, seeking to marginalize them with 20 years of inferior studies and cherry-picked stories. It seems to me that to the extent the industry substitutes gut beliefs (“truthiness”) for critical thinking, it risks further ugly surprises. These could take the form of more bad publicity or real people getting hurt by programs implemented and funded too much on faith.

In a post last month on CGD’s main blog I gave the last word to Stephen Colbert. I’ll do it again. Turns out he put “truthiness” into currency on the very first Colbert Report. In doing so, he made a serious point about the dangers of anti-intellectualism in policymaking:

Tweet  

1. Saira Says:
   April 11th, 2010 at 12:26 am

   Well said David! However, I think before the release of the next RCT on microfinance, the industry will have another publicity storm to deal with – namely the IPO of SKS (India’s largest MFI with over 4 million clients). Already exposés are appearing. As the following was posted by “Prakash” on the Microfinance Practitioners Yahoo listserv:

   The Economic Times, Mumbai, of 08 April 2010 reports that Vikram Akula, the founder of SKS Microfinance Limited, has become the ninth biggest individual income taxpayer in India, and has paid advance income tax of Indian Rupees 142 million (approx $ 3.55 million) during 2009-10. As Professor Sriram has shown in his paper, this tax was paid for the pittance of income of only Rupees one BILLION ($ 25 million) earned by selling, within 18 months, 10 Rupee shares of the company which were got allotted to himself on par in March 2007, aggregating Rupees 16 million, for a price of Rupees 103.91 per share!

   The paper referred to in the posting is that of MS Sriram, entitled: Commercialisation of Microfinance in India: A Discussion on the Emperor’s Apparel, a Working Paper of the Indian Institute of Management, Ahmedabad (W.P. No. 2010-03-04), published 2010

   The paper also describes dubious governance practices within SKS. However, the lack of sound, conflict-free governance in SKS is only exemplary of the poor governance practices that permeate the Indian microfinance industry (the world’s largest microfinance market). Governance continues to be a top-rated concern in the annual Banana Skins report on the industry’s performance. I think it would be in the better interest of “industry leaders” to focus their energies on instituting accountable governance rather than to continue calling for “sympathetic (i.e. soft) regulatary environments” and deflecting attention to these callous, penetrating problems through PR rebound stunts.

   Indeed, practitioners themselves, are revealing the sometimes negative impact of microfinance on clients. One practitioner from the Phillipines recently posted the following to the Imp-Act SPM (Social Performance Management) Network on a discussion thread related to client over-indebtedness:

   the local government is taking measures to avert incidences of over-indebtedness of MF clients – of MF clients in their area who turn to prostitution to be able to pay off the MFIs that they are indebted to. Another story was told of a husband who committed suicide thinking that the wife would be able to pay all her loans to different MFIs when she receives her insurance proceeds from these MFIs.

   There needs to be a hippocratic oath of “first, do no harm” that is upheld, honored, and respected by all MFI industry leaders and MFI executives. Then, surely the hard work of curing toxic and ailing governance practices can begin. Because in the end it is not just some industry Image that is damaged, but CLIENTS’ Lives!

2. Ryan Calkins Says:
   April 11th, 2010 at 10:39 am

   Is this an example of institutional inertia overcoming mission? Or is it an example of not seeing the forest for the trees? Both? (Or maybe neither? They may be right, with the positive impacts taking more than a couple of years to show up in research.)

   Each of these organizations have poverty eradication/alleviation as a part of their mission statements, with microcredit as a tool in their kit. It would be difficult, cognitively, to accept that microcredit might not be working–that, to stay true to their mission, they might need to employ a different tool.

3. Phil Says:
   April 12th, 2010 at 4:36 am

   I agree with Ryan’s comment. But there is really little to be surprised about in realising that microfinance organisations will always want to continue doing what they do (best), which is: microfinance. Aside from all the financial, political and cultural capital invested by the microfinance industry into its domination of early 21st century development discourse, the careers of thousands of well-educated people depend on the legitimacy of microfinance. If I were one of them, I would probably be trying to shift the goal-posts too; and likely not out of calculated self-interest, but as a result of “institutional blindness”.

   Jonathan, you are right that the standards of evidence applied in the latest public relations statement are low, but are they at all low by industry standards? The RCT studies were brave in the sense that a few academics who have dedicated most of their careers to advocating microfinance have published these results—though not without a bit of added spin, and with a blind eye to some of the epistemological issues you’ve addressed here. But, had their numbers turned out differently, these academics would have been fawned over by the microfinance industry, rather than becoming the subject of such feeble rebuttals as the Acción (etc.) one. I guess that for them it might have been worth the risk.

4. Alex Counts Says:
   April 12th, 2010 at 12:29 pm

   David,

   You have put out a thoughtful response to what I believe was a thoughtful statement by practitioners including Grameen Foundation. One point that I would raise is this: You seem to believe that the drafters of this statement lack an understanding of the qualitative impacts of microfinance. (You wrote, “I strongly doubt that the drafters of this document have access to insight about the impacts of microfinance on clients that is beyond quantification.”) I am not sure exactly what you meant but I was one of the drafters of this statement and I lived at the field level for six years spending much of it studying the qualitative impacts of microfinance (and wrote a book about my learnings). I have not only read Helen Todd’s book but also served as a research assistant to Helen in 1993 as part of my training to be part of an effort to extend her study to other MFIs in the region. Others who were involved in drafting this statement probably have some insights and experience with respect to qualitative impacts as well.

   Alex

5. David Roodman Says:
   April 12th, 2010 at 1:41 pm

   Alex, sorry if this bogs you down in semantics, but can you explain to me what a “qualitative impact” is? I imagine that microfinance affects many, many aspects of people’s lives. I don’t think of these impacts as qualitative or quantitative. Quantitative research and qualitative research each have strengths and weaknesses in helping us understand these complex impacts. The point that I was trying to make is that I don’t see that quantitative research as unable, by virtue of being quantitative, to roughly pick up the overall impact on poverty, to get the basic pattern, contrary to the message I feel sensed in this passage:

   it is extremely difficult for studies to quantitatively demonstrate the impact of microfinance. Such studies face two fundamental challenges: their ability to capture and analyze all the benefits of microfinance, and the duration of the study itself. [emphasis in original]

   I support what I see as a premise of the Grameen Foundation’s Progress out of Poverty Index (PPI), which is that we can actually quantify outcomes to a useful degree. We can’t capture the nuances of people’s lives, but we can get a sense of their, well, progress out of poverty.

   To put it another way, it seems to me that I could turn the argument in the statement around, to say that the impossibility of quantifying all the benefits is a “fundamental challenge” to the PPI. But I think that would be wrong.

6. Alex Counts Says:
   April 15th, 2010 at 6:54 am

   Thanks for raising the question about “qualitative impacts”. It was probably not the most precise phrase to use when asking you to help clear up confusion on my end from your original post that seemed to question some aspect of the knowledge/experience of the statement authors.

   What I meant is that there are certain impacts that are difficult to quantify for a variety of reasons. For example, increases in self-confidence among clients. Yet the fact that these impacts are hard to quantify does not, in my judgment, make them any less real. (However, they do not lead easily to generalized conclusions about microfinance, but neither do many RCT studies due to their own limitations.) I believe that the fullest picture of how microfinance performs can be gleaned by looking at the full landscape of quantitative and qualitative research on typical institutions, and among the former, experimental (randomized), quasi-experimental and non-experimental studies all have things to teach us. (The next edition of our “impact white paper”, originally written by Goldberg, will explore this idea in some depth — I am reviewing drafts now.)

   I spent years interviewing microfinance clients in Bangladesh in order to steep myself in their life histories and the realities of their ongoing struggles, which provided much of the content for my book “Small Loans, Big Dreams.” While some of what I observed could have been studied quantitatively, I made no effort to do so in that context (relying, to the extent I needed such information, on others’ research). Perhaps you disagree with some aspect of this or would suggest I use some alternative terminology?

7. David Roodman Says:
   April 15th, 2010 at 7:57 am

   Alex, reading your book, with its unparalleled, closely observed scenes from the lives of a few Bangladeshis, is a rare privilege. Almost never do we get such deep insight into the lives of others. I appreciate that you saw many things that are hard to quantify.

   But I would also think that you could quantify any particular aspect of what you saw in a rough but useful way. True, there is no scientific instrument to measure self-confidence; but neither is their for income (which the Portfolios of the Poor authors discovered took a lot of work to measure too). I bet you could roughly represent the self-confidence of the various women (and men?) you followed, just as Helen Todd categorized her 60 subject women according to their say in household affairs. In some sense, you must have, to the extent that you concluded that microfinance was raising self-confidence. At base, that’s a quantitative call. Indeed, you chose to devote much of your subsequent life to microfinance rather than something else; that choice involves a ranking, which is a kind of quantification, a crude mapping of life’s complexities onto the number line.

   My point is only that the limitations of quantification qua quantification are not the sort of “fundamental challenge,” the almost fatal flaw, seemingly implied by the language in the document. There are certainly problems with all quantitative data sets, but many of those apply to all information we have about the world, quantitative or qualitative, formal or informal.