|
Independent research & practical ideas for global prosperity
|
|
|
|
Independent research & practical ideas for global prosperity
|
|
March 18, 2010As a Matter of Policy, Put Policies to the Experimental TestPosted by David Roodman in Uncategorized Tags: media, RCTsColumnist Tim Harford in today’s Financial Times (free with registration, I think):
What I like about this column is that Harford does not allow himself to become bogged down in the quagmire questions about what randomized controlled trials (RCTs) can and can’t tell us. That has been my focus as I have worked to understand what we know about the impacts of microfinance. He takes RCTs’ superiority as given (where they are practical) and proceeds to a moral call: those who advise and make policy on everything from education to child rearing should institutionalize experimentation. That, in order to support vital societal learning about what helps children learn math, say, what causes breast cancer, or whether sleeping face down kills babies. Comment »January 17, 2010Eavesdropping on a Microfinance ConferencePosted by David Roodman in Uncategorized Tags: debt trap, RCTs, the poor and their moneyCouldn’t make it to the annual research conference hosted by the Centre for Microfinance (CMF) and the College of Agricultural Banking (CAB) in Pune, India, last week? Neither could I. Not invited? Actually neither was I. But as far as I can tell, CMF is the brightest spot on earth for quality research on financial services for the poor. They are worth watching. I suppose this post is the next best thing to being there. Better, in jet lag and carbon terms. Abhijit Banerjee presented the CMF-executed Spandana RCT. I embed the slides here less for the content, which is familiar, than the form. Someone at the Poverty Action Lab makes slides like I do, and probably because of similar influences, in my case Edward Tufte and Andy Goodman. 3 Comments »December 29, 2009Randomistas Attempt Message ControlPosted by David Roodman in Uncategorized Tags: impacts, media, RCTsAs I mentioned, 2009 is a milestone year in the study of the impacts of microfinance. It’s also a case study in how subtle conclusions get distilled and simplified and misunderstood as they ricochet around the media. Remember Tim Harford plaintively tweeting, “Note to all microfinance enthusiasts: I DO NOT WRITE MY OWN HEADLINES”? It is tempting to blame “the media” for all this oversimplification. But the media responds to incentives such as people’s narrowing bandwidths and preferences for clear-cut conflictual stories. And the media pass those incentives back to people like me who position themselves as experts. And sometimes we respond. Anyway, it gets harder to blame the media as we hurtle into a media-R-us world. 2009 is the year of Twitter too. So what’s my point? Other than an invitation to blame yourself (which I’d prefer to blaming me), it is that public conversations about complex topics are inevitably cacophonous and inefficient, even comical. I guess the best you can do is keep at it. I gather that Abhijit Banerjee, Esther Duflo, and Dean Karlan, the leaders in testing microfinance with randomized trials, have felt somewhat dismayed at how their careful experiments and nuanced conclusions became “Perhaps microfinance isn’t such a big deal after all” and “Billions of dollars and a Nobel Prize later, it looks like ‘microlending’ doesn’t actually do much to fight poverty.” As blogging guests of Nicholas Kristof yesterday, they attempted to regain control of the message:
3 Comments »December 14, 2009Pearl’s Wisdom on CausalityPosted by David Roodman in Uncategorized Tags: econometrics, RCTs, readingI’ve had no regrets since I left mathematics almost 19 years ago. My last memories of studying it are of sitting in a library at the University of Cambridge, staring at lecture notes full of lifeless equations, struggling and failing to care. But occasionally I read something that reminds me of the beauty that can be found in math, and of the remarkable power of formal analysis. Then I see again what once made her attractive to me, if you’ll forgive the metaphor. That happened a few days ago as I read Judea Pearl’s Causal inference in statistics: An overview (hat tip to Holden Karnofsky). Everything I know about Pearl I learned from his web site. He works in the computer science department at UCLA and is older than he looks in the picture. He researched advanced electronic devices in the 1960s. He has apparently written the bible on Causality, which is to say, the formal study of what we mean when we talk about variables such as diet and health influencing each other. He just released this overview paper in September, which gives a “gentle introduction” to a subject he appears to have done much to develop. The analysis is beautiful, but not merely that. It is insightful enough about what statisticians in economics, medicine, and other fields do every day that I think the paper should be required reading for graduate students in all fields that use statistics to study causality. Reading the paper may be a particular thrill for me because I entered econometrics untutored, running my first regressions in 2002 for a project with Bill Easterly. I have learned econometrics on the fly, and only gradually grasped what I was really doing. From my point of view, Pearl has systematized a few things I had come to understand while offering a much greater vista. Comment »November 13, 2009Two Studies Do Not an Indictment MakePosted by David Roodman in Uncategorized Tags: impacts, RCTsOver on CGD’s main blog, my colleagues Jenny Aker and Michael Clemens explain Why MFIs Shouldn’t Be Freaking Out (Yet) about recent studies finding little clear sign that microcredit makes people better off:
The post is about why you shouldn’t generalize too much from one or two studies. This is the “Sure, air-dropping insecticide-treated bed nets into rural upper-middle-northwestern Uganda in the spring worked great for stopping malaria there, but will it work in central Laos in the fall?” challenge. Comment »September 29, 2009Expecting MiraclesPosted by David Roodman in Uncategorized Tags: effects of causes, RCTsAbhijit Banerjee in a joint interview with Esther Duflo by Philanthropy Action:
Comment »September 7, 2009Is Obama’s Budget Director an Epistemological Nihilist?Posted by David Roodman in Uncategorized Tags: RCTsThe Institute of Medicine’s Roundtable on Evidence-Based Medicine has the weighty job of strategizing about how the American health care behemoth should learn—that is, how doctors, scientists, and other players should generate knowledge about what works and how they should incorporate it into practice. This graph is from the Roundtable’s 2008 annual report: In the Chronicle of Philanthropy, Lisbeth Schorr cites this report as showing that even in medicine, conventional wisdom is stripping randomized trials of their gold standard status. (See also her Education Week article, this chart, and this working paper. Hat tip to Paul Isenman.) Schorr appears to be reacting against a new U.S. government policy, articulated by Office of Management and Budget director Peter Orszag on a White House blog, to factor rigorous evaluation into social program spending decisions. Orszag: 1 Comment »August 5, 2009My Brain Made Me Do It (Again)Posted by David Roodman in Uncategorized Tags: debt trap, psychology, RCTs, reading, transparency
Writing about this for the book just now forced me to push the analysis farther. I realized that transparency is in a way a distorting metaphor for disclosure. After all, pages of fine print are “transparent”: all the information is right there. Accepting that MFIs should give clients a clear window onto costs, there remains the question of how to describe those costs. The window must be framed. One of the most important ongoing developments in economics is the subfield called behavioral economics. The famous duo of Daniel Kahneman and Amos Tversky demonstrated through experiments that how information is presented (framed) often matters at least as much as the information itself for how human beings act. I’ll show you an example below. 2 Comments »July 21, 2009The Other Shoe Drops: 2nd Randomized Microcredit StudyPosted by David Roodman in Uncategorized Tags: impacts, RCTsIn February I enjoyed a preview by CGD non-resident fellow Dean Karlan of the results of his randomized experiment with microcredit in Manila. (Dean has other barely noteworthy affiliations.) Dean and coauthor Jonathan Zinman applied a randomization method they first used to good effect in South Africa. They persuaded the First Macro Bank in Manila to adopt a credit scoring system—a computer program that automatically approves or rejects loan applicants based on factors such as age, wealth, and years in business—and to tweak the program to randomly “unreject” some borderline applications. Karlan and Zinman posted the results of their experiment in the last few days. This is the second randomized controlled trial (RCT) of what is usually considered microcredit, the other being the J-PAL Hyderabad paper that came out in May. The lender in Karlan and Zinman’s earlier South Africa study made four-month loans mostly to people who could show pay stubs—what is more commonly labeled consumer credit than microcredit. Then there’s the new experiment with microsavings conducted by Pascaline Dupas and Jonathan Robinson. All in all, 2009 is a pivotal year in the study of microfinance’s impacts. So…did microcredit “work” in Manila? Mostly not, as far as the evidence goes. Rossi’s Stainless Steel Law has held: the better the study, the weaker the effect found. 6 Comments »July 20, 2009Microcredit RCTs in the EconomistPosted by David Roodman in Uncategorized Tags: media, RCTsThe Economist just carried a nice piece on the new, randomized generation of microcredit impact studies. The article covers two studies in brief: the one by Abhijit Banerjee, Esther Duflo, Rachel Glennerster, and Cynthia Kinnan of group microcredit in Hyderabad and the brand new one by Dean Karlan and Jonathan Zinman of individual loans to microenterprises in Manila. See my review for a fuller summary of the first and watch this space for a review of the second soon (update: here it is). Comment »July 13, 2009Rossi’s RulesPosted by David Roodman in Uncategorized Tags: impacts, RCTsA friend of mine who runs the evaluation division of the U.S. government’s Department of Housing and Urban Development once pointed me to a clever 1987 article by sociologist Peter H. Rossi, who distilled years of experience evaluating social programs into a few simple “metallic laws”:
Comment »July 3, 2009First Randomized Trial of MicrosavingsPosted by David Roodman in Uncategorized Tags: impacts, RCTs, savingsI fear I got swept up in the disproportionate focus on credit in pouncing to blog the first randomized trial of microcredit while neglecting its counterpart for microsavings. I learned about it from Jonathan Morduch’s blog in February just as I was launching this one. (Hat tip to Daniel Radcliffe for nudging it back up my reading list.) Pascaline Dupas and Jonathan Robinson, both rising stars in economics, performed the study. They did so outside the big evaluation houses, J-PAL and IPA, and have the (small) budget and sample to prove it. Their sample contains about 200 people, compared to nearly 7,000 households in J-PAL’s microcredit evaluation. Nevertheless, the paper strikes this amateur economist (have I mentioned I practice without a license?) as creative, careful, and seminal. 5 Comments »May 29, 2009First Randomized Trial of MicrocreditPosted by David Roodman in Uncategorized Tags: econometrics, impacts, RCTsI have blogged about the power and limitations of randomized controlled trials (RCTs). Overall, I am a believer. I think the question researchers impertinently ask practitioners—can we show statistically that microfinance is helping?—is worth asking. And non-randomized methods have largely failed to answer it with credibility. So in my view it was for decades essentially correct to say that we have zero solid studies of whether microfinance makes clients better off on average. But it is no longer true that we have zero studies. Now we have (drum roll)…one. Seriously, the new working paper by Abhijit Banerjee, Esther Duflo, Rachel Glennerster, and Cynthia Kinnan, along with an imminent one by Dean Karlan and Jonathan Zinman, is historic. (And in fact so is the earlier Karlan and Zinman RCT of “cash loans” in South Africa, which are not usually thought of as microfinance.) But once one gets over the excitement and carefully reads the new paper, the realization sets in that it contributes only incrementally to knowledge. Each RCT tells us a bit about what happened to particular groups of people at particular places and times. Banerjee et al. ask a big question—does microcredit “work”?—but do not pretend to offer a big answer. The PSD blog’s pronouncement, “The verdict is in on microfinance…And it’s not pretty,” is too sweeping. 4 Comments »April 14, 2009Roodman’s Law of Instrumental ValuePosted by David Roodman in Uncategorized Tags: econometrics, RCTsI just made this up:
A big challenge in the social sciences is to go beyond merely observing correlations to showing causation—e.g., that microcredit borrowing is not merely correlated with the well-being of households but increases it, on average. A common statistical technique for ferreting out causation is to use instruments, which are factors that are assumed to affect outcomes of interest only through a determinant of interest (caveat for experts: “…after linearly controlling for observed covariates”). For example, the Pitt and Khandker study sets up this picture:
1 Comment »March 9, 2009The Power of Randomized TrialsPosted by David Roodman in Uncategorized Tags: RCTs
For most of the 1990s, American doctors advised postmenopausal women to take artificial hormones to replace the natural ones dwindling in their bloodstreams. By the end of the decade, half of such women were on the drugs (source). Evidence from “cohort” studies, notably from the long-standing Nurses’ Health Study group of 200,000+ nurses, showed that women who took artificial estrogen and progestin experienced less heart disease. Comment »March 3, 2009The Rapid Rise of the Randomistas and the Trouble with the RCTsPosted by David Roodman in Uncategorized Tags: RCTsAs I mentioned, “randomized control trials” (RCTs) are proliferating in development economics, being used to study such questions as whether microcredit puts more girls in school. I have spent much time challenging non-experimental studies of the causes of economic growth. Trained as a mathematician, I am less skeptical of the proposition that foreign aid or financial system development typically speeds national progress than I am of the number-crunching economists have done to back such claims. I’m more of an aid regression skeptic than an aid skeptic. (Though my skepticism of the evidence does lead me toward agnosticism on the substance.) The biggest problem is that technical efforts to rule out reverse causality usually fail, so that we can’t be sure about what is causing what. By contrast, randomized studies compel respect. A researcher flips a coin to determine who gets a service; a year later, those who got it are happier, or healthier, or more stressed, or whatever, than those who didn’t. Short of the supernatural, the only explanation for the correlation is that the intervention caused the outcome. What’s to argue with? 4 Comments » |
|
|
CENTER FOR GLOBAL DEVELOPMENT
1800 Massachusetts Avenue NW, Third Floor
Washington DC 20036
Tel 202.416.4000 | Fax 202.416.4050
|
|
|
Print
Share
