Roodman’s Law of Instrumental Value

April 14, 2009

I just made this up:

For a study to teach us about the world, the assumptions on which it rests must be more credible than the assumptions that it tests.

A big challenge in the social sciences is to go beyond merely observing correlations to showing causation—e.g., that microcredit borrowing is not merely correlated with the well-being of households but increases it, on average. A common statistical technique for ferreting out causation is to use instruments, which are factors that are assumed to affect outcomes of interest only through a determinant of interest (caveat for experts: “…after linearly controlling for observed covariates”). For example, the Pitt and Khandker study sets up this picture:

landholdings => microcredit borrowing => household well-being

The first arrow says that how much land a household owns before it starts with microcredit affects how much it borrows; in fact, in 1990s Bangladesh (Pitt and Khandker’s study setting) owning enough land formally disqualified one from borrowing altogether. The second arrow embodies the hope that microcredit makes households better off, as measured, say, by their spending on food and other needs and wants. But by assumption no arrow runs from land directly to well-being. Landholdings are held to affect consumption only through microcredit. So if we observe in the data that the things on the two ends of the diagram are correlated—moving up and down together—then both the arrows in between must be at work. In particular, microcredit is making a difference. Here, we say that land “instruments” for credit; and having the first arrow, running from the instrument, lets us study the second arrow.

(I don’t mean to pick on Pitt and Khandker. Reading a similar microfinance evaluation by Joseph Kaboski and Robert Townsend last night made me think of this, but theirs is harder to explain.)

Notice the reasoning here. We assume:

A. Landholding affects household well-being only through microcredit.

That plus the data leads to:

B. Microcredit affects household well-being.

A few comments about this structure:

• Just about all reasoning works this way. You have to assume something to conclude something. Think of Euclid’s classic text on geometry, The Elements, which begins with a handful of axioms, such as that for any two points, a straight line can be drawn to connect them.
• This logical structure is often buried. It is the rare social science abstract that reads, “If you assume A, you can conclude B” even when that is actually what the paper shows. More often, B is highlighted while the dependence on A is deemphasized, even left implicit.
• It is often not clear that we should believe A more easily than B. If I am ready to make one assumption about causality in a society I hardly understand—landholdings only affect well-being through microcredit—why stop there? Why don’t I just assume B—that microcredit raises household well-being on average? It would save me a lot of time. The answer has to be that A is easier to believe than B, just as Euclid’s axioms are easier to believe than what Euclid proves with them, such as the Pythagorean Theorem (a²+b²=c²).

And there’s the rub. Implicitly or otherwise, social scientists evaluating microcredit chide the “believers”—those who believe fervently that microcredit helps the poor. You can’t just make a leap of faith, the scientists say. You have to check the evidence. It turns out that social scientists take their own leaps of faith. It is not obvious to me that the assumptions they make are in general less heroic than those of the “believers” (with an exception I’ll come to). And, unfortunately, social scientists often effectively hide their leaps of credulity from non-experts in clouds of jargon such as “exclusion restrictions” and “overidentification test,” creating a false appearance of objectivity.

Given the complexities of social systems and the resulting difficulty of proving causation through correlation, scientists may have no choice but to reason on substantial faith. But then how different are they from the practitioners who do microfinance every day, who also must forge ahead with little strong evidence on the effects of their work, and who also probably allow their own biases to creep into how they interpret that evidence? The scientists differ only to the extent that the assumptions they make are more clearly true than they assumptions they test. And that extent seems rarely examined.

But, you ask (if you are a good student), “What about RCTs?” Randomized controlled trials of microcredit demand a leap of faith too, but it is a faith easily practiced. In RCTs, the instruments are the random numbers generated by a computer, which determine who is offered credit. They take the place of landholding in the diagram above. So RCTs ask us to believe:

A. The random number generator affects household welfare only through microcredit.

This is as easy to buy as Euclid’s axioms, which is what makes RCTs powerful. And it illustrates Roodman’s Law on the Instrumental Value of Instrumental Variables. Studies have “instrumental value”—they serve greater ends—when the assumptions on which they rest are easier to believe than the assumptions that they test.

You can follow any responses to this entry through the RSS 2.0 feed. You can leave a response, or trackback from your own site.

1. Steve Says:
   April 27th, 2009 at 1:40 am

   I’m a big fan of RCTs, but I do have one concern which is that sometimes its hard to guess the extent some of the program can be replicated effectively. When an NGO runs a tutoring program and its proven successful its not that clear to me that the tutoring program wasn’t particularly well managed or benefited from an NGO good standing in a community (which leads to the external validity issue). And even if the programs scale, what about the general equilibrium effects–how big will they be? I think that’s not relevant for some projects in education and health, but for micro-credit and programs for agricultural productivity especially it might be a huge issue for isolated villages.

   The example that jumps to mind is Partners in Health which has had great success from what I’ve read everywhere it has worked, and they are trying to build a lot of information about the “PIH model” for use by other elsewhere. But I haven’t heard a lot about the success of replicating that model.

   Anyway, as I said, I’m a big fan. RCTs have resolved once and for all at least one big outstanding question of practical importance (giving away bed nets or not?) and have provided a lot of theoretical nuggets about agricultural investment, demand/supply for health care, demand for education, etc.

We value frank and constructive exchanges and encourage you to use your real name in your comments.