Print
Share
If Randomized Evaluations Are So Great, Why Don’t Businesses Use Them?
September 8, 2011
This is a joint post with Michael Clemens.
Michael Clemens recently wrote me, saying that he gets asked this question a lot. I do, too. So I was interested when he brought my attention to a 2007 article in Forbes that discusses a number of companies that do use randomized studies. I wasn’t surprised to see Google in the list, but I never imagined that all the junk mail that I receive from Capital One might be guided by sophisticated research (though it hasn’t convinced me to sign up yet!). Progressive Insurance apparently discovered profitable lines of business (middle-aged motorcycle drivers) by randomly accepting a portion of applicants who would normally be rejected and studying their claims behavior. According to Hunt Alcott, other companies that have used randomized studies include H&R Block, PNC Bank, Amazon, Subway and Harrah’s Casino.
Businesses have an advantage in the evaluation arena because they get very effective feedback from sales data – but they still have the basic problem of comparing their performance against an appropriate counterfactual. If ice cream sales go up after a change in strategy was it due to the new approach or to an especially hot summer? And the key to answering these questions usefully means thinking in terms of testing assumptions with information that can prove your hypotheses wrong as well as confirm when you’re right. This kind of “testing mind-set,” as argued by Thomas Davenport in the Harvard Business Review (2009), is used by companies all the time. A lot of Davenport’s points could apply equally well to improving the way aid agencies or public policies are assessed and improved.
So if companies are doing so much sophisticated evaluation work, why don’t we hear about it more often? As Michael pointed out, they have incentives to keep what they learn secret. Companies that have found a new profitable niche or effective marketing strategy don’t want to share the news – which is why you’ll never know when the iPhone 5 is coming out even if your best friend works at Apple. This is where public policy really diverges from the for-profit world. Companies are accountable to their shareholders; governments and non-profits are supposed to be accountable to the public through transparency and by widely disseminating knowledge from good evaluations.
Possibly Related Posts
8 Responses to “If Randomized Evaluations Are So Great, Why Don’t Businesses Use Them?”
Post a Comment
We value frank and constructive exchanges and encourage you to use your real name in your comments.
September 8th, 2011 at 1:24 pm
In the online sphere randomized (or a/b) testing is used all the time. It’s used constantly to optimize websites. I used to work at a firm that helped nonprofits run online fundraising programs, and we did randomized testing all the time, especially on large-scale email fundraising blasts that would go to hundreds of thousands of people. We were also conscious of the need to test things that were generalizable (ie, test the same ‘intervention,’ such as making text on links a different color, etc, across multiple clients at different times of the year before concluding that it was always beneficial). We would share the results with clients to help them improve (that’s what they paid us for after all) but no single clients saw results from everyone, because that insider knowledge is what made our services valuable. We did publish ‘white papers’ touting our research when we found something generalizable — and the publication of those results helped get new clients.
However, I’m not sure this sort of a/b testing is really that great an analogy for randomized controlled trials in economic/health/development research. The Progressive Insurance example you mention above is closer.
September 8th, 2011 at 9:52 pm
bill, very interesting, especially from the author of “organization matters”! the specific value of randomized controls are, i think, clear. however, we need always remind ourselves of a few things. 1) they are not double blind. 2) reforms that seek over a long period of time to change fundamentally the nature of service provision and touch upon nearly every aspect of a system are difficult if not impossible to evaluate in terms of their impact on outcomes. In this sense, these reforms are quite different from many of the accountability reforms that have been analyzed recently using randomized controls, which most often examine a discrete intervention (e.g., cameras in classrooms, informational campaigns, parental involvement in school management, etc). 3) this means that institutional case study approaches are a necessary complement to the recent emphasis on randomized experiments that generally analyze a focused “accountability” intervention. and in the specific case of businesses, i guess one needs to consider that 4) very few businesses need to tackle the kind of system wide, nation-wide changes that social sector reforms need to tackle, nor do many businesses have 20 or 30 year time horizons for success, and this too perhaps minimizes the relevance of whether or not they do randomized evaluations. thanks for the very inspiring info! hope you are well. alec @AlecGershberg
September 8th, 2011 at 10:50 pm
My sense is that private organizations in competitive markets do have better feedback loops, but I think trials of new products or marketing strategies play a very small role. I think their advantage rather lies in their better, and easier, monitoring of performance. Your ice cream manufacturer doesn’t need to do an RCT – it doesn’t need to have a plausible counterfactual. If sales, profits and market share are going up – then their strategy is “working”. And successful companies monitor these indicators very closely.
From what I’ve seen, the weakest links in development program feedback loops are the monitoring systems and then the lack of pressure to improve performance (in that order). And RCTs, such as they are, do little or nothing to strengthen the feedback loops.
September 8th, 2011 at 10:55 pm
There was a good Harvard Business Review article earlier this year about the use of experiments by firms for marketing. I blogged about it here, noting the lessons for development:
http://blogs.worldbank.org/imp.....t-research
September 10th, 2011 at 2:09 am
I would support April on this: in most cases, effects are so stark, that the owner sees them and just applies it.
However, as benefit and loss are marginal compared to the total business volume, smaller changes that are not obvious could be important for the bottom line. However, as RCT is expensive to do (mostly because you need scientists) I guess the complexity is just too big for a lot of companies.
September 10th, 2011 at 8:38 am
As someone who’s seen RCTs both in the research and corporate fields, I was quite surprised to see the enthusiasm in academia about it. This was because in the bank where I worked before this was our bread-and-butter job as a business-intelligence analyst. Most organizations that work with a large number of customers are bound to have some kind of randomized test going on, as a part of their regular business process. I disagree with the previous comments about there being no need to do RCTs in the corporate world. Business managers often have multiple strategies being suggested to them and they are keen to understand which is the one that actually brought about a change so that they can run with it. And RCTs are the best way to learn that.
September 12th, 2011 at 10:01 am
April and Sam’s comments reminded me of two things that have a bearing on this – in particular showing that the private sector has less useful information than is commonly believed.
First, when I was working at the Inter-American Development Bank (IDB) in the 1990s, I had a conversation with a college friend who worked for JP Morgan. I complained to him that in the IDB we didn’t have good incentives for developing good loans because public purpose projects didn’t have clear performance measures the way private banks have a clear performance measure in profitability and loss ratios. His answer to me: Don’t be silly. Just because JP Morgan has quarterly financial data doesn’t help them decide who is making good or bad loans. He said the incentives within private banks were equally screwy – with some managers rewarding risk-takers and others rewarding cautious ones independent of the ultimate performance of those loans over the course of the following 5 years at which time the loan officer has already been promoted or fired for other reasons. His point was how weak the link is from all these individual in-house decisions to the department or Bank’s overall profitability. To generalize this anecdote, one need look no further than the popularity of Dilbert cartoons (http://dilbert.com/strips/comic/2011-09-12/).
The second thing your comments brought to mind was a business book I read recently by David Apgar called “Relevance” (http://bit.ly/nSb14W). He points out – with case studies – how companies spend a lot of time and money monitoring data that can’t help them assess whether their strategies are working or not. I agree with April that basic monitoring is deficient in development programs, but a focus on improving monitoring without the mind-set of testing hypotheses about why a project is or isn’t working won’t get us anywhere.
The lesson I take from both of these stories is that the private sector has less of an advantage over public sector programs than we generally maintain. In fact, the private sector can tell when things are going well or poorly but often has no idea why and whether it can/should attribute that performance to any of its own actions. I think the unconscionably high payouts to CEOs for companies that have done no better than average (or even lost money) is another demonstration of that.
Finally, I don’t get the sense that either of you were reacting to the main point in the Forbes article: businesses are indeed using RCTs and this is evidence that sophisticated counterfactual studies are both feasible and valuable. The Thomas Davenport article in the Harvard Business Review also does a nice job explaining exactly *when* and under what conditions RCTs or other kinds of study that use a clear counterfactual are worth the effort.
October 3rd, 2011 at 4:13 pm
Howard White shared a blog that he had written in August on a similar theme (http://bit.ly/rp9Gcm). He also references another book on the topic: Hard Facts, Dangerous Half-Truths, and Total Nonsense: profiting from evidence-based management (http://www.evidence-basedmanagement.com/books/) by Jeffrey Pfeffer and Robert Sutton.