There’s been a minor flurry of recent blog posts about randomized controlled trials (RCTs) in international aid, including William Easterly’s take and responses by Chris Blattman and GiveWell Board member Tim Ogden.

A central theme has been the difficulty of generalizing from one experiment to whether something “works in general.” There seems to be a loose consensus that while RCTs can highlight interesting behavior patterns, they shouldn’t be expected to stamp programs with “effective” or “ineffective” to the extent that a page like this one suggests.

To me, this concern is another example of “two worlds” in philanthropy. Scholars naturally consider it an open question whether a program “works,” and rightfully point out that a single experiment (or even many experiments) can’t provide a general answer; within this context, the cautions about RCTs are good to note. But charities constantly “sell” the logic of a single approach, independent of time and place; within this context, RCTs provide a meaningful and much-needed reality check.

Microfinance is an excellent example. The Grameen Foundation states, “Microfinance helps people to escape poverty by giving them collateral-free loans and other financial services to support income-generating businesses.” Though it works in four continents, its success stories all follow the same basic pattern, in which a loan is used to expand a small business. Messaging from other microfinance organizations is generally similar, leading to what I perceive as a highly unified impression among donors that microfinance fights poverty by enabling small business expansion.

In this context, the argument of a recent paper (PDF) is highly relevant. It asserts (and we have reviewed it only superficially at this point) that, among other things,

• The impact of microloans in this case (an expansion of credit in Manila) was generally to make businesses shrink (“treated businesses shed unproductive employees. One explanation is that increased access to credit reduces the need for favor-trading within family or community networks” – pg 3)
• Loans were more beneficial for “male, and relatively high-income, borrowers” (pg 4)

These claims substantially challenge (a) the story that microfinance institutions have been using to explain and “sell” their operations; (b) the way they have been carrying these operations out; and (c) the assumptions that donors normally bring and the questions they normally ask.

From what we’ve seen, charities don’t proceed with the caution of researchers, trying to fully understand what happened in one area before moving on to the next. They tend to run with a single theory of how an intervention changes people’s lives, scaling up the intervention substantially – often in many countries – with little or no evaluation.

Hence the value of even a single truly rigorous test, in a single time and place, of whether reality was consistent with the theory. We recognize the limitations of RCTs (and have not relied on them much in generating our current recommendations), but we feel strongly that a charity is far more credible when it is testing its claims with RCTs, and that the existence of more RCTs would greatly help donors.