I recently attended a seminar with the fascinating Seth Godin and heard an interesting anecdote about VisionSpring:

I could see that every single person who came to this meeting had enough money [$3] … to buy a pair of new reading glasses. And I could tell from how old they were that they were qualified and I knew what they did for a living so I knew that this would pay for itself in two weeks, three weeks, certainly in 20 years it’s going have a huge return on investment. So they get the demonstration, they take the eye test, they see that they need glasses, they take the sample glasses off, they walk over to the table where there are 8 kinds of glasses to choose from … all carefully wrapped in plastic … and 40% of the people bought a pair of glasses. 60% of the people left. And I’ve thought about this about a million times … If they knew how great the glasses would be, if they could overcome the momentum they had and the desire to keep the money … there’s no question they would have bought a pair of glasses. Maybe for $6 or $9 or $12, they could afford it, they needed it, they were in the right place at the right time, and yet the transfer didn’t occur.

To Mr. Godin, it seemed obvious that the customers should be buying glasses, and that they were being held back by “momentum.” He proposed a change:

Instead of giving the person the eye test, taking off the glasses and having them go over … and now make the decision do you care enough about yourself to buy those, instead, give the guy the eye test, and say those are your glasses, you owe me $3. Now the person has to make a new decision, which is better, giving up the $3 and keeping on what I have or going to the trouble of taking them off and reminding myself that I don’t deserve to see? And when you do that it turns out that the close rate goes up 30%.

To be sure, it sounds like an impressive improvement for a simple change. And yet a 30% improvement on a 40% close rate still leaves about half the people not spending the glasses - now under circumstances that arguably make it pretty difficult to turn them down.

Is it possible that Mr. Godin – and the charity he was observing – have simply overestimated the demand for what they’re doing? Underestimated the extent to which imperfect substitutes may be available? It appears that Village Phone did exactly this in at least one case.

It’s easy to be sure that your product is great and that it’s needed. Yet if you’re wrong, and you have donors subsidize it, you may essentially be giving out cash in a less efficient, less empowering way.

Mr. Godin analogized the situation to fundraising:

You’re sitting in class and the person next to you … is coughing … and you take out a container of Fisherman’s Friend menthol lozenges and you offer it to her and she puts it in her mouth and the coughing goes away and everyone is happy … and yet you’ve been on those calls to raise money from a donor, who apparently has money to donate … and you hand them the equivalent of a Fisherman’s Friend, “look at this program, we’ve been working really hard on it, it’s really important, we need your money” and they say “I need to get back to you” and you know what that means …

To me, there’s a major problem with the assumption that a charity’s program is at effective at solving complex social problems as a Fisherman’s Friend is at soothing throats.

Confidence in your product is great, but it can be misplaced. When charities let assumptions like “Everyone here needs glasses” and “Our program is reliable as a cough drop” go unexamined, they’re going to be left scratching their heads at results-oriented donors.

Here’s what we know about the Gates Foundation’s agriculture program:

• Gates believes it’s suggestive that “Apart from a few states and small, oil-rich countries, no country has managed a rapid rise from poverty without increasing agricultural productivity. In the poorest countries, agriculture employs a majority of the people.”
• This isn’t a new argument or an undisputed one. See Peter Timmer on Green Revolution “optimists” vs. “pessimists”.
• Gates’s approach is “comprehensive,” targets “no single, simple solution”, and includes farmer training/support, irrigation initiatives, market access initiatives, and funding of agricultural research with a focus on gender empowerment.
• This isn’t a new approach or a historically successful one. The World Bank has focused on essentially the same set of interventions recently, with unclear results, and the previous “holistic” approach of “Integrated Rural Development” is widely considered to have failed. Details at our overview of agriculture aid.

In other words, the Gates Foundation approach – as described – appears to be neither a continuation of things that have worked before nor a fundamentally new approach to the problem. So what might be different this time around?

Lots of things. Better technology could make all the difference; so could a greater degree of commitment. And one way in which the Gates Foundation could really distinguish itself from past efforts would be by doing a superior job learning about what works and what doesn’t – past initiatives have suffered from poor evaluation and very little accessible information about how things have really worked out.

The Gates Foundation’s progress reports so far are extremely preliminary, looking at “inputs” such as “number of farmers organized into groups”. We find these measures wholly appropriate given how early it is in the initiative; and yet, we’ve seen too many programs that still haven’t moved beyond these measures even after claiming “success” and asking individuals to donate and help them scale up.

If the Gates Foundation moves to more rigorous and outcome-focused evaluation over time, we might learn more about what works and what doesn’t, and the “impatient optimism” could turn out to be justified. If not, the Gates Foundation will be making a very expensive gamble with very little information about its odds.

The Gates Foundation states that “Funders have sharply cut their international aid to agricultural development over the past few decades.” It is implied that this is a major reason for the failure to see a “Green Revolution in Africa.”

We have been unable to locate support for this claim.

Using data from OECD - the most reliable source of official aid flows as far as we know - we graphed the proportion of (disbursed) aid to Africa that was classified in the “agriculture” sector since the 1970s. It’s possible that the numbers are artificially depressed for early years due to less standardized reporting, but we see no trend of the type the Gates Foundation describes.

In addition, we previously looked at funding on some of the main vehicles credited with the original “Green Revolution” and found no substantial drop since the 1970s.

It’s frustrating that the Gates Foundation doesn’t provide a source for its claim. In general, the material on its website is at a very broad level and does not make it possible for people curious about its underlying reasoning to drill deeper. That means that any criticism or examination of its work has to be either very superficial or done offline (i.e., in direct communication with the Foundation, an opportunity few people seem to get easily or often).

Added 10/30/09: here’s our source data

Ever since I heard about the Grameen Foundation’s Village Phone program, I’ve been optimistic. The program involves helping people in remote villages run pay-for-use cellphone services: they get their cellphone, and a loan to buy it, via Grameen, then charge other villagers to use it. It’s an approach to fighting poverty that’s (a) relatively new; (b) using a product that hasn’t been available for a long time but seems clearly useful to anyone doing business in remote areas; (c) utilizing a “franchise model” where people in the villages take a stake in the product.

It was near the top of my “Probably helping people, even though we don’t yet have systematic evaluation yet” list. Now Chris Blattman points to a discouraging evaluation that found “absolutely no impact of the phones on trading activity or availability of goods in local markets” and very small (non-significant) impacts on profits and measures of well-being (school enrollment, consumption of meat, etc.)

This bottom-line result does not, by itself, mean the program “doesn’t work.” It could work very differently in different contexts (discussed below), and there are some possible issues with the paper (which is very recent, and is not a randomized controlled trial). But one thing I like about the study is that it doesn’t just discuss impact - it examines many aspects of the program, and exposes assumptions that may otherwise have gone unquestioned.

Assumption 1: the phones are in high demand and operators easily cover their costs. In fact, usage of the phones was around 4 hours a month, or 8 minutes a day (pg 19). As a result, profits from the phones were not enough to keep up with loan payments (pg 19). Grameen reportedly has responded by changing loan and franchise terms (pg 30). Tuvugane (pg 5), a less sophisticated phone product that was already common in the villages, may have been good enough for most purposes.

Assumption 2: farmers who use the phones benefit from better pricing power. Even though farmers with access to the phones became much more likely to arrange their own transport to market, there was no apparent effect on the prices they received for their goods, possibly due to established relationships with buyers (pg 16).

Assumption 3: if someone chooses to become a cellphone operator, they’re going to benefit from it. In fact, there was a very strange pattern in the businesses of people who became phone operators. Their hours worked rose significantly both for their new phone business and for their already-existing businesses, but their profits and wages paid did not rise (pgs 17-18). A possible explanation is that operators wanted to be available for cellphone users and so stayed at their workplaces longer, but that the extra hours didn’t translate into extra profits. In any case, it’s a pattern that doesn’t seem encouraging, and seems to deserve further investigation.

Bottom line: a product that was supposed to be helpful and in high demand arguably ended up as a bad investment for the franchise operators. This doesn’t mean it shouldn’t have been tried, or that it shouldn’t be tried in the future. But it points to the importance of testing assumptions empirically, rather than scaling up a program as widely as possible based on an appealing story.

Imagine that a charity is able to teach a farmer some basic, useful things about farming (like “crop rotation, dip irrigation and the planting of trees that enrich over worked soil” or “disease-resistant cassava replication, distribution and sale; crop diversification; soil conservation; and expanding market opportunities”). Such simple knowledge could last the farmer forever and be far more useful - especially for the cost - than cash or loans. It’s an often-sold story, and an appealing one.

What charities don’t tell you about “improved farming techniques and technology” is just how long the aid world has been trying to spread them, and how much it has struggled. The basic challenges:

Can agriculture programs reach enough farmers? The right farmers?

A 2006 World Bank paper examines the long history of “agricultural extension” programs and is frank about their problems. For traditional programs, it states that

The cost of reaching large, geographically dispersed and remote smallholder farmers is high, particularly given high levels of illiteracy, limited access to mass media, and high transport costs. Farming systems often entail several crops, livestock, and even within given geographical area, there are variations in soil, elevation, microclimate and farmers’ capabilities and access to resources. With such a large and diversified clientele, only a small fraction of farmers can be served directly (face-to-face) by extension, and agents tend to focus on the larger, better resourced and more innovative farmers. This reduces the potential for farmer-to-farmer diffusion. (Emphasis ours)

The “Training & Visit” model attempted to address these issues through a strong, clear set of hierarchies and responsibilities (see pgs 11-14), but its substantially higher costs - coupled with the fact that, as with previous programs, impact was hard to see - led to its essentially universal abandonment (see pgs 14-15 and pgs 22-23).

When World Vision or Save the Children speaks of spreading improved practices, is it using a “T&V” style intensive-but-costly approach, or a lighter touch that could fail to reach enough (and the right) farmers? It isn’t clear.

Do charities even know what to teach and what to change?

Another general problem cited by the World Bank paper is that “Weak accountability (linked to the inability to attribute impact) is reflected in low-quality and repetitive advice given to farmers, and in diminished effort to interact with farmers, and to learn from their experience.” (Emphasis ours.) In other words, those giving advice may not actually be giving the right advice.

It is hard to find honest and thorough descriptions of how such projects have actually played out in the past, but a couple of striking failure stories should make it clear just how badly outsiders can misjudge what farmers need to learn:

• The DrumNet program in Kenya aimed, successfully, to transition farmers from growing “local crops” (i.e., crops for local/personal consumption) to growing “export crops” (i.e., crops to be sold on the export market). However, a year after the project evaluation was completed, the firm that had been buying the “export crops” stopped due to European regulations, leading to “the collapse of Drumnet as farmers were forced to undersell to middlemen, leaving sometimes a harvest of unsellable crops and thus defaulting on their loans.” (Details at this paper published on the Poverty Action Lab site (PDF).)

• A development program in Lesotho aimed to help local people with crop and livestock management, as well as building roads so they could access markets. However, few of the people in the region were farmers, and conditions were not good for farming. Harsh weather destroyed pilot crop projects, and the roads allowed in competitors who drove the existing local farmers out of business. (From pgs 193-4 of White Man’s Burden)

These aren’t cases of minor missteps - they’re cases where those giving aid did not perceive essential and fundamental aspects of the local economy. That doesn’t mean they were incompetent - it means that understanding a local economy well enough to give truly useful advice may not be easy.

The long and murky history of agricultural assistance

Agricultural programs in Africa have struggled to produce tangible results, both at the micro level (little evidence about how programs have gone) and at the macro level (disappointing progress in Africa-wide crop yields over time).

A variety of approaches have been tried, including the “holistic” approach of simultaneously addressing health, transportation, credit, and agricultural knowledge. This approach was referred to as “Integrated Rural Development” in the 1970s and 1980s and appears to be acknowledged as a failure, although the basic idea behind it may be making a comeback in the “holistic” approach of the Millennium Villages Project and other large charities.

Details at our writeup on agriculture-focused aid.

Bottom line for donors: agricultural technology is not like medicine

Agriculture aid is often presented as a matter of extending the reach of proven technologies and methods. However, the track record of such programs is simply nothing like that of health programs, which often have track records including multiple highly rigorous studies and large-scale, demonstrable successes.

We feel that the burden of proof on agriculture programs is high, but outcomes tracking of any kind is extremely rare. The evaluations that are available tend to raise many concerns about whether results are “cherry-picked” and whether results even point to improved lives.

We recommend that donors be extremely wary of charities working heavily in this area, no matter how good their intentions. We have not identified any that we can have confidence in.

Is microfinance a good bet for a donor? We feel the answer is complicated, and that the many extreme exaggerations of microfinance’s impact get in the way of making an informed decision.

This post summarizes the differences between the stories you’ve probably heard and the reality according to available evidence.

Myth #1: the way microfinance charities help is by giving people loans to expand businesses. Success stories like Andrea’s, Lucas’s and Sophia’s are representative.

Reality: there isn’t much reliable information on how people are using loans, but the evidence there is suggests that “microloans” are often used for consumption purposes: food, visits to the doctor, etc. This isn’t a bad thing - the poorest people in the world face considerable financial uncertainty, and loans can empower them to manage their own lives.

So, however, can savings, which some scholars feel are more beneficial for the poor than loans. Funding institutions to help people save may not have the same sex appeal as “lending your money to help people grow their businesses,” but it might do more good.

For more, see:

• “Will the real microlending please stand up?” - a summary of this idea, 2 years old (more evidence has become available since)
• Portfolios of the Poor, a study on how extremely low-income people use financial services (formal and informal)
• Post on evidence of impact for microfinance - most encouraging effects came from programs not focused on business investment
• Conversation with David Roodman about microfinance
• More information on how loans are used

Myth #2 The best way to support microfinance is to lend your money to specific individuals.
Reality: Choosing your own borrowers is not really possible or desirable. The recent debate over Kiva.org (summarized by GiveWell Board member Tim Ogden) makes clear that even when your donation is “officially” matched to a borrower, you’re really funding an institution. And as we discuss immediately below, this is likely a good thing.

Myth #3: a gift to a microfinance charity gets lent out again and again, making its impact essentially infinite.
Reality: Many of the most important challenges of microfinance (such as developing effective outreach, creating incentives for repayment, and helping people to save as well as borrow) involve significant institutional expenses. (See our discussion with David Roodman as well as any microfinance charity’s budget.) Update 5/2010: also see our rough estimate of the overall “cost-effectiveness” of microfinance, concluding that it is hard to argue that microfinance donations in general are more cost-effective than donations to top health programs.

Myth #4: microfinance has been shown to reduce poverty.
Reality: many studies on the impact of microfinance have been done, but most have serious and widely recognized flaws. The few - and recent - stronger studies show mixed effects. The most encouraging effects are for programs that don’t fit the traditional “lend to expand a business” story.

Details at our post on evidence of impact for microfinance charities.

Myth #5: a high repayment rate means that things are going well and clients are benefiting from loans.
Reality: the repayment rate can be both technically and conceptually misleading. See our post on why the repayment rate may not mean what you think it means.

Myth #6: microfinance works because of (a) the innovative “group lending” method; (b) targeting of women, who use loans more productively than men; (c) targeting of the poorest of the poor, who benefit most from loans.
Reality: all three of these claims are often repeated but (as far as we can tell) never backed up. The strongest available evidence is limited, but undermines all three claims.

Bottom line: should you give to a microfinance charity?

We feel that the marketing of microfinance is exaggerated, excessive, and full of unsupported myths - to a degree unusual even in the world of fundraising.

Once you put these myths out of your head, the fact remains that microfinance institutions are often working with people in extreme poverty and empowering them to better manage their own financial lives. The fact remains that high numbers of clients for a product that costs clients money (interest) - while not necessarily demonstrating positive impact - suggest that MFIs are offering something clients want. All in all, this is more than most charitable causes can say for themselves.

We feel that global health is a better area for a donor overall, especially because we have identified outstanding charities in global health that have far more to recommend them than any microfinance charity we’ve seen to date. We continue to search for an outstanding microfinance charity (through methods including our ongoing grant application). Make sure you’re signed up for updates (or following our blog or Twitter) and you’ll know if and when we find one.

I was glad to read of a new $110 million initiative for insecticide-treated bednet distribution, which we find one of the better-established ways to spend money to improve lives.

But what does this mean for you if you’ve been giving to a malaria charity? Do independent bednet distributions now run the risk of being redundant with the new one? Has USAID provided enough funding that your donation is no longer as needed?

Unfortunately, we have no way of answering this question. While there are some attempts to coordinate government aid, we know of no one asking questions like “How much total room is there for funding distribution of bednets? How can we make sure that all the malaria organizations are on the same page? How can we track the extent to which individual donations are still needed?”

If donors focused on how to have real impact (as opposed to, say, fictions about where “their” money goes), such a question would be extremely important to them.

What’s wrong with this “evidence of impact” for high-profile charities?

• The Millennium Villages project’s main evidence of impact so far appears to be increasing crop yields by a factor of 4-15x in 3 villages. (H/T to the recent coverage of Millennium Villages on Aid Watch)
• A KickStart evaluation (available on our website (DOC)) reports that “Family/household incomes of 3,000 farmers increased by 100% within 24 months following acquisition of micro-irrigation pumps.”
• The Carter Center’s agriculture program “provides the tools and means to help farmers in sub-Saharan Africa increase their crop yields through agricultural development, sometimes two- or even threefold.”

Among other possible problems, two major issues jump out:

1. No context on what “normal” variation in incomes looks like for poor farmers. Some years have more favorable weather - and local economic situations - than others. Enough that one year’s income or crop yield could be double another’s? 4x? 20x?

Unfortunately, one of the better pieces of “evidence” that jumps to mind is a 75-year-old novel, The Good Earth, whose farmer protagonist is comfortable one year and has literally zero income the next, for no other reason than the weather. If a given year’s yield were close enough to zero, the next year could be a huge increase (2x, 4x, 20x or more) simply by returning to normal.

I have seen little information on the local year-to-year volatility that poor farmers can experience, but I imagine that it (a) varies greatly from region to region and (b) could easily involve incomes falling and jumping by enormous amounts.

None of the above reports provide any context on this question, beyond qualitative statements about how favorable the rains were in each year examined. None of them employ any sort of “comparison group” of farmers (aside from one vague reference to “farms not using improved seeds and fertilizers” in the Malawi Millennium Village). Ultimately, none accomplish one of the most basic goals of an evaluation: giving a sense of how likely the “gains” they describe are to have arisen by pure chance.

With larger sample sizes, we might be able to use country-level volatility for context. But that brings me to the next problem.

2. We have no assurance that the described gains are representative, as opposed to “cherry-picked.”

All of the above organizations have reputations for consistent and thorough monitoring and evaluation, yet in all cases, we find ourselves looking for “impact” from a tiny subset of their projects.

• The Millennium Villages evaluation “includes results from the Sauri baseline survey, annual reports from Sauri and Koraro and crop-harvest data from Malawi.” How were these three chosen from among the 78 villages mentioned in the abstract?* The report doesn’t say.
• KickStart implies that it is constantly and consistently tracking impact, yet the report linked above discusses a single project over a single year.
• The Carter Center is somewhat explicit that its number is a “high-end” one (”sometimes two- or even threefold”). Examining the most recent annual report for its agriculture program shows a very small upward trend, if any, in crop yields across the countries in which it is working.

Some ways to produce more compelling evidence of impact

1. Be clear about what is being measured and what is being published, and when. It seems to us that in this area, charity evaluation lags far behind clinical trials, which are constantly registered before they are complete so people can track their progress. (The Poverty Action Lab is similarly transparent with its own ongoing projects.)
2. More sample size; more context; use of comparison groups. Discussed above.
3. Look for more sustained improvements in people’s lives. One measure I find superior to straight “income” or “crop yields” is asset accumulation. A jump in income could be temporary; if someone upgrades their roof or sanitation, it’s likely that at least they expect the gain to be a real and lasting one. The Village Enterprise Fund’s evaluation is one of the better charity evaluations I’ve seen in the area of economic empowerment, partly because it focuses on standard of living rather than a simple measure of income.

*It’s possible that the yields mentioned are for “clusters” of villages rather than individual villages; there are only 12 clusters. However, the source documents available for Sauri and Koraro appear to be at the village rather than the cluster level, and the details of how the measurements were made are unclear.

In a recent debate, David Hunter’s article on the nonprofit sector has taken heat for its assertion that “While nonprofits work incredibly hard, with passion and dedication, and often in incredibly difficult circumstances to solve society’s most intractable problems, there is virtually no credible evidence that most nonprofit organizations actually produce any social value.”

We agree with the claim for the sectors we’ve examined, which we believe are similar to the sectors Mr. Hunter has examined: particularly thorny areas such as charities working to improve education and international charities addressing extreme poverty overseas. These are problems on which experts have struggled for decades to make any progress, and while we don’t necessarily agree that most charities are failing to produce value, we agree that most charities cannot produce any credible evidence that they are. This is different from the claim that Sean Stannard-Stockton attributes to Mr. Hunter (”most nonprofits and the social sector as a whole is not currently producing social value”), but it still means that it’s very hard for a donor to give with confidence.

The information we have

Our belief is based on two years of looking for this evidence; we’ve published the full details of our findings online, and you can see our summary of international charities (only 19 out of 320 examined publish any impact-related evaluation reports) and U.S. equality of opportunity charities (only 6 of 83 examined provide credible impact-related reports, and 2 of these show negative or no impact).

In addition, in a guest post on the GiveWell Blog, David Anderson of the Coalition for Evidence-Based Policy estimates that 75% of rigorous evaluations show weak effects, no effects, or negative effects.

More information needed

On the other hand, we also believe the criticism that Mr. Hunter doesn’t support his own claim with evidence has merit. We would like more clarity on which sectors Mr. Hunter has examined and is referring to, and information on where he has looked for evidence and what he has found.

In addition, we feel that examples of failing/harmful programs, such as “Well intentioned but ineptly run mentoring programs where failed matches reinforce in youngsters a sense of their low worth and poor prospects” (and the other items on the list on page 2 of the article) should be clearly referenced to summaries of evidence.

The truth is that we cannot have a very informed debate about how much value nonprofits create because we have so little evidence of any kind. Some people adamantly believe that nonprofits create enormous value; others are skeptical that they create any; and there is very little to go on, at least in the sectors under discussion.

Nonprofits that do have credible evidence of their social impact

The good news for donors is that they need not be in the dark if they give to the right charity. Our top-rated charities do produce credible evidence of their social impact. We encourage individual donors to expand and fund these charities until and unless others follow suit.

Advertising is based on one thing: Happiness. And you know what happiness is? Happiness is the smell of a new car… It’s freedom from fear. It’s a billboard on the side of the road that screams with reassurance that whatever you’re doing is okay. You are okay. — Don Draper on AMC’s Mad Men. (Video on YouTube.)

I sometimes wonder how Don Draper would react to someone telling him that in 2009, many consumers make their buying decisions based on a product’s quality rather than the feeling a product’s advertising instills in them, or the narrative in which the ad places them.

I picture Don saying, “User reviews? Restaurant ratings? Product testing? No way. That’s just not how consumers are hardwired to think.”

When I see Sasha Dichter and Nathaniel Whittemore saying the same thing, I wonder whether they’re right that the problem is donors as opposed to the information donors have access to.

Sure, it’s possible that donors are hardwired a certain way and may never care about the real impact their gift has. But it’s also possible that many donors would look past happiness and use information on impact - if only it were easier to find. That’s what we’re trying to bring about.

Why has Kiva been singled out for so much criticism lately (see GiveWell Board member Tim Ogden’s summary)?

Part of the answer is that Kiva has arguably been misleading donors - but that can’t be the full answer. David Roodman’s original post could never have come about without the fact, as Mr. Roodman puts it, that “the way Kiva actually works is hidden in plain sight.” And our followup analysis on repayment rates was only possible because Kiva makes all its repayment data publicly and easily available. As Elie said in that post, “Similar analysis would be impossible for nearly every other charity I can think of.”

Contrast Kiva with, for example, UNICEF. Kiva makes it possible to trace the path of your donation, to the extent that such tracing is realistic (and it largely turns out to be more along the lines of “you funded a certain MFI” rather than “you funded a certain person”). UNICEF doesn’t even seem to have a breakdown of how much money is going to each continent. We definitely can’t find information on questions like (a) What specific projects are you funding? (b) What is your role in each? (c) What new projects are planned, and where? (d) How is each project going, whom is it affecting, and how?

There are no strange patterns in UNICEF’s numbers because there are no numbers. There are no contradictions because there is no concrete information. And the intent here isn’t to single out UNICEF - it’s merely one of the vast majority of international aid organizations about which we know essentially nothing.

Giving an impression to donors that’s undermined by the facts is a minor scandal. When will complete opacity - simply sharing no information at all - be a major one?

This Tactical Philanthropy post implies that the salient difference between DonorsChoose and Kiva is that DonorsChoose is more “authentic” in terms of connecting donors to projects. (Update: Sean points out in the comment below that most of the post I cited was a quote from a former DonorsChoose employee and doesn’t necessarily reflect TacticalPhilanthropy’s opinion.)

We think there’s another important difference. Kiva and DonorsChoose work on very different types of problems, that offer donors vastly different opportunities to cause significant impact. And if we had to pick one, we’d bet on Kiva as the better option if you want to change lives.

The problem of improving children’s education is far, far more difficult than most people suspect - for our most recent coverage, see our multi-part series on the recent Harlem Children’s Zone study, and how excited the academic community has been over a one-time and modest improvement. And DonorsChoose does not offer the chance to support any of the most promising and tested educational interventions, such as the creation of intensive charter schools. Funding DonorsChoose is making a bet that classroom materials are where it’s at, even when the teachers, school systems and children’s outside environments remain the same. As far as we know, there is zero evidence that improved school supplies have resulted in any measurable improvements in the U.S., and even in the developing world the proposition seems iffy.

Funding Kiva, even if it doesn’t mean funding individuals, means funding microfinance charities. We are very ambivalent about microfinance; recent posts on this topic include questions about whether there’s any empirical support for microfinance charities’ claims and an interview with David Roodman about microfinance charity in which many unresolved questions come up. But the bottom line is that when you fund microfinance charities, your money ends up in the poorest parts of the world, and often (though we don’t know how often or how much) helps people there manage their volatile financial lives with credit, savings and other assistance.

(In fact, the way in which your money is most helpful may be by doing exactly what it has been criticized for doing - effectively funding partners’ other projects, such as savings vehicles, rather than the loans you see on your screen.)

Fund financial services in the poorest parts of the world, or fund an untested and low-intensity approach to a problem that higher-intensity programs have struggled with for decades. $900 can be five digital cameras for a classroom or a year’s income for 3 people.

That’s the decision as it looks to us, as we ask not “How can I make sure that I can see the exact person who’s getting my money?” but “How can my money accomplish as much good as possible?”

And on that note, we feel that our top-rated charities offer far better opportunities to improve lives than either, even if they can’t deliver the same emotional experience. As a donor, which would you rather have?

GiveWell Board member Tim Ogden asks, is Kiva reporting phantom payments?

It’s hard to say for sure, but it seems that the answers is either (a) yes, they are or (b) no, but borrowers listed on Kiva’s website are not representative of the average borrowers visiting that MFI.

The table below shows default rates for Kiva’s largest field partners (in terms of total loans provided through Kiva) and compares the default rates reported on Kiva to default rates reported for those same institutions on MixMarket, a site devoted to sharing data on microfinance.

Data from the two sources aren’t perfectly compatible but the broad story is clear: Kiva’s partners have higher repayment rates on their Kiva-listed loans than on their portfolio overall, as listed on MixMarket. This could come about in a couple ways:

• The borrowers that partners list on Kiva could be substantially more likely to repay than “other” borrowers. If Kiva loans are effectively fungible between partners (as the recent discussions imply), this would mean that donors are likely “effectively” funding clients who are less likely to repay than those in the profiles they’re scanning.
• Kiva’s partners could be “covering for” their clients to keep their Kiva-listed repayment rates high.

(Kiva data comes from a superb site, Kiva Data, a site stating that it is unaffiliated with Kiva as well as the partners’ pages on the Kiva website itself. MixMarket data comes from the MixMarket website. For definitions of items in the table, see the notes section below it.

It’s worth noting that this analysis is only possible because Kiva makes its data available and due to the efforts of those at Kiva Data and MixMarket who have created extremely useful information resources. Similar analysis would be impossible for nearly every other charity I can think of.)

Notes on the data:

• The data in the table above comes from Kiva’s 10 largest partners (in terms of number of loans made to date) who provided full MixMarket data for 2008. 2 of Kiva’s 10 largest partners doesn’t have this data accessible at the moment. AMK, Kiva’s 7th largest partner, reported no outstanding loans for 2008 on MixMarket and ADMIC, Kiva’s 9th largest partner, has not reported any data for 2008 yet.
• p refers to the probability that, if the MFI’s Kiva borrowers were equally likely to default as its borrowers in general, the MFI would happen to have a 0% default rate for its Kiva loans. (Note that this calculation assumes that the time periods and the two definitions of “default rate” are comparable.) Low values mean it is extremely unlikely that this phenomenon would arise by chance, implying instead that the Kiva borrowers are systematically better than the normal borrowers, or that defaults are simply not being reported to/by Kiva.
• The data on MixMarket is self-reported. Nevertheless, Mix is a respected source, and we think it’s reasonable to rely on it.
• The date-ranges used for the Kiva data and Mix data above don’t sync up perfectly. Mix reports data for each calendar year. Kiva reports all-time data for each of its partners, some of whom joined in the middle of a calendar year. We’ve roughly adjusted for this by ignoring a Kiva partner’s first partial year and weighting the Mix data for the remaining years by the number of loans outstanding each year.
• MixMarket doesn’t use the term “default rate.” Instead, we’ve used “Write-off ratio %,” defined as Write Offs / Loan Portfolio, gross, average. We’d prefer to also look at Portfolio at Risk figures. However, Kiva only reports delinquencies for loans currently outstanding and therefore can’t be matched well to Mix data.
• MixMarket reports data as a % of the portfolio’s dollar value. Kiva reports data as a % of loans made and value loaned. Given that Kiva reports 0% default rates and Mix reports >0% default rates, there’s clearly some inconsistency. Nevertheless, without information regarding the size of the loans defaulted on at Kiva, we can’t guess at the magnitude of the inconsistency.
• All data used in this table is available in an Excel file. If you’re interested see Kiva major partners data (XLS). This file also contains links directly to Kiva’s partners page both on Kiva and MixMarket.

Change.org global health blog:

we know how to avoid maternal mortality (it’s 1 in 47,600 in Ireland, but 1 in 7 in Niger), it’s just that “poor, uneducated women in Africa and Asia have never been a priority either in their own countries or to donor nations.” Kristof is demanding action and aid to help save lives, advocating tried and tested methods that just need funding in order to make a real difference.

We agree that maternal mortality is a vital problem, but we don’t agree that there are “tried and tested methods that just need funding in order to make a real difference.” Our review of maternal mortality programs concludes that even though the developed world has made great progress in making childbirth safer, there have been no consistently successful programs for extending these gains to the developing world, and some prominent failures.

The observed difference between Ireland and Niger (in the quote above) indicates that there is a problem, but does not indicate that we know how to solve it. After all, Ireland has over 50x the per-person income of Niger (based on 3 data sets collected by Wikipedia), yet there is no consensus on how to promote growth and reduce poverty overall.

We don’t have the same pessimism about all causes. There are many health programs that have had repeated success in the past and likely deserve the designation of “tried and tested methods that just need funding in order to make a real difference.”

If donors are to be well-informed and live up to their potential effectiveness, it is vital not just to identify important problems, but to distinguish between areas where funding is the primary constraint to impact and areas where it isn’t.

From the donor-facing website:

From a flyer for partner microfinance institutions (linked by GiveWell Board member Tim Ogden):

Update: responding to criticism, Kiva has updated its donor-facing page.