We ask you, as a donor, to turn down some great pitches – “Your interest-free loan will help this person escape poverty forever,” “You can give a cow to a poor family for Christmas,” etc. – and give instead to charities that aren’t terribly good at storytelling. Why?

It comes down to this. We think that most of those stories are just that – stories. (For more, see our summary of recent posts on “big-name” charities, which we feel are representative of the full set of charities we’ve reviewed.) But if you give to one of our top charities, you really can save (or dramatically change) a human life.

It hasn’t been easy to find charities that we can honestly say this about. That’s what our process is built around and where most of our energy goes. This week we’ve blogged about the best we’ve found, VillageReach and Stop Tuberculosis Partnership. There is plenty of room for doubt even with them, but overall we think there is a strong case - even for the skeptic - that your donation to them can save a life.

What do we mean when we say “save a life?”

By “you can save a life,” we don’t mean anything as simple, concrete, or easy to grasp as the stories charities usually tell.

• Your gift can’t literally be linked to an individual. It will help an organization that, all things considered, is achieving a lot of impact for what it spends.
• If you must know what “your” dollars are doing, it’s likely that they’ll be sitting in reserves to ensure financial stability, or enabling a slightly larger travel budget for evaluators, or something similarly unexciting.
• It’s even highly possible that your donations will be wasted, and that the charity you give to – even the best you can find – will fail. We don’t think there are true guarantees in aid.
• Even if these charities are succeeding, it’s very likely that your donation won’t ultimately result in the charity doing anything differently. It’s pretty hard to think about how $1000, by itself, could really change anything about Stop TB Partnership’s plans for next year.
• Yet donations add up. 50-100 of these donations could mean a significantly larger grant, more people getting tuberculosis treatment … and that could mean families staying intact instead of being struck by sudden death.

The truth is that it takes a lot of abstraction and analytical thinking to really think about how your donation saves a life. The life you can save is an “expected” life (”expected” in the sense of probabilistic expected value) - it isn’t a person we can point to or show you a picture of. More than with typical charities, you have to use your imagination. But more than with typical charities, your impact is real.

With opportunity comes responsibility

In The Life You Can Save (which prominently features GiveWell and which we have reviewed), Peter Singer writes:

By donating a relatively small amount of money, you could save a child’s life … we all spend money on things we don’t really need, whether on drinks, meals out, clothing, movies, concerts, vacations, new cars, or house renovation. Is it possible that by choosing to spend your money on such things rather than contributing to an aid agency, you are leaving a child to die, a child you could have saved? (pg 5)

Our corollary: is it possible that you are leaving a child to die when you choose to donate to a charity with a “feel-good” story rather than a charity with a great case for real impact?

It is true that, as our critics often point out, a charity can be impactful without being demonstrably impactful. But when one charity proves itself and another leaves you guessing, it seems clear to us which one offers the “better bet” – and more “expected lives saved” – given the information available. When you have the option of giving to an outstanding charity that demonstrably can save a life, how do you justify giving to a charity whose true impact is essentially a big question mark?

I’ll leave this blog’s last words for 2009 to Natalie, a relatively recent GiveWell hire (she has been working full-time on research since July).

Sometimes I’m almost tempted to give to a charity I know less about. I’ve been over VillageReach and I’ve seen how complex the situation is and how many questions there are. If I gave to some charity I know nothing about, I could just think about the story they tell and feel good and not have these nagging doubts. But I’m not going to do that – in the end it’s more important to me that I really make a difference.

VillageReach currently holds the top slot in our top charities list. While I will be donating to the Stop Tuberculosis partnership (discussed yesterday), Elie’s donation this year went to VillageReach.

VillageReach consults on health system logistics in high-poverty, remote areas to help life-saving supplies get to those who need them. It may not be the most attention-grabbing story (”improving operational efficiency” doesn’t have the same ring to it as giving a cow or fixing a smile), but

• It’s directly addressing a key issue - limited health system capacity - that may limit the amount of good donations to any health-related cause are able to do.
• Immunizations in sub-Saharan Africa are a top global health priority (ranked #4 by the Copenhagen Consensus, for example and have been estimated to avert infant deaths for ~$200 each (estimate includes all costs).

In other words, VillageReach would be accomplishing tremendous good if it could improve logistics to deliver more vaccines. Can it?

Pilot project

VillageReach assessed its pilot project in Mozambique (Cabo Delgado province) with more thoroughness than we’ve seen almost anywhere outside of JPAL and IPA. It is a great demonstration that just because a charity is relatively small, and experimental, doesn’t mean it can’t collect substantive and compelling evidence regarding its impact.

VillageReach collected (and has now published) data on trends in vaccine/supply deliveries, “stock-outs” of vaccines (the single most compelling indicator, in our view) and vaccination coverage rates. It also conducted vaccination coverage surveys in both the province where it worked and a nearby province for comparison. After examining this evidence along with some independent analysis with our own, we’ve concluded that the case for impact (despite not including a randomized controlled trial) is quite compelling: VillageReach caused a significant jump in vaccination rates, which we believe can be fairly extrapolated to lives saved at well under $1000 each.

VillageReach hoped to transfer responsibility for the modified logistics system (which has a dedicated team responsible for delivering supplies, rather than each clinic responsible for its own collection) to the government, but available evidence suggests that the transfer failed. The fact that vaccination rates have since fallen is further evidence that VillageReach made a difference while they were there, but obviously discouraging relative to what they had hoped for. We have expressed doubts in the past about donors’ getting their hopes up for “sustainability” in aid, and think it’s worth noting that even VillageReach’s project – with its strong case for impact as well as a case that it actually would save the government money – has so far failed at the ambitious goal of changing government practices.

More details at the “Does it work?” section of our review.

The future of VillageReach

There are a lot of questions about the future of VillageReach. As a small organization, it sees frequent changes in its plans and engagements. It has taken on a variety of consulting projects, some of which we have little information about and could be significantly different from where it has succeeded before.

However, it has provided an unusual amount of clarity and specificity about the likely use of additional donations. We feel that VillageReach can productively absorb up to $2.5 million in unrestricted donations over the next year and that these funds will be used for work very similar to its pilot project: reactivating its program in the original province, in addition to possibly expanding to one or more neighboring provinces.

Pros and cons

VillageReach is very different from the “ideal charity” we had in mind early in our international aid research. (Our “ideal charity” is much closer to the charity I discussed yesterday, the Stop Tuberculosis Partnership). It is relatively small and still represents something of an experiment, which has both pros and cons.

VillageReach is a small charity that has struggled at times to raise enough funding, not a large charity built to absorb huge sums of money.

• This adds a certain kind of uncertainty to its situation, as its plans hinge heavily on how fundraising goes as well as other factors (such as government cooperation) outside its control.
• But being small also comes with advantages for a donor. The need for more funding is extremely concrete and some extra funding could make a huge difference to VillageReach’s ability to perform up to its potential. In addition, it is feasible – if difficult – to have a complete view of VillageReach’s work. Where a large charity like Stop TB Partnership relies heavily on audits and spot-checks – leaving open a lot of room for unobservable error – VillageReach is measuring, and accountable for, the results of everything it does.

VillageReach has only one demonstrable success to its credit. It is still establishing the potential of its model.

• VillageReach is “experimental” in a sense. It doesn’t have the same consistency, accomplishing similar results in many times and places, as Stop TB Partnership.
• However, we think this is a very different level of “experiment” from an organization with no demonstrated impact. We think there is enough information about VillageReach’s past impact for an individual donor to be confident that its future activities are a “good bet.”
• And with the risk of an “experiment” comes upside. If VillageReach continues to demonstrate success, it could affect the way health logistics works all over the world, through the spread of information rather than through its own funds. The problem we have with most “experiments” in international aid is that we aren’t convinced that they will produce the information needed to (a) determine whether they’ve succeeded and (b) bring about widespread adoption if they have. By contrast, VillageReach has demonstrated a willingness and ability to produce credible evidence of impact.

VillageReach does not fully share GiveWell’s priorities.

• We are most interested in “doing more of something that demonstrably improves lives.” VillageReach has expressed great interest – and allocated funds to - the more ambitious goal of “advocacy” (broadly defined), changing the way governments and larger funders operate.
• Yet even as it aims for these more ambitious goals, we believe that VillageReach is in fact saving lives at a level of cost-effectiveness that rivals that of any other charity. So we see its more ambitious goals almost as a “bonus” – if it succeeds at them, we’ll have helped fund a truly incredible level of impact, and if it fails at them, we still expect to have gotten our “money’s worth” (lives saved for under $1000 each).

Bottom line

The pros and cons listed above are largely mirror images of each other. Stop TB Partnership is an outstanding “large charity” – with both the advantages and concerns that accompany great size – and VillageReach is an outstanding “small charity,” with a different set of advantages and concerns. Both give good answers to all our major questions, leaving only the kind of concerns that are likely inevitable with charities of their size.

Which you have more confidence in, and prefer to donate to, is a judgment call. We have ranked VillageReach as our top charity because Elie and I happen to agree that it seems like a “better bet” overall (to approach the impact implied by the academic cost-effectiveness estimates) – we think the ability to hold it fully accountable outweighs the risks that come with being small and experimental.

Elie and I do part ways when it comes to the ultimate donation decision. I am giving to Stop TB Partnership for the personal values-related reasons discussed yesterday. Elie doesn’t share the values I mentioned and has donated to VillageReach.

The Stop Tuberculosis Partnership is one of our top-rated charities and the one I will personally be giving to this year.

Like most charities, the Stop Tuberculosis Partnership has an admirable goal. It uses donations to fund grants of tuberculosis drugs to governments, and also assists them with (and monitors their implementation of) the WHO-recommended tuberculosis control strategy. It’s an appealing activity because:

• Tuberculosis is an infectious disease that is frequently fatal, yet highly treatable.
• Standard tuberculosis control is estimated at $150-$750 per death averted (independent estimate including all costs – more on the organization providing this estimate).
• The authors of the Copenhagen Consensus disease paper rate tuberculosis control as the #1 priority for donors (pg 51).
• The more drug-resistant forms of tuberculosis may represent a major catastrophe risk, making it particularly important to avoid medical mismanagement and inappropriate use of anti-tuberculosis therapy. We haven’t looked into the “catastrophic risk” aspect of TB control much, but I see this as an added benefit that would come from supporting a good TB control organization.

As with most charities, the story doesn’t end there – there is a lot that can go wrong with tuberculosis control, and we came at the Stop Tuberculosis Partnership with a lot of questions. But unlike most charities, the Stop Tuberculosis Partnership has provided stellar answers that have increased our confidence in what it’s doing. A sample:

Do governments use the drugs for treatment, or sell/lose/misuse the drugs?

• In order to receive the drugs, governments must agree to provide a certain level of coverage, i.e., free treatment for patients, as well as adhering to multiple other conditions of the WHO-recommended tuberculosis control strategy.
• Stop TB then performs spot checks of medical facilities (audits at locations chosen by Stop TB, not by the governments) as well as examination of country-wide data (particularly treatment success rates) and data reporting systems.
• Treatment success rates are generally high.

The answer to the above question relies on audits. Does Stop TB carry out these audits consistently and produce substantive information on country compliance? We have positive though not fully conclusive indications.

• We have seen four audit reports – those of the highest-burden countries – and found them to be quite substantive and straightforward about strengths and weaknesses of the TB control programs.
• We believe that the audits are carried out consistently, as Stop TB’s annual reports from 2003-2005 summarized the outcomes of audit reports from all drug grant recipients. Stop TB has changed its reporting format, but has told us that it still does audit all grantees.
• We have seen specific instances of funding being cut off for noncompliance. (See Stop TB’s progress reports). It is very rare to see a charity publicly disclosing a shortcoming like this, and to be able to confirm that it was handled appropriately.
• Details at the “Does it work?” section of our Stop TB review.

But even if the governments are carrying out quality tuberculosis control programs, how much of Stop TB’s support is really resulting in more coverage – as opposed to simply substituting for funds the government would have spent anyway? We would guess that there is a degree of “fungibility,” but feel that Stop TB takes significant steps to maximize the impact of its funds.

• Pre-grant, it conducts detailed analysis of governments’ other sources of funding.

• Post-grant, it examines trends in the coverage of treatment before and after it became a funder. Though these trends do not have the rigor of a randomized controlled trial, we feel they provide reasonable evidence that Stop TB’s funding results in additional coverage.
• Details at the discussion of this issue in our review.

Does Stop TB, itself, have room for more funding?

• Stop TB has provided us with analysis of projected revenues and expenses, showing a significant gap (about $10 million a year over the next four years). We are not cleared to post this analysis publicly, but Stop TB will consider sharing it with interested donors who contact them.
• We have also tried to dig into the exact meaning of this “gap,” and have only been partially satisfied with the response.
  • Stop TB has cited specific examples of underfunded countries whose “buffer stock” has had to be cut (so that they have enough drugs to treat the projected number of patients, but little margin for error).
  • It has also stated to us that more funding would enable it to approve the treatment of more patients.
  • It has not provided quantitative scenario analysis, so it’s hard for us to be fully concrete about what additional donations can be expected to lead to (in terms of # people approved for treatment and in terms of treating more people vs. providing “buffer stock”).
  • Details at the discussion of this issue in our review.

Bottom line

• Big picture: Stop TB is the best international example we’ve found of a “proven, cost-effective, scalable, transparent” charity. It focuses on one highly cost-effective and proven intervention, collects and shares the information needed to address the questions we’ve come up with (and then some), and projects a significant funding gap. It is both large (significant ability to absorb more funding) and programmatically focused. It has both an excellent goal/program (tuberculosis treatment) and outstanding reporting on how its specific activities are going.
• That doesn’t mean we have no concerns. There are a lot of moving pieces here and a lot of complexity (as with just about any charity working internationally). Stop TB takes all the measures we find reasonable to assess its impact, and provides a strong picture of impact overall, but we would guess that there are many problems that its audits simply aren’t comprehensive enough to catch, as well as questions about “fungibility of money” that can’t be fully answered.
• Tomorrow we will discuss VillageReach, a very different charity that we are ultimately slightly more confident in, although there are major pros and cons to both. I am personally giving to Stop TB because (a) I put more weight on preventing adult deaths than on preventing child deaths (and tuberculosis affects adults more than vaccine-preventable diseases do); (b) VillageReach works in the most remote areas, while Stop TB works in a great variety of areas. I prefer (all else equal) to help people in less remote areas, who I believe have more opportunities (for themselves and for helping others).

Before giving season ends, we wanted to make sure to put in a good word for the UPenn Center for High-Impact Philanthropy (which we have mentioned before).

We have carefully reviewed the Center’s reports on malaria and U.S. education, both of which overlap heavily with the research we have done. We consistently find their statements to be research- and evidence-based, with citations that make it clear where their reasoning comes from. Their reports are focused on the practical issues of philanthropy and give examples of specific successful nonprofit programs.

We have learned from these reports and we believe they have significant value for donors.

We haven’t yet carefully reviewed their most recent report, on U.S. housing, health and hunger, but if and when we research these causes it will be one of the first sources we consult.

As Peter Singer and William Easterly recently discussed, there is a severe shortage of groups that provide substantive public information about the facts of giving and the research relevant to it. The Singer/Easterly discussion noted GiveWell and Good Intentions are Not Enough (another resource we find helpful) as examples of such groups. We think that the Center belongs on the list as well.

Normally, we focus on identifying outstanding charities, and minimize the time spent on opaque or otherwise lackluster ones. But lately, we’ve gone into a bit more detail about our take on several of the best-known and most appealing charities out there.

What all of the charities below have in common is that (a) we have major questions and concerns about their activities; (b) the information necessary to see how serious these concerns are does not seem to be available. (In most cases our assessment is based on significant back-and-forth with the charities themselves, though in some cases we are going off their website.)

• Kiva: commonly perceived as a way to (a) make interest-free loans to (b) entrepreneurs you personally select. In fact, it is neither. (See GiveWell Board member Tim Ogden’s summary of the recent discussion/controversy.)

  We ultimately see a Kiva “loan” as a gift to a microfinance institution, and we have major concerns about supporting any but the most outstanding microfinance institutions as well as concerns about Kiva’s due diligence process specifically.

  Full set of blog posts on Kiva

• Grameen Foundation and other large U.S. microfinance charities: in addition to our general concerns about microfinance charities (and the fact that these charities have not provided the information that would address them), we find that these organizations are rarely even clear about what their value-added is. Also see the disappointing (and eye-opening) evaluation of the Village Phone program.

  Full set of blog posts on Grameen Foundation

• Heifer International: commonly perceived as a way to “give a cow to a poor family as a gift,” but this is in fact a donor illusion -donations support Heifer International’s general “agricultural assistance” activities. We have concerns about this general area and concerns about giving livestock specifically. Neither Heifer’s website nor its grant application have provided the kind of information needed to address these concerns.

  Full set of blog posts on Heifer International

• Smile Train: Prof. Steven Levitt of Freakonomics sees it as using an innovative model to deliver cleft surgeries for $250 apiece. Our analysis suggests that it has “more money than doctors” and that it is therefore out of room for more funding. Smile Train’s recent removal of the charts we questioned and its own explanation of why it is giving grants to other major charities appear to support our analysis. In addition, we have major concerns about surgery charities that Smile Train has not provided enough information to address.

  Full set of blog posts on Smile Train

• UNICEF offers donors the chance to “buy” vaccines and other items. We suspect that this offer is a donor illusion, but the larger issue is that UNICEF provides almost no information at all about its huge variety of activities and how they are going. Most giant international aid charities (Save the Children, World Vision, etc.) are similarly complex and opaque, with CARE coming the closest to meaningful transparency.

  Full set of blog posts on UNICEF

• The Acumen Fund stresses its metrics-driven approach to “patient capital,” investing in businesses that aim for long-term sustainability (i.e., profitability) and social impact. As of now, we see no substantive available information about its track record.

  Full discussion

• The Robin Hood Foundation also stresses its emphasis on metrics, quantification, and analytical thinking. But we have found that it is unwilling to share any information about the results of its grantmaking. (Note that it also pushes the “0% overhead” donor illusion.)

  Full set of blog posts on The Robin Hood Foundation

• The Millennium Villages project states that it is “demonstrating that it’s possible to escape … extreme poverty” (front page of its website). As far as we can tell, very few results have been demonstrated to date, with a few boosts in crop yields as its main evidence of impact - evidence we feel is highly questionable.

  Full discussion

• The Worldwide Fistula Fund has been praised by Nicholas Kristof as “giving new life to the pariahs.” Our assessment is that the Fund is pushing a debatable agenda whose effectiveness has yet to be shown.

  Full discussion

• The Carter Center is, in our view, by far the “best bet” (for an individual donor, our focus) of the charities listed in this post. It has achieved substantial, demonstrable impact in the past through its work on guinea worm eradication, and unlike all of the charities listed above, it publicly discloses substantive technical information on the impact of its past activities.

  However, there is a crucial piece of information missing: how it allocates its budget between programs (in particular, how much it spends on its stronger and better-documented vs. more questionable programs). It has declined to share this (basic, in our view) information in repeated exchanges.

  Full set of blog posts on The Carter Center

We think the above charities are fairly representative of “average” charities in international aid. Some tell better stories than others and some have more disclosure than others. But in almost all cases, international aid charities are (a) carrying out complex projects that can fail to do good (or even do harm) in a variety of ways, and (b) not systematically sharing the information that would make it possible to assess how their work is going.

GiveWell is devoted to finding charities in which we can have more confidence. We’ll be discussing our two top-rated charities working internationally in forthcoming posts.

It seems particularly hard to find information about the past impact of “gifts of livestock” programs (such as those promoted by Heifer International). I’ve been thinking about such programs conceptually, though, and I have a lot of trouble understanding the reasoning behind these programs. Two key points:

• It seems like giving out livestock brings with it all of the problems and challenges of giving out cash.
• It seems like giving out livestock also brings additional problems and challenges that don’t apply to giving out cash.

Giving out livestock brings the same problems and challenges as giving out cash

We’ve written before about the idea of cash transfers. One potential problem with giving out cash is that the more powerful people in a community may end up dominating (monopolizing?) the benefits. Giving out truly valuable gifts could interfere with power dynamics, incite jealousy, and fail to reach those that donors actually intend to help.

It seems to me that all of these concerns apply in full force to gifts of livestock. I doubt there are many people in the world who would turn down a free cow (I’m not sure I would). Even if one has no ability to take care of the cow, there is always the option of selling it (if one has access to markets) or simply slaughtering it and eating or selling the meat.

I couldn’t feel confident in a charity giving out livestock unless I saw compelling evidence that they were getting livestock to people in need, and not just to anyone (or just to the most powerful people) interested in free livestock.

Giving out livestock brings other problems and challenges as well

• Are the livestock in good health? Will they meet recipients’ expectations, or will they die or underproduce, potentially causing people to make bad plans and investments?
• Do the recipients of livestock gifts have the ability, in terms of knowledge and resources, to take care of the livestock well? (Similar problems as in the above bullet point could arise if they don’t.)
• Do the recipients of livestock intend to take care of the livestock well? Or is there reason to be concerned that gifts of livestock could lead to cruelty to animals?
• Are there other unforeseen consequences of introducing large numbers of livestock into a community? A few years ago, there were some allegations that “over grazing by goats in arid environments has disastrous effects on the fertility of the land … these ‘gifts’ merely add to the problems of hard-pressed communities because of the drain on limited resources the animal represents.” I haven’t been able to find any facts behind these allegations and I’m not sure what they’re based on.
• Most importantly, might recipients benefit more from other gifts - and why shouldn’t they make that assessment themselves? Perhaps the story Heifer International tells is correct, and livestock would make a tremendous difference for a family. If that’s the case, then a cash gift could be expected to be spent on livestock. If one of the many concerns above applies - or livestock is not what’s most helpful for any number of reasons we simply haven’t thought of - then a cash gift will be used on something else.

  Why is it better for a charity to decide people’s needs for them? This question isn’t entirely rhetorical - there could be a good reason - but it seems that the burden of proof on a statement like “A cow is better for you than anything else you could buy with what the cow costs” should be on the charity.

Bottom line

I have trouble understanding the idea of livestock gifts, from the perspective of maximizing positive impact.

I understand that they make a good ad campaign, possibly because they draw people’s attention to the possibility of using a one-time gift to permanently escape from poverty (even though a cash gift can just as easily lead to a story like this, and could also lead to a lot of other positive stories that we simply haven’t considered).

It’s a scary thought, but it seems possible to me that these programs exist entirely because of how they can be marketed to donors, instead of for any reasons relating to maximizing good accomplished. What am I missing?

We’ve thought and written a lot about microfinance lately. As of now, here’s where we stand.

What microfinance is and isn’t

First, it’s important to recognize that most of what you’ve heard about microfinance is false. It isn’t primarily about funding business expansion.. It isn’t a “proven solution” to poverty. And it doesn’t leverage your donation far more than other options.

Rather, we think of microfinance as a way to help people with low, volatile incomes manage their financial lives, an idea that is well argued in the recent Portfolios of the Poor study. This study implies that microfinance is really about providing one more option for borrowing rather than the only way to borrow, and that the borrowing is continual rather than “one crucial loan to escape poverty” - more like a credit card than a business investment. (This would explain why “graduation” from microlending programs appears rare).

What to look for

Does microfinance do good? It depends on a lot of things.

• If loans are constantly and heavily subsidized, they can be thought of as similar to giving out cash, in which case our primary concern is that benefits reach the right people.
• On the other hand, if loans are not subsidized, a microfinance institution’s profits could be taken as a sign that it has paying customers. This in turn could be a sign that it is providing empowerment.

With the latter goal (which seems to be the more common one), there is a big question about what role donations can and should play. We have expressed serious concerns about mixing donations with for-profit enterprises, with the possible result that donations end up padding profits (concept; example). In addition, we worry that there are too many donations blindly chasing the microfinance “story,” with the result that donations end up disappearing into nebulous activities.

There is also a question about the extent to which loans are truly providing empowerment. There is evidence that borrowing is bad for at least some borrowers.

We have developed a set of critical questions both about microlending and microsavings, to get at the question of whether an institution is helping people. We’ve looked hard for organizations that can answer our questions.

What we’ve found

In trying to answer the above questions, we’ve become fairly pessimistic about the area of charitable microfinance in general.

• The large U.S. charities are rarely clear even about what their value-added is. To the extent that I’ve been able to see their due diligence on partners (most of which is unfortunately confidential), it seems focused on financial performance, with much weaker examination of social impact-related questions.
• Something as simple as the “repayment rate” turns out to be reported, by standard practice, in what we feel is a highly misleading way. We’ve been shocked at how hard it is even to get what we consider the “real repayment rate” out of a microfinance charity.
• We’ve also been surprised by how often people drop out of microlending programs, generally for negative reasons.
• All in all, the picture we have is of a sector that tells stories about helping people but largely measures and rewards financial performance. As a result, microfinance institutions seem to have strong incentives to maximize their scale and their profits, even if doing so isn’t good for the borrowers.

Bottom line

All in all, we would guess that microfinance as a whole has done a great deal of good, but has also probably done some harm. We are more pessimistic specifically about microfinance donations in the current environment. For the reasons outlined above, we believe that giving to an “average” or “typical” microfinance charity – or giving with an illusory “peer to peer” relationship as the extent of your due diligence – is a fairly bad bet. At the very least, it will deliver far less good, and far more potential harm, than the typical microfinance narrative suggests.

Yet we still find the basic idea of providing financial services to people with low and volatile incomes very appealing as a way to help people … if it is done in a way that stresses social impact and uses donations responsibly.

We believe that microsavings is a particularly promising area, although we haven’t found a microsavings charity we can be confident in.

We believe that the Small Enterprise Foundation is a microlending institution that is truly and appropriately focused on achieving positive social impact. We’ll be writing more about it.

We’ve just updated our review of our top-rated organization, VillageReach. (Our July 2009 version is still available for posterity.) The big picture remains the same – we feel this organization has a stronger case for cost-effective impact than any other charity we’ve seen – but there have been changes in VillageReach’s financial situation and plans, as well as substantial new information on its answer to the “room for more funding” question.

Ultimately, VillageReach has answered this question in what we consider the “ideal” way: by specifically outlining its projected activities, and outputs, at different levels of projected revenue.

• VillageReach’s top priority for unrestricted donations is to reactivate its pilot logistics project in Cabo Delgado (the Mozambique province where the project was carried out) and part of Niassa (a neighboring province).
• It currently has enough funding to expand into one or the other (Cabo Delgado or part of Niassa).
• About $400,000 in additional funding would close the full gap needed to expand into both over the next year. An additional $1.2 million in needed funds are projected over the full 3 years.
• Beyond that point, VillageReach has committed to expand into additional provinces at a cost of about $1 million per year.
• Thus, if VillageReach brings in over $1.5 million in unrestricted donations, we will expect to see it planning further expansion of its logistics program in Mozambique. If it brings in over $2.5 million over the next year, we will likely suspend our recommendation on the basis that it has as much funding as it can productively use.

(Full details available at our discussion of VillageReach’s future activities.)

We know that these numbers are projections, and subject to significant change and error. But what’s valuable about them is that they’re specific enough, and concrete enough, that VillageReach can be held accountable. Based on how much revenue it brings in, we know which activities we can expect it to carry out. This is a truly meaningful projection of the “impact of your donation.” It’s a picture of organization priorities, which can be checked over time (”we will do X if we get $Y”), instead of a discussion of how “your” funds are allocated, which we ultimately can’t see a way to verify in the face of concerns about fungibility.

It’s interesting to me how much work it took to get to this point. When we originally asked VillageReach for “funding gap analysis,” it provided a picture of the minimum amount of funding it would need to continue activities (the October funding gap memo (PDF)). It seems to us that funders often ask the question “Is your organization stable? Are there other funders contributing here?” but rarely ask the question “What is the maximum amount of funding you can productively absorb?”

But we don’t ultimately see why analysis like this should be so rare. The final product here is simple. It’s a matter of being clear about organizational plans and priorities – it doesn’t involve nearly the same kind of complexity and expense as, for example, gauging the impact of past activities.

We feel that it’s feasible for almost any charity to provide substantive, useful, relatively simple information answering “How will future donations be used?” The reason few charities provide it seems to be that few donors push for it.

I’m very concerned about the incentives for microfinance charities. As I see it, these are the things that they are “rewarded” for:

• Profitability, for obvious reasons. Any for-profit investors in microfinance institutions are presumably putting great pressure on them to produce a good bottom line. (We believe it’s not safe to use profitability as a proxy for social impact.)
• “Scale” (clients reached, loans made, etc.) and “repayment rate.” These are the numbers that are put front and center by charities like Kiva, Grameen Foundation, and Accion among others. (We have serious questions about what the “repayment rate” even means as well as whether repaid loans = people helped.)

We feel strongly that there are many more questions a microfinance institution must answer to give an idea of whether it is helping people, and worth donating to. And in theory, many others feel this as well. But looking around MixMarket shows how much more attention the “scale and profits” indicators are getting as of today.

• The frontpage cites “1,678 total MFIs reporting data” (and we have collected the data, which is overwhelmingly complete on the financial indicators front, for the 268 members of U.S. networks). Yet only 66 institutions have filed social performance reports.
• The independent ratings found on MixMarket also seem overwhelmingly focused on issues of scale and potential profitability, as opposed to social impact (for a representative example, see the Microrate documents at WISDOM’s profile).

The bottom line? It looks to us like all of the pressure that these institutions face is to maximize scale and profits, without much eye to making sure that they’re improving clients’ lives. And that could explain some eye-popping quotes from publicly available surveys of microfinance dropouts (full documents available at that link; emphasis ours below).

• “At the end of the meeting the research team watched the process of one group being pressured to complete payment. They were told that they should not leave the meeting until the money was paid. There was heated discussion among them, with members telling each other to contribute, and denying that they themselves had the money with which to contribute. There was Ush. 8,000 missing. Finally a man from another five person group provided the missing money. He told the team that he would probably get the money back from the defaulter (who was not present) and that the defaulter would ‘probably buy him a soda’.” (Uganda report pg 13)
• “The research team also found one instance where one MFI had started operations in a poor area, issued loans, experienced quite widespread problems with loan repayment and decided to withdraw the programme in its entirety. This resulted in many people’s savings being attached to repay the outstanding loans of others, and the MFI left behind a fair degree of chaos and bad feeling.” (Uganda report pg 13)
• “the treasurer said that the group’s monthly repayment (Tsh. 300,000) that was to be deposited in SEDA account had been stolen. Two days later the treasurer got very sick; he was bewitched by some of the group members. However after seeing a witchdoctor he recovered. We took him to the Ward Executive Officer; who forced him to repay the money in instalments. He did repay the money but then he was forced to leave Arusha town completely. If he had continued to stay they would have killed him through witchcraft. I lost Tsh. 4,000 through the ROSCA.” (Tanzania report pg 10)
• “Most MFI’s in Kenya have a very strong emphasis on credit but some are getting into the position where clients are virtually forced to take loans. This is partly because of the MFI’s needs to pump out loans to achieve operational sustainability and partly due to a belief that loans are good for small entrepreneurs – even if they don’t want them!” (Kenya report pg 7)
• “One of the key determinants of drop-out, often lost in the category ‘failure to repay loan’ by these studies, is the insistence by field staff that clients take loans. Irrespective of what official Head Office policy says, there is a clear understanding among most field staff that they should push out loans - often with little care for whether the clients need or can use them… Matin (1998) also notes, ‘MFI lending technology is insensitive to variations in household conditions. Most MFIs put all households on a treadmill of continuously increasing loan size and insist on a fixed repayment schedule.’” (Bangladesh report pgs 3-4)

We’ve written before about problems with the way a microfinance institution’s “repayment rate” is commonly cited. We’ve been surprised to find that most institutions do not report what most of us would think of as a “repayment rate,” i.e., the percentage of loans/dollars due that have been paid on time. Instead, they report proxies such as “portfolio at risk” that can (theoretically) be very different.

We now have an example of just how different they can be. ID-Ghana, a microfinance institution, has given us permission to post its application for funding from GiveWell. Page 2 discloses that “The write-off ratio has shot up lately because of the clean up that followed the phase out of our old loan products which proved to be inefficient and impactless,” yet the repayment ratio is reported as 99% (same paragraph). That’s because the definition of “repayment ratio” being used ignores loans that have been defaulted on and written off. Only “at-risk” (but still-on-the-books) loans lower this “repayment ratio.”

These charts show how drastic the disrepancy is, particularly from June 2009 on:

To be clear, we think ID-Ghana’s reporting is entirely consistent with standard reporting practices. To a large degree, that’s what worries us. By industry standard reporting practices, a 99% “repayment ratio” can be consistent with a 30%+ default rate - and is in this case.

This is why we’ve insisted on requesting what we call the “real repayment rate”, defined as the percentage of loans that have been paid off on time divided by the percentage of loans that have come due over a given time period (”loans” here can refer to number of loans or dollars lent).

What has shocked us is how few microfinance institutions are able to provide the real repayment rate. In fact, all of the major U.S. microfinance institutions we’ve contacted (excluding FINCA, which declined to apply for funding at all) have explicitly told us that they cannot or will not provide real repayment rates for their partners.

We’ll be writing more about the Small Enterprise Foundation, the only institution that we’ve seen be fully clear about its repayment rate.

Just because someone is repaying their loans doesn’t mean they’re benefiting from the loans.

We have given some conceptual/anecdotal support for this idea in the past, linking to David Roodman’s posts on possible “overlending” and comparing microloans to payday loans. Lately we’ve been investigating something a bit more concrete: how often, and why, do microfinance clients “drop out” of microlending programs?

The basic idea is that a client could repay a loan due to pressure (from their “lending group” or the microfinance institution), making sacrifices or borrowing from elsewhere (such as moneylenders) to do so. We would expect such clients to show up as “repayers” while not necessarily staying in the program for more loans.

Our observations (details and full sources below):

• Dropout rates appear substantial, averaging 28% and often exceeding 40%, among institutions that publicly report them (via MixMarket).
• Survey data on why clients drop out is limited, but what we’ve seen suggests that “graduation” (i.e., moving to better sources of credit or no longer needing credit) is not a common reason for dropping out. Business failure and dissatisfaction with the group/staff/institution are common reasons.
• A high dropout rate can be consistent with reasonably good reported client satisfaction.

Dropout rates

We have gone through all microfinance institutions that report “social performance reports” on MixMarket (you can see the complete list of institutions, with their publicly posted reports, at MixMarket’s social performance report section) and collected the data into this spreadsheet (XLS). (Note that dropout rates are not on the list of “standard” indicators and are not reported by all MixMarket participants, but are included in “social performance reports.”) Here’s a summary of the 60 institutions that report dropout rates:

Taking the average across all 60, weighted by number of clients, yields an “average” dropout rate of 28%. Details here (XLS). That implies that in a given year, 28 out of 100 clients become non-clients (see the “social performance reports” for the details of the calculation).

Reasons for dropping out

We don’t know of any comprehensive studies of the reasons clients drop out, but in the process of searching for an outstanding microfinance institution, we have encountered several small-scale surveys. We have posted the non-confidential dropout surveys along with a summary in Excel, and hope to clear a couple more in the future (they are broadly consistent with the summary below).

The Bangladesh study specifically states that “One of the reasons that is notable by its almost complete absence from these listings of grounds for drop-out is ‘graduation’” (pg 4). The rest of the studies give the same picture: “graduation” (i.e., the idea that clients now no longer need microfinance because they can access better sources of credit and/or do not need credit) is not cited as a significant factor in any of them, except in the Uganda study (which does not state how common this factor is, but cites it as a factor specifically for “Relatively Well-off drop-outs” (pg iii)).

Business failure is a commonly cited factor (37-58% of clients cite this factor, in the studies that report numbers - see Excel summary). Issues with the “lending group,” the organization or its staff are the other most common factors. The Tanzania study cites “The inability of clients to cope with the rigid MFI policies and procedures” (pg 9) and also vividly describes a group conflict in which the treasurer claimed funds had been stolen (pg 10). The Bangladesh study states that “One of the key determinants of drop-out … is the insistence by field staff that clients take loans” (pg 3-4).

Client satisfaction

The LAPO study (see previous discussion of LAPO, Kiva’s largest partner) looks not only at the reasons for dropping out, but at overall reported client satisfaction.

The former figures seem cause for concern: there is a dropout rate of around 25% (estimated from graph on pg 7) and reasons given include “poor business performance” (applying to 24.2% of dropouts), “Burden of paying for others who had defaulted” (29.5%), and “the attitude of some staff” (cited as a major factor but without quantification). But overall, reported satisfaction looks reasonably strong:

We feel these numbers should be taken with a grain of salt, since it seems possible to us that clients could have felt pressure to report positive experiences. But the numbers do serve as a reminder that microfinance institutions have many clients who are (apparently) happy repeat customers.

Bottom line

Most microfinance institutions don’t appear to publicly report dropout rates, much less the reasons for dropping out (this observation based on the small percentage of MixMarket participants who have shared social performance reports). Those that do are likely to have more encouraging numbers than the others, and yet their numbers seem to leave substantial room for concern. Clients seem to drop out, for overwhelmingly negative reasons, at rates averaging 28% and often exceeding 40%.

We don’t mean to overfocus on the negative here. Microfinance institutions could be providing valuable services for many people, and we wouldn’t want donors to stay away from an activity that’s doing good overall even if it is doing damage to some.

But it does seem that the more we dig through the information on microfinance, the less it resembles the stories commonly told about it. Making loans can do good or harm. We feel strongly that people donating money to microfinance institutions should be asking for substantial due diligence - not anecdotes and pictures, and not the commonly cited, misleading metrics like “repayment rate,” but systematically collected information that gets at services’ actual impact on clients.

We haven’t done much work on charities that try to help orphans and vulnerable children, and we intend to do more. Here are some preliminary thoughts, though.

At first glance, this area might seem among the simplest and least controversial. SOS Children’s Villages states, “Our sponsors and donors help children whose parents are not there for them. They may be AIDS orphans, street children, child soldiers or children orphaned by war, poverty or natural disasters. We give these children a mother and a family in a home within an SOS Children’s Village. Donations pay to build the Villages and run them until child sponsors cover the running costs.” Could any sort of “impact evaluation” be helpful here? How can one deny that children without homes should be provided homes if at all possible?

However, the picture becomes far more complex upon reading something like Saundra Schimmelpfennig’s series on orphanages.

• Donor demand for funding orphanages may be outstripping actual need (we have speculated that cleft surgery and microcredit may face similar issues).
• Many of the children in orphanages are not actually orphans. Parents may send them there because they find caring for them to be too expensive; orphanages may weaken the incentives for children’s other relatives and community members to take them in.
• If, in fact, orphanages are one option rather than the only option for care, it becomes much more crucial to determine whether they are providing good conditions for children. Ms. Schimmelpfennig raises questions about this issue.
• There is an ongoing debate in academia about whether abandoned children are better off in institutions or being cared for by relatives/community. One person with field experience told me he personally saw a situation in which he believed that orphanages were actively making the situation worse, and Ms. Schimmelpfennig’s series also implies that this is a serious possibility.

None of this means that donating to orphanages is a bad idea. What it means is that, as usual, the appealing story you see on a charity’s website has a great deal of complexity and open questions behind it. As usual, it is essential to ask critical questions, and not to let your due diligence end with “That sounds like a clear need.”

We recently argued that Smile Train has “more dollars than doctors” for its core program. In that light, yesterday’s Virginian-Pilot article (which quotes me) is interesting:

• The main story is that Smile Train has been trying to make substantial and unrestricted grants to another major cleft surgery charity, Operation Smile. This despite the fact that Smile Train has in the past raised concerns about Operation Smile (”intentionally fabricated tens of thousand of surgeries… distorted its financial reporting… squandered millions of dollars… provided shoddy medical care….”)
• The story quotes Smile Train’s President as appearing to explicitly support the “more dollars than doctors” idea: “Smile Train focused on funding operations by doctors in the countries in need … Mullaney concedes, though, that in some countries, such as Somalia and Haiti, the need outstrips the number of surgeons available to do the work.”

It’s hard to make sense of Smile Train’s wish to make unrestricted grants to Operation Smile, except by accepting that Smile Train is out of room for more funding in its core program.

Possibly, Smile Train’s concerns about Operation Smile have been addressed. Arguably, the decision to grant extra funds to other organizations is admirable (we don’t know whether other charities respond to the same situation by simply piling up assets). But it certainly seems difficult to argue that Smile Train’s donors should think of themselves as funding more of the “$250 per surgery” core program.

We previously discussed some simple ways to get at the difficult question of “room for more funding.” One approach that has been more difficult than expected is asking charities themselves to help figure out where more funds are likely to go - answers tend to be vague and tend to target what the charities think we want to hear. How to get specific, concrete, and credible info?

One approach can work specifically for grantmaking charities, i.e., charities that pool donations and make large grants to other organizations. Grantmaking charities are more common than you might think in international aid - arguably any large organization like UNICEF or CARE can be thought of as making grants from its central office to its many projects around the world.

The question we ask such charities is: Can you share examples of strong grant proposals that went underfunded - or rejected - because of insufficient funds?

This is a very different request from asking for rejected grant proposals. An organization may be funding a small percentage of grant proposals, while still funding all reasonable ones. However, an organization that can show proposals that are both strong and underfunded is making a strong case that it has room for more funding.

The Against Malaria Foundation, our 3rd-ranked international aid charity, gave a stellar response to this request. At its GiveWell review, you can see both approved and rejected proposals, and it seems fairly clear that funds are indeed the bottleneck to approving distribution of more insecticide-treated nets.

Asking for a grantmaking organization’s “best unfunded grant proposals” is a pretty simple idea. But you won’t see it requested in any tax return, included in any online charity information database, or mentioned in any of the various impact/outcomes frameworks that various nonprofit sector actors are putting together. Evidence that the “room for more funding” question is largely neglected as of today.

We have been struggling with the “room for more funding” question since the first days of GiveWell, and we have gradually developed a variety of approaches to it.

The most basic approach, and the one we’ve used for most of our history, consists of the following:

1. Gain confidence in an entire organization; do not overfocus on one program.
2. Examine financial data, looking for a few basic patterns and warning signs.
3. Ask the charity how additional funds will be used.

Gain confidence in an entire organization

We generally seek to scrutinize and examine activities accounting for over 50% (at a minimum) of a charity’s budget. Some organizations are small enough or simple enough that doing so is fairly straightforward. Other organizations are large and complex but very well-documented. Organizations that are both complex and poorly documented generally don’t get past the first stage of our process.

Our decision to evaluate entire organizations instead of individual programs has, arguably, drastically reduced our options for recommended charities. We have found that many large organizations can’t even answer the “What do you do?” question at the organization level.

But as of now, we see no other reasonable choice. We strongly doubt that donating to a particular program is wise or efficacious.

If you gain confidence in a whole organization, you can give there without worrying too much about what specific activity is next on the agenda. As long as you continue to hold the organization accountable over time.

Basic patterns in financial data

Late in the process with a strong organization, we will analyze its financial data. The details of our financial metrics here. Questions we ask include:

• Is the charity large enough that it can plausibly absorb substantial additional funds? (This question is actually applied at the beginning of our process.)
• Have the charity’s expenses been growing over time, implying that it is on a general trajectory of expansion?
• Does the charity have a reasonable level of assets, given its size? If its assets are too low, we worry that it isn’t stable; if they are too high, we worry that it is piling up reserves because it cannot productively spend additional funds.
• Which programs does the charity spend most of its funds on? Which programs have been expanding in the past, and are projected for future expansion, implying that they are on a general trajectory of expansion? (Note that this question generally requires a different kind of financial data than is provided in audited financials and tax returns. And we have been surprised at some of the charities that cannot/will not share such financial information.)

We don’t have a set formula; no single “No” answer will disqualify a charity from getting recommended. But asking these basic questions has raised serious questions about some charities (examples: Smile Train, The Carter Center), while our top charities have more or less sailed through these basic tests.

Asking charities what their plans are

This approach has turned out to be far more difficult than it sounds.

In our first year of research, we used a grant application with a question asking specifically:

What would a significant increase in funding (including, but not limited to, a Clear Fund grant) allow your organization to do that it could not do otherwise?

You can see answers via the grant applications submitted by U.S. equality of opportunity charities. Overall, responses were not very helpful.

• Charities have a tendency to try to tell funders what they want to hear.
• Charities often are very bad at guessing what we want to hear.

In many cases we followed up with charities and tried harder to make our meaning clear and get meaningful answers, but it’s only recently that we’ve really developed questions that seem to get us somewhere. We’ll be discussing these in future posts.