Two interventions that command a lot of attention are microfinance (financial services, particularly small loans, for the very poor) and improved cookstoves (with the hope of reducing air pollution). We’ve recently seen a couple of helpful summaries of relevant research:

• David Roodman summarizes the most rigorous research on microfinance. There are now five randomized controlled trials on microlending that have at least published some preliminary results; it looks like there is very little in the way of direct poverty reduction or wellbeing improvements, though there is positive impact on “stimulating enterprise.”
• Charles Kenny discusses a recent study that randomized heavy subsidies of cookstoves in India, and found that “Households failed to use the stoves regularly or appropriately, did not make the necessary investments to maintain them properly, and use ultimately declined further over time,” leading to no significant positive impact. According to Mr. Kenny, this result is consistent with previous literature on the matter. On the other hand, Aid Thoughts points to another study in Senegal reporting, after one year, that “households receiving an improved cooking stove used less wood, spent less time cooking meals, reported better indoor air quality and (for women, who presumably did all the cooking) were significantly less likely to have respiratory disease symptoms, eye problems. Nearly all recipients of a stove used it at least seven times a week.” We note that the latter study discusses only one-year effects, while the India study found “a meaningful reduction in smoke inhalation in the first year [but] no effect over longer time horizons.” Note that we haven’t carefully examined these papers and that cookstoves are not a focus of ours, but since the recent studies are both fairly rigorous we thought it was worth noting them and their conflicting results for interested readers.

VillageReach currently holds the top slot in our top charities list. While I will be donating to the Stop Tuberculosis partnership (discussed yesterday), Elie’s donation this year went to VillageReach.

VillageReach consults on health system logistics in high-poverty, remote areas to help life-saving supplies get to those who need them. It may not be the most attention-grabbing story (”improving operational efficiency” doesn’t have the same ring to it as giving a cow or fixing a smile), but

• It’s directly addressing a key issue - limited health system capacity - that may limit the amount of good donations to any health-related cause are able to do.
• Immunizations in sub-Saharan Africa are a top global health priority (ranked #4 by the Copenhagen Consensus, for example and have been estimated to avert infant deaths for ~$200 each (estimate includes all costs).

In other words, VillageReach would be accomplishing tremendous good if it could improve logistics to deliver more vaccines. Can it?

Pilot project

VillageReach assessed its pilot project in Mozambique (Cabo Delgado province) with more thoroughness than we’ve seen almost anywhere outside of JPAL and IPA. It is a great demonstration that just because a charity is relatively small, and experimental, doesn’t mean it can’t collect substantive and compelling evidence regarding its impact.

VillageReach collected (and has now published) data on trends in vaccine/supply deliveries, “stock-outs” of vaccines (the single most compelling indicator, in our view) and vaccination coverage rates. It also conducted vaccination coverage surveys in both the province where it worked and a nearby province for comparison. After examining this evidence along with some independent analysis with our own, we’ve concluded that the case for impact (despite not including a randomized controlled trial) is quite compelling: VillageReach caused a significant jump in vaccination rates, which we believe can be fairly extrapolated to lives saved at well under $1000 each.

VillageReach hoped to transfer responsibility for the modified logistics system (which has a dedicated team responsible for delivering supplies, rather than each clinic responsible for its own collection) to the government, but available evidence suggests that the transfer failed. The fact that vaccination rates have since fallen is further evidence that VillageReach made a difference while they were there, but obviously discouraging relative to what they had hoped for. We have expressed doubts in the past about donors’ getting their hopes up for “sustainability” in aid, and think it’s worth noting that even VillageReach’s project – with its strong case for impact as well as a case that it actually would save the government money – has so far failed at the ambitious goal of changing government practices.

More details at the “Does it work?” section of our review.

The future of VillageReach

There are a lot of questions about the future of VillageReach. As a small organization, it sees frequent changes in its plans and engagements. It has taken on a variety of consulting projects, some of which we have little information about and could be significantly different from where it has succeeded before.

However, it has provided an unusual amount of clarity and specificity about the likely use of additional donations. We feel that VillageReach can productively absorb up to $2.5 million in unrestricted donations over the next year and that these funds will be used for work very similar to its pilot project: reactivating its program in the original province, in addition to possibly expanding to one or more neighboring provinces.

Pros and cons

VillageReach is very different from the “ideal charity” we had in mind early in our international aid research. (Our “ideal charity” is much closer to the charity I discussed yesterday, the Stop Tuberculosis Partnership). It is relatively small and still represents something of an experiment, which has both pros and cons.

VillageReach is a small charity that has struggled at times to raise enough funding, not a large charity built to absorb huge sums of money.

• This adds a certain kind of uncertainty to its situation, as its plans hinge heavily on how fundraising goes as well as other factors (such as government cooperation) outside its control.
• But being small also comes with advantages for a donor. The need for more funding is extremely concrete and some extra funding could make a huge difference to VillageReach’s ability to perform up to its potential. In addition, it is feasible – if difficult – to have a complete view of VillageReach’s work. Where a large charity like Stop TB Partnership relies heavily on audits and spot-checks – leaving open a lot of room for unobservable error – VillageReach is measuring, and accountable for, the results of everything it does.

VillageReach has only one demonstrable success to its credit. It is still establishing the potential of its model.

• VillageReach is “experimental” in a sense. It doesn’t have the same consistency, accomplishing similar results in many times and places, as Stop TB Partnership.
• However, we think this is a very different level of “experiment” from an organization with no demonstrated impact. We think there is enough information about VillageReach’s past impact for an individual donor to be confident that its future activities are a “good bet.”
• And with the risk of an “experiment” comes upside. If VillageReach continues to demonstrate success, it could affect the way health logistics works all over the world, through the spread of information rather than through its own funds. The problem we have with most “experiments” in international aid is that we aren’t convinced that they will produce the information needed to (a) determine whether they’ve succeeded and (b) bring about widespread adoption if they have. By contrast, VillageReach has demonstrated a willingness and ability to produce credible evidence of impact.

VillageReach does not fully share GiveWell’s priorities.

• We are most interested in “doing more of something that demonstrably improves lives.” VillageReach has expressed great interest – and allocated funds to - the more ambitious goal of “advocacy” (broadly defined), changing the way governments and larger funders operate.
• Yet even as it aims for these more ambitious goals, we believe that VillageReach is in fact saving lives at a level of cost-effectiveness that rivals that of any other charity. So we see its more ambitious goals almost as a “bonus” – if it succeeds at them, we’ll have helped fund a truly incredible level of impact, and if it fails at them, we still expect to have gotten our “money’s worth” (lives saved for under $1000 each).

Bottom line

The pros and cons listed above are largely mirror images of each other. Stop TB Partnership is an outstanding “large charity” – with both the advantages and concerns that accompany great size – and VillageReach is an outstanding “small charity,” with a different set of advantages and concerns. Both give good answers to all our major questions, leaving only the kind of concerns that are likely inevitable with charities of their size.

Which you have more confidence in, and prefer to donate to, is a judgment call. We have ranked VillageReach as our top charity because Elie and I happen to agree that it seems like a “better bet” overall (to approach the impact implied by the academic cost-effectiveness estimates) – we think the ability to hold it fully accountable outweighs the risks that come with being small and experimental.

Elie and I do part ways when it comes to the ultimate donation decision. I am giving to Stop TB Partnership for the personal values-related reasons discussed yesterday. Elie doesn’t share the values I mentioned and has donated to VillageReach.

This Tactical Philanthropy post implies that the salient difference between DonorsChoose and Kiva is that DonorsChoose is more “authentic” in terms of connecting donors to projects. (Update: Sean points out in the comment below that most of the post I cited was a quote from a former DonorsChoose employee and doesn’t necessarily reflect TacticalPhilanthropy’s opinion.)

We think there’s another important difference. Kiva and DonorsChoose work on very different types of problems, that offer donors vastly different opportunities to cause significant impact. And if we had to pick one, we’d bet on Kiva as the better option if you want to change lives.

The problem of improving children’s education is far, far more difficult than most people suspect - for our most recent coverage, see our multi-part series on the recent Harlem Children’s Zone study, and how excited the academic community has been over a one-time and modest improvement. And DonorsChoose does not offer the chance to support any of the most promising and tested educational interventions, such as the creation of intensive charter schools. Funding DonorsChoose is making a bet that classroom materials are where it’s at, even when the teachers, school systems and children’s outside environments remain the same. As far as we know, there is zero evidence that improved school supplies have resulted in any measurable improvements in the U.S., and even in the developing world the proposition seems iffy.

Funding Kiva, even if it doesn’t mean funding individuals, means funding microfinance charities. We are very ambivalent about microfinance; recent posts on this topic include questions about whether there’s any empirical support for microfinance charities’ claims and an interview with David Roodman about microfinance charity in which many unresolved questions come up. But the bottom line is that when you fund microfinance charities, your money ends up in the poorest parts of the world, and often (though we don’t know how often or how much) helps people there manage their volatile financial lives with credit, savings and other assistance.

(In fact, the way in which your money is most helpful may be by doing exactly what it has been criticized for doing - effectively funding partners’ other projects, such as savings vehicles, rather than the loans you see on your screen.)

Fund financial services in the poorest parts of the world, or fund an untested and low-intensity approach to a problem that higher-intensity programs have struggled with for decades. $900 can be five digital cameras for a classroom or a year’s income for 3 people.

That’s the decision as it looks to us, as we ask not “How can I make sure that I can see the exact person who’s getting my money?” but “How can my money accomplish as much good as possible?”

And on that note, we feel that our top-rated charities offer far better opportunities to improve lives than either, even if they can’t deliver the same emotional experience. As a donor, which would you rather have?

Note: this post does not refer to the economic empowerment grant whose process we opened on August 4. That process is still ongoing, and the funds we will grant there are in addition and distinct from the funds discussed below.

Having released our updated recommendations for international aid, we will be making grants to the two charities we have designated as “top-rated.” These grants are being made from restricted funds: donations made to GiveWell and earmarked for regranting to top charities.

We are splitting the available funds between our top two charities, and are therefore granting a total of $85,047.10: $42,523.55 to VillageReach and $42,523.55 to the Stop Tuberculosis Partnership.

We are also recommending that our donor-advised fund grant $12,523.94 to VillageReach and $12,523.94 to the Stop Tuberculosis Partnership. (In general, we make grants from this fund according to donor wishes, as specified at our GiveWell Advance Donation description, but these grants are attributed to the startup funds that GiveWell put into the fund.)

That makes a total of $110,094.98 that we are directing (through grants or grant recommendations) to top charities. This does not include GiveWell Pledges, GiveWell Advance Donations, or any other case where donors give using our research.

We still have $250,000 that is earmarked for regranting specifically within economic empowerment, and are conducting an open application in order to identify the grantee.

How transparent is CARE?

On one hand, it maintains a site at www.careevaluations.org that currently lists 448 project evaluation documents (352 of which are in English). We haven’t found anything comparable for any other of what we call the “household name” charities – enormous, well-known, aggressively fundraising international aid charities (usually members of the InterAction network) that conduct a huge array of different programs in different places.

On the other hand, it does not appear to link to this website anywhere from its main website - in fact, there appear to be only four external links to the site anywhere on the Web.

Looking through the evaluations provides an interesting example of what one of these “household name charities’” operations and impact evaluation look like. The variety of the projects and of the evaluations is huge. Some evaluations examine measures quite relevant to “impact,” such as reported behavior change and children vaccinated (example); others are looser, mentioning regional trends in disease burden but focusing on qualitative generalizations (example); others do not examine life outcomes at all, but simply make qualitative observations about strengths and weaknesses of the program evaluated (example). The quality and tone of the studies varies considerably as well. The use of “control groups” to assess impact is rare but occasional; none that we examined have what we consider to be a high level of rigor, but many appear encouragingly honest about program weaknesses as well as strengths.

Note that this set of evaluations appears to be far from comprehensive: CARE currently lists 845 active projects, whereas the database (which in some cases includes more than one evaluation per project, and goes back to 1991) contains only 448 evaluations as of this writing.

This isn’t the level of impact evidence that we see from our recommended charities, but some evaluation is better than no evaluation and non-publicized disclosure is better than no disclosure.

As a side note, CARE appears to be the only “household name charity” that turned down government funds during the debate over US-provided food aid. We aren’t sure whether they have the right side of this debate, but the turning away substantial money is unusual among charities, and suggests that CARE’s staff aren’t always putting fundraising first.

Bottom line: we’d recommend these charities over CARE, but we’d recommend CARE over other “household name” charities.

As we’ve written before, we tend - deliberately - not to focus on charities that are small and/or “experimental” in nature. From what we’ve seen, these charities rarely can demonstrate that their program has “worked” (in the sense of changing lives) before, and so the only way to evaluate them is to have a deep understanding of the environment they’re entering, the history of similar projects, etc. Our basic response to such charities is: “Get funding from people who are better positioned to evaluate whether this risk is worth taking; evaluate yourself over time; once you can demonstrate impact, a GiveWell recommendation becomes possible.”

However, there’s nothing about our requirements that’s incompatible with relatively small, innovation-focused charities, and VillageReach is a case in point. VillageReach is not a global conglomerate working toward universal coverage of tuberculosis drugs or insecticide-treated nets; it is aiming to transform health system logistics, and to date has completed one pilot project in one district of Mozambique. But it has emphasized monitoring and evaluation from the start, and its pilot project has impressive (if not fully conclusive) results. That distinguishes it from any other “transformative” or “experimental” charity we’ve seen.

In this case, we feel that enough information is available to show us - and individual donors - that this particular risk is worth taking. We feel that VillageReach’s approach has most likely “worked” (impressively and cost-effectively) in the first area where it was tried, and that future attempts are likely to be thoroughly evaluated as well.

There are important differences between VillageReach and the larger charities we recommend. As with any small/innovation-focused charity, giving to VillageReach is a high-risk, high-upside proposition, and we don’t mean our endorsement to indicate otherwise. To date it only has results from one project in one area, and future settings could differ in any number of ways. (The flip side of this risk is hope - hope that if a still-experimental approach proves to be repeatedly and demonstrably successful, it will impact the way other organizations operate, creating effects far beyond its own expenses.)

The choice between VillageReach and a larger charity like the Stop Tuberculosis Partnership is not a straightforward one. There are inherent concerns both with small charities (limited track record; no clear “pattern of success”) and with larger ones (so many activities in so many places that it’s hard to be confident in the organization as a whole), and neither of our top-rated charities has a case ironclad enough to dismiss these concerns. But we feel that on balance, both are extremely “good bets” (and better than any other we’re aware of) for a donor.

This post is based on a discussion of these issues on our public email list.