VillageReach currently holds the top slot in our top charities list. While I will be donating to the Stop Tuberculosis partnership (discussed yesterday), Elie’s donation this year went to VillageReach.

VillageReach consults on health system logistics in high-poverty, remote areas to help life-saving supplies get to those who need them. It may not be the most attention-grabbing story (”improving operational efficiency” doesn’t have the same ring to it as giving a cow or fixing a smile), but

• It’s directly addressing a key issue - limited health system capacity - that may limit the amount of good donations to any health-related cause are able to do.
• Immunizations in sub-Saharan Africa are a top global health priority (ranked #4 by the Copenhagen Consensus, for example and have been estimated to avert infant deaths for ~$200 each (estimate includes all costs).

In other words, VillageReach would be accomplishing tremendous good if it could improve logistics to deliver more vaccines. Can it?

Pilot project

VillageReach assessed its pilot project in Mozambique (Cabo Delgado province) with more thoroughness than we’ve seen almost anywhere outside of JPAL and IPA. It is a great demonstration that just because a charity is relatively small, and experimental, doesn’t mean it can’t collect substantive and compelling evidence regarding its impact.

VillageReach collected (and has now published) data on trends in vaccine/supply deliveries, “stock-outs” of vaccines (the single most compelling indicator, in our view) and vaccination coverage rates. It also conducted vaccination coverage surveys in both the province where it worked and a nearby province for comparison. After examining this evidence along with some independent analysis with our own, we’ve concluded that the case for impact (despite not including a randomized controlled trial) is quite compelling: VillageReach caused a significant jump in vaccination rates, which we believe can be fairly extrapolated to lives saved at well under $1000 each.

VillageReach hoped to transfer responsibility for the modified logistics system (which has a dedicated team responsible for delivering supplies, rather than each clinic responsible for its own collection) to the government, but available evidence suggests that the transfer failed. The fact that vaccination rates have since fallen is further evidence that VillageReach made a difference while they were there, but obviously discouraging relative to what they had hoped for. We have expressed doubts in the past about donors’ getting their hopes up for “sustainability” in aid, and think it’s worth noting that even VillageReach’s project – with its strong case for impact as well as a case that it actually would save the government money – has so far failed at the ambitious goal of changing government practices.

More details at the “Does it work?” section of our review.

The future of VillageReach

There are a lot of questions about the future of VillageReach. As a small organization, it sees frequent changes in its plans and engagements. It has taken on a variety of consulting projects, some of which we have little information about and could be significantly different from where it has succeeded before.

However, it has provided an unusual amount of clarity and specificity about the likely use of additional donations. We feel that VillageReach can productively absorb up to $2.5 million in unrestricted donations over the next year and that these funds will be used for work very similar to its pilot project: reactivating its program in the original province, in addition to possibly expanding to one or more neighboring provinces.

Pros and cons

VillageReach is very different from the “ideal charity” we had in mind early in our international aid research. (Our “ideal charity” is much closer to the charity I discussed yesterday, the Stop Tuberculosis Partnership). It is relatively small and still represents something of an experiment, which has both pros and cons.

VillageReach is a small charity that has struggled at times to raise enough funding, not a large charity built to absorb huge sums of money.

• This adds a certain kind of uncertainty to its situation, as its plans hinge heavily on how fundraising goes as well as other factors (such as government cooperation) outside its control.
• But being small also comes with advantages for a donor. The need for more funding is extremely concrete and some extra funding could make a huge difference to VillageReach’s ability to perform up to its potential. In addition, it is feasible – if difficult – to have a complete view of VillageReach’s work. Where a large charity like Stop TB Partnership relies heavily on audits and spot-checks – leaving open a lot of room for unobservable error – VillageReach is measuring, and accountable for, the results of everything it does.

VillageReach has only one demonstrable success to its credit. It is still establishing the potential of its model.

• VillageReach is “experimental” in a sense. It doesn’t have the same consistency, accomplishing similar results in many times and places, as Stop TB Partnership.
• However, we think this is a very different level of “experiment” from an organization with no demonstrated impact. We think there is enough information about VillageReach’s past impact for an individual donor to be confident that its future activities are a “good bet.”
• And with the risk of an “experiment” comes upside. If VillageReach continues to demonstrate success, it could affect the way health logistics works all over the world, through the spread of information rather than through its own funds. The problem we have with most “experiments” in international aid is that we aren’t convinced that they will produce the information needed to (a) determine whether they’ve succeeded and (b) bring about widespread adoption if they have. By contrast, VillageReach has demonstrated a willingness and ability to produce credible evidence of impact.

VillageReach does not fully share GiveWell’s priorities.

• We are most interested in “doing more of something that demonstrably improves lives.” VillageReach has expressed great interest – and allocated funds to - the more ambitious goal of “advocacy” (broadly defined), changing the way governments and larger funders operate.
• Yet even as it aims for these more ambitious goals, we believe that VillageReach is in fact saving lives at a level of cost-effectiveness that rivals that of any other charity. So we see its more ambitious goals almost as a “bonus” – if it succeeds at them, we’ll have helped fund a truly incredible level of impact, and if it fails at them, we still expect to have gotten our “money’s worth” (lives saved for under $1000 each).

Bottom line

The pros and cons listed above are largely mirror images of each other. Stop TB Partnership is an outstanding “large charity” – with both the advantages and concerns that accompany great size – and VillageReach is an outstanding “small charity,” with a different set of advantages and concerns. Both give good answers to all our major questions, leaving only the kind of concerns that are likely inevitable with charities of their size.

Which you have more confidence in, and prefer to donate to, is a judgment call. We have ranked VillageReach as our top charity because Elie and I happen to agree that it seems like a “better bet” overall (to approach the impact implied by the academic cost-effectiveness estimates) – we think the ability to hold it fully accountable outweighs the risks that come with being small and experimental.

Elie and I do part ways when it comes to the ultimate donation decision. I am giving to Stop TB Partnership for the personal values-related reasons discussed yesterday. Elie doesn’t share the values I mentioned and has donated to VillageReach.

Ever since I heard about the Grameen Foundation’s Village Phone program, I’ve been optimistic. The program involves helping people in remote villages run pay-for-use cellphone services: they get their cellphone, and a loan to buy it, via Grameen, then charge other villagers to use it. It’s an approach to fighting poverty that’s (a) relatively new; (b) using a product that hasn’t been available for a long time but seems clearly useful to anyone doing business in remote areas; (c) utilizing a “franchise model” where people in the villages take a stake in the product.

It was near the top of my “Probably helping people, even though we don’t yet have systematic evaluation yet” list. Now Chris Blattman points to a discouraging evaluation that found “absolutely no impact of the phones on trading activity or availability of goods in local markets” and very small (non-significant) impacts on profits and measures of well-being (school enrollment, consumption of meat, etc.)

This bottom-line result does not, by itself, mean the program “doesn’t work.” It could work very differently in different contexts (discussed below), and there are some possible issues with the paper (which is very recent, and is not a randomized controlled trial). But one thing I like about the study is that it doesn’t just discuss impact - it examines many aspects of the program, and exposes assumptions that may otherwise have gone unquestioned.

Assumption 1: the phones are in high demand and operators easily cover their costs. In fact, usage of the phones was around 4 hours a month, or 8 minutes a day (pg 19). As a result, profits from the phones were not enough to keep up with loan payments (pg 19). Grameen reportedly has responded by changing loan and franchise terms (pg 30). Tuvugane (pg 5), a less sophisticated phone product that was already common in the villages, may have been good enough for most purposes.

Assumption 2: farmers who use the phones benefit from better pricing power. Even though farmers with access to the phones became much more likely to arrange their own transport to market, there was no apparent effect on the prices they received for their goods, possibly due to established relationships with buyers (pg 16).

Assumption 3: if someone chooses to become a cellphone operator, they’re going to benefit from it. In fact, there was a very strange pattern in the businesses of people who became phone operators. Their hours worked rose significantly both for their new phone business and for their already-existing businesses, but their profits and wages paid did not rise (pgs 17-18). A possible explanation is that operators wanted to be available for cellphone users and so stayed at their workplaces longer, but that the extra hours didn’t translate into extra profits. In any case, it’s a pattern that doesn’t seem encouraging, and seems to deserve further investigation.

Bottom line: a product that was supposed to be helpful and in high demand arguably ended up as a bad investment for the franchise operators. This doesn’t mean it shouldn’t have been tried, or that it shouldn’t be tried in the future. But it points to the importance of testing assumptions empirically, rather than scaling up a program as widely as possible based on an appealing story.

Sean at Tactical Philanthropy has continued his discussion of “high-performing” vs. “high-impact” organizations, which we previously commented on. The message he is sending (see posts here and here) is partly that we need to take the emphasis off of “funding organizations that have shown results” and put it on “funding organizations that seem ‘on the way’ to proving results.”

I believe there is a place for funders who invest as Sean advocates. However, I think that when taken too far, the idea of rewarding charities for being “on the way” is damaging - and the idea is currently being taken too far.

As we’ve written before, our experience is that there are far more nonprofits with impressive evaluation processes and evaluation plans than there are nonprofits with impressive evaluation results. The ratio is so out of whack that it actually appears to be systematic, not an accident of timing.

When you see - as Sean does - that “very, very few nonprofits have ever gone through extensive analysis that has proven that their programs have impact,” you can react in one of two ways. You can hold up those few as the best targets for more funds (especially from casual donors), or you can decide that the “high-impact” bar is too high altogether. The problem with the latter approach - at least when too many funders take it - is that there are no financial incentives for charities to show actual results, as opposed to showing impressive processes and plans.

We believe that what gets rewarded is what gets done. We hope to reward proven impact, leading to more proven impact. We believe that rewarding promises will lead to more promises.

There is also a place for funders who reward the nonprofits that are “on the way” - as Sean observes, without such funding no nonprofits could even get off the ground and become high-impact. But someone has to save their donations for the charities that have actually gotten results - and for reasons we outlined before, we think that someone can and should be individual donors.

A couple of other observations on this discussion:

• It’s refreshing to see widespread acknowledgement that “high-impact nonprofits” - nonprofits that can truly demonstrate past success - are incredibly rare. It’s worth keeping in mind next time you are confronted with traditional nonprofit marketing.
• Sean believes that identifying high-performance nonprofits can be easy. We disagree, but rather than getting into a theoretical debate, we prefer that Sean (or someone else) try to apply the proposed method to actual charities, and make recommendations for giving within certain causes. At that point it should be easier to assess how viable this approach is.

Education used to be my favorite cause. My enthusiasm waned as I saw both the cost-effectiveness of international aid and the apparent futility of education. (Elie’s 2007 post captures many of my thoughts.) The study that I’ve been blogging about today (here and here) provides a firmer grounding for our optimism about high-intensity charter schools, and challenges the idea that there aren’t good opportunities for donors in education.

However, I’m still not ready to prioritize education again, personally. One of the things that surprised me most in studying education was not just the difficulty in finding programs that could improve academic performance, but also the complete lack of rigorous evidence that education is key to later life outcomes. I would be fascinated to see a rigorous study of how the students who benefit from excellent charter schools perform later in life - in terms of income, job satisfaction, criminal records, etc. Without evidence, I’m not convinced that raising a child’s math score raises their life prospects, especially in a way that goes beyond “signaling” (i.e., allowing them to outcompete other people due to a superior credentials).

Would putting every child in America in a good school that makes sure they can do math lead to a much better society? I used to assume it would; I’m no longer so sure, and recent information doesn’t change that.

For now, I’m going to wait and see. I’d like to see the academic reaction the Fryer and Dobbie paper on the Harlem Chilren’s Zone. If others agree about the rigor and significance of its findings, I’d like to see who steps forward to continue replicating and examining this approach. The Social Innovation Fund would seem to be one strong candidate.

In the meantime, I’m going to be putting my own money into programs that are proven and replicable and don’t have enough funding - things like tuberculosis control and distribution of insecticide-treated nets.

Perhaps, at some point, I will feel that there is an education program that meets all three of these criteria as well. At that point I may start giving to it, even if it’s many times as expensive per person as developing-world aid.

Related posts:

• Fryer and Dobbie on Harlem Children’s Zone: what they found
• Fryer and Dobbie on the Harlem Children’s Zone: significance
• Followup on Fryer/Dobbie study of “Harlem miracle”

Imagine that someone came to you with an idea for a startup business and offered you a chance to invest in it. Which of the following would you require before taking the plunge?

• Familiarity with (or at least a lot of information about) the people behind the project
• Very strong knowledge of the project’s “space” (understanding of any relevant technologies, who the potential customers might be, etc.)
• As much information as possible about similar projects, both past and present

Unless you’re an unusually adventurous investor, you probably answered with “All of the above.” After all, there’s a risk of losing your investment – and unlike with established businesses (which have demonstrated at least some track record of outdoing the competition), here your default assumption should be that that’s exactly what will happen.

Now what is the difference between this situation and giving to a startup charity?

One difference is that with a charity, you know from the beginning that you won’t be getting your donation back. But this doesn’t mean there isn’t risk – the risk just takes a different form. Presumably, your goal in donating to a charity is to improve the world as much as possible. If the startup charity you help get off the ground ends up being much less impactful (on a per-dollar basis) than established charities, then your support was a mistake. If it ends up having no meaningful impact, you’ve lost your shirt.

And in my opinion, the worst case possible is that it succeeds financially but not programmatically – that with your help, it builds a community of donors that connect with it emotionally but don’t hold it accountable for impact. It then goes on to exist for years, even decades, without either making a difference or truly investigating whether it’s making a difference. It eats up money and human capital that could have saved lives in another organization’s hands.

As a donor, you have to consider this a disaster that has no true analogue in the for-profit world. I believe that such a disaster is a very common outcome, judging simply by the large number of charities that go for years without ever even appearing to investigate their impact. I believe you should consider such a disaster to be the default outcome for an new, untested charity, unless you have very strong reasons to believe that this one will be exceptional.

So when would I consider appropriate for a donor to invest in a small, unproven charity? I would argue that all of the following should be the case:

1. The donor has significant experience with, and/or knowledge regarding, the nonprofit’s client base and the area within which it’s working. For example, a funder of a new education charity should be familiar with the publicly available literature on education, as well as with the specific school system (and regulations) within which the project is working. A funder of a project in Africa should be familiar with past successes and failure in international aid in general, and should spend time in the area where the project will be taking place.
2. The donor has reviewed whatever information is available about past similar projects and about the assumptions underlying this project. If similar, past projects have failed, the donor has a clear sense of why they failed and what about the current project may overcome those obstacles.
3. The donor has a good deal of confidence in the people running the nonprofit, either because s/he know them personally or because s/he has an excellent sense for their previous work and past accomplishments. (Enough confidence in the people can lower the need for the above two points, to some extent.)
4. The donor feels that the organization is doing whatever it reasonably can to measure its own impact over time. The donor is confident that– within a reasonable time frame – if the project succeeds, it will be able to prove its success; if it fails, it will see this and it will fold. Until impact is demonstrated, there is no need for the kind of scale that comes with taking many donations from casual donors. As stated above, I believe that the overwhelming majority of charities do not meet this criterion.

If you know a lot about cars, you might try to build your own car. But if you don’t, you’re much better off with a name brand. Likewise, casual donors are better off funding charities that have track records; experimental charities should start small and accumulate track records. This is why we are comfortable with our bias toward larger charities.