One of the aspects of our research process that has generated some objections is our use of “heuristics,” i.e., shortcuts to winnow the field of recommended charities from 300+ to a manageable number for closer investigation. The heuristics we use are described here. A good statement of the objections comes this comment at Hatrack forums:

I don’t care if a charity’s evaluation and monitoring reports are on their website, as long as they are publically available in some way. And while I agree with many of thier priorities, 75% of funding or more matching a list of specific programs is not vetting cost-effectiveness, it’s vetting whether or not the organization has the same priorities as Givewell does.

This post briefly explains and defends our approach. It does not discuss our criteria (proven, cost-effective, scalable, transparent), but rather the shortcuts we use to identify the charities most likely to meet those criteria.

The most important thing to know is that we are always ready to look at charities that don’t pass these heuristics, if they meet our broader criteria. If you know of such charities, please alert us using our submission form. If it appears that the information we require is available - whether or not it’s available on the charity’s website - we will change a charity’s status to “Pending” until we have reviewed it more thoroughly.

Why do we look at what information is available on a charity’s website, instead of searching more comprehensively and contacting them directly?

We have found that going back-and-forth with charities to see what they have internally is extremely difficult and time-consuming for both us and them. We are generally first connected to fundraising staff, and it takes a lot of communication and waiting just to end up talking to someone who knows what information is available. Repeating this process for all 300+ charities we have examined would not be practical, so we use a heuristic to identify the most promising candidates for further investigation.

We are explicit that our research is constrained by practical considerations. Our goal is not to be “perfect” in our assessments but rather to provide better information than donors can find anywhere else.

We do contact all rated charities to let them know about their status and how they can change it if they feel we are in error.

Is there independent evidence that “what information is on the website?” is a reasonable proxy for “what information is available at all?”

Yes. We have also used alternate research methods that involve much more back-and-forth with charities, and feel that the results give support to the “website scanning” heuristic as an imperfect but pretty good predictor of which charities actually have the information we require (particularly evidence of impact).

• Our first-year research process involved applications for grants of $25,000-$40,000. All non-confidential application materials have been publicly posted. For all five charities that earned a 2 star or better rating through this process, the primary evidence of impact we used is available on or via their website.
• We’re currently conducting a grant application process for $250,000 and will be publishing the full details of what it turns up in early 2010.

Also note that our “website scanning” heuristic is similar to the method used by William Easterly and Tobias Pfutze to rate aid agencies (PDF). Our aim is similar in that we seek to reward organizations that have both good practices and the transparency to share their practices publicly.

Do we require that charities be running “priority programs” in order to receive further investigation and/or high ratings?

No. The two heuristics we use are “or”, not “and.” We don’t require charities to share our program priorities. Rather, we investigate charities that do share these priorities even if they don’t pass the other heuristic. We do this because we have enough capacity to deeply investigate some “extra” charities, but not all 300+.

Why do we issue ratings to charities that don’t pass our heuristics, rather than simply marking them as “Not examined?”

We feel it would be misleading to simply say “not examined” for the charities that didn’t pass the heuristics. Given the constraints of what information is available and what’s practical, we feel strongly that there is a better case for the highly-rated charities than for the examined-but-not-rated charities. By contrast, a charity that doesn’t appear at all is one we simply haven’t looked at.

We feel it is accurate and important to call our top-rated charities the standouts (by our criteria) from a field of 300+.

It’s one thing to pay for children’s immunizations. It would be quite another to pay for a project that increased immunization rates over the long term, without continued donor support (either thanks to improvement in private-sector or government operations).

Aiming for the latter – or more broadly, aiming to use donations as “startup funds” for lasting and self-sustaining improvements (rather than as indefinite subsidies) – is often referred to as “sustainable development.” Pursuing “sustainability” is often very important to funders. (For example, the World Bank’s latest Independent Evaluation Group report aggregates figures on the likelihood of sustainability for all projects evaluated, and lists these aggregates as one of its key indicators - see Table A.4).

Yet while we would far prefer sustainable programs to unsustainable programs, we don’t currently list “sustainability” as one of our key criteria.

The reason is that while we’ve seen many charities aiming for sustainability, we’ve yet to see any that can demonstrate both future plans and past success in this area.

Sustainability is hard to come by

Most of our recommended international aid charities are aiming at least partially for lasting impact, but evidence that they’ve achieved it is murky at best. Two examples:

• We are extremely skeptical of Partners in Health’s plan to hand off funding to the Rwandan government.
• VillageReach’s logistics-focused program has already seen some discouraging signs from its handoff attempt.

Non-recommended organizations rarely appear to be documenting long-term impact of any kind.

Independent literature reinforces the concern that achieving sustainability is difficult and rare.

• A 2004 paper entitled “The Illusion of Sustainability” states, “Some anecdotal evidence suggests that sustainability has often been a chimera – and sometimes a costly one.”
• A 2008 paper by William Easterly adds, “This hope [for sustainable projects] has turned out to be an illusion, as the failure to cover recurrent costs has been nearly universal.”
• A book reviewing aid effectiveness (and generally positive on it) states that “The bulk of the NGO evaluation material which looks into the issue of sustainability has concluded that most NGO projects are not financially sustainable without the continued injection of external funds … Exceptionally few studies have provided a long-term view of [nonprofit] projects and programmes. Most views on the sustainability of NGO projects are based on forward extrapolation of short-term assessments” (281).

We do believe that sustainability in aid is possible, and has been achieved in the past. For example, Aravind Eye Care System appears to be a self-supporting humanitarian organization, using revenue from paying customers to perform cataract surgery on those with lower incomes. This program, in fact, appears so sustainable that there is little donors can add to it, and donations support tangentially related activities. (More broadly, we were recently pointed to family planning as an example of an area where aid has been associated with lasting behavior change.)

But as with impact, we believe that sustainability needs to be demonstrated, not just claimed. In fact, the burden of proof for demonstrating sustainability is even higher than for demonstrating impact.

The importance of long-term evaluation

We constantly stress the importance of systematic, long-term impact evaluation, and note how rarely it is seen in the world of charity. It seems to us that if one is pursuing sustainable impact, the challenge and importance of evaluation become more significant. In order to assess sustainability, a charity must be systematically examining an area (with all the challenges that involves) even after it has withdrawn support.

VillageReach appears to be doing exactly this, and is therefore at least in a position to identify and respond to disappointing results (according to VillageReach representatives, the local government appears to have lapsed back into its former, less efficient logistics approach, but has agreed to renew its focus on sustaining the new model).

In our view, any project that does not include rigorous evaluation can’t make any credible claim that it has achieved sustainability – only that it hopes to.

A desirable goal, but not a reasonable requirement

We think it may often be wise for experts and major funders to focus on sustainability – doing so could make sense even with a high failure rate, because successes are so significant.

But in trying to help individual donors – who aren’t well positioned to innovate – we focus on finding programs that have worked before and are likely to work again. If we required recommended charities to show evidence of “sustainability,” we simply would have no recommended charities, at least at this point. As such, we’ve evaluated our strongest charities by what they can accomplish even assuming that their hopes of sustainability don’t pan out. We recommend VillageReach, despite concerns about sustainability, because its program appears to do a lot of good for relatively little money even if its improvements are never maintained (more here).

If VillageReach’s program does end up having a lasting impact, it will be an even better deal. If we later find a charity that can convincingly demonstrate an ability (not just a plan) to create sustainable and demonstrable impact, we will recommend it strongly. But in the meantime, it doesn’t seem reasonable to criticize a charity on the grounds that its work isn’t sustainable. Simply having real, demonstrable impact is hard enough.

One of the consistent refrains we’ve seen in aid literature is the importance of local participation/enthusiasm/ownership for aid projects. Many programs have been criticized for being too “top-down” (i.e., imposing outsiders’ designs on local communities), with the implication that more “bottom-up” programs (i.e., getting local people to participate in the design of execution of programs) would be more likely toi create real and lasting change. For an example of this reasoning, see this USAID review of Integrated Rural Development programs (PDF).

The basic reasoning makes sense, but making a program “bottom-up” is easier said than done. For an illustration of why, see this World Bank review of “community-based development,” a term referring to “projects that actively include beneficiaries in their design and management.”

The frequent tendency for participatory projects to be dominated if not captured by local elites is highlighted by several case studies. Katz and Sara (1997), in a global review of water projects, find numerous cases of project benefits being appropriated by community leaders and little attempt to include households at any stage … even well trained staff are not always effective in overcoming entrenched norms of exclusion. In a study of community forestry projects in India and Nepal that worked reasonably well, Agarwal (2001) reports that women were systematically excluded from the participatory process because of their weak bargaining power. Rao and Ibanez (2003) find that in the participatory projects in their Jamaican case study, wealthier and better networked individuals dominated decision making. In a similar case-based evaluation of social funds in Jamaica, Malawi, Nicaragua, and Zambia, the World Bank (2002) Operations Evaluation Department concludes that the process was dominated by “prime movers.”

Abraham and Platteau (2004) present evidence on community participation processes in Sub-Saharan Africa based largely on anecdotal evidence from their work in community-based development and on secondary sources. They argue that rural African communities are often dominated by dictatorial leaders who can shape the participation process to benefit themselves because of the poor flow of information. (40-41)

These notes capture a concern of ours that applies to all aid projects: while the goal is to help those in the most need, those with the least need may be most likely to have the resources, connections and free time to get the inside track on any particularly generous aid project. This is also a major reason to be skeptical of simple evaluations comparing “project participants” to “non-participants,” as many microfinance evaluations do. Project participants may simply be better off to begin with (and some studies show that they are, such as the Coleman study referenced on the previous link).

We don’t believe that a simple and straightforward way to overcome this challenge is available. That’s why, although we agree with the basic concept that local ownership will improve a project, we don’t tend to judge projects by their formal commitment to local ownership - i.e., we don’t favor programs that work in formal community votes, meetings, etc. over programs that don’t. The former could be improving local participation or transferring more power to elites; the latter could be generating local enthusiasm simply through a good match between what people want and what they’re being offered.

It’s easy to claim that one is involving community members, but the ultimate test is in outcomes - whether the project ran well enough and generated enough local participation to accomplish its ultimate goals (improved health, incomes, etc.)

Our reviews have a tendency to discount stories of individuals, in favor of quantitative evidence about measurable outcomes. There is a reason for this, and it’s not that we only value quantitative evidence - it’s that (in our experience) qualitative evidence is almost never provided in a systematic and transparent way.

If a charity selected 100 of its clients in a reasonable and transparent way, asked them all the same set of open-ended questions, and published their unedited answers in a single booklet, I would find this booklet to be extremely valuable information about their impact. The problem is that from what we’ve seen, what charities call “qualitative evidence” almost never takes this form - instead, charities share a small number of stories without being clear about how these stories were selected, which implies to me that charities select the best and most favorable stories from among the many stories they could be telling. (Examples: Heifer International, Grameen Foundation, nearly any major charity’s annual report.)

A semi-exception is the Interplast Blog, which, while selective rather than systematic in what it includes, has such a constant flow of stories that I feel it has assisted my understanding of Interplast’s activities. (Our review of Interplast is here.)

I don’t see many blogs like this one, and I can’t think of a particularly good reason why that should be the case. A charity that was clear, systematic and transparent before-the-fact about which videos, pictures and stories it intended to capture (or that simply posted so many of them as to partly alleviate concerns about selection) would likely be providing meaningful evidence. If I could (virtually) look at five random clients and see their lives following the same pattern as the carefully selected “success stories” I hear, I’d be quite impressed.

But this sort of evidence seems to be even more rare than quantitative studies, which are at least clear about how data was collected and selected.