Recently, we’ve criticized (in one way or another) many well-known, presumably well-intentioned charities (Smile Train, Acumen Fund, UNICEF, Kiva), which might lead some to ask: should GiveWell focus on the bad (which may discourage donors from giving) as opposed to the good (which would encourage them to give more)? Why so much negativity and not more optimism?

The fact is, we are very optimistic about what a donor can accomplish with their charity. Donors can have huge impact — save a life, improve equality of opportunity, or improve education. Our research process, and our main website, are (and always have been) built around identifying outstanding charities.

GiveWell hasn’t set a bar that no charity can meet. Six international charities have met and passed the bar. Where most charities fall short, they succeed.

The problem is: because the nonprofit sector is saturated with unsubstantiated claims of impact and cost-effectiveness, it’s easy to ignore me when I tell you (for example), “Give $1,000 to the Stop Tuberculosis Partnership, and you’ll likely save someone’s life (perhaps 2 or 3 lives).” It’s easy to respond, “You’re just a cheerleader” or “Why give there when Charity X makes an [illusory] promise of even better impact?”

We don’t report on Smile Train, Kiva, Acumen Fund, UNICEF, or any others for the sake of the criticism; we write about them to show you how much more you can accomplish with your gift if you’re willing to reconsider where you’re giving this year.

Unless you have strong reason to believe otherwise, I’d recommend you choose a great charity as opposed to one that’s merely better-than average. If you only have a fixed amount to give, why not support the very best?

One of the aspects of our research process that has generated some objections is our use of “heuristics,” i.e., shortcuts to winnow the field of recommended charities from 300+ to a manageable number for closer investigation. The heuristics we use are described here. A good statement of the objections comes this comment at Hatrack forums:

I don’t care if a charity’s evaluation and monitoring reports are on their website, as long as they are publically available in some way. And while I agree with many of thier priorities, 75% of funding or more matching a list of specific programs is not vetting cost-effectiveness, it’s vetting whether or not the organization has the same priorities as Givewell does.

This post briefly explains and defends our approach. It does not discuss our criteria (proven, cost-effective, scalable, transparent), but rather the shortcuts we use to identify the charities most likely to meet those criteria.

The most important thing to know is that we are always ready to look at charities that don’t pass these heuristics, if they meet our broader criteria. If you know of such charities, please alert us using our submission form. If it appears that the information we require is available - whether or not it’s available on the charity’s website - we will change a charity’s status to “Pending” until we have reviewed it more thoroughly.

Why do we look at what information is available on a charity’s website, instead of searching more comprehensively and contacting them directly?

We have found that going back-and-forth with charities to see what they have internally is extremely difficult and time-consuming for both us and them. We are generally first connected to fundraising staff, and it takes a lot of communication and waiting just to end up talking to someone who knows what information is available. Repeating this process for all 300+ charities we have examined would not be practical, so we use a heuristic to identify the most promising candidates for further investigation.

We are explicit that our research is constrained by practical considerations. Our goal is not to be “perfect” in our assessments but rather to provide better information than donors can find anywhere else.

We do contact all rated charities to let them know about their status and how they can change it if they feel we are in error.

Is there independent evidence that “what information is on the website?” is a reasonable proxy for “what information is available at all?”

Yes. We have also used alternate research methods that involve much more back-and-forth with charities, and feel that the results give support to the “website scanning” heuristic as an imperfect but pretty good predictor of which charities actually have the information we require (particularly evidence of impact).

• Our first-year research process involved applications for grants of $25,000-$40,000. All non-confidential application materials have been publicly posted. For all five charities that earned a 2 star or better rating through this process, the primary evidence of impact we used is available on or via their website.
• We’re currently conducting a grant application process for $250,000 and will be publishing the full details of what it turns up in early 2010.

Also note that our “website scanning” heuristic is similar to the method used by William Easterly and Tobias Pfutze to rate aid agencies (PDF). Our aim is similar in that we seek to reward organizations that have both good practices and the transparency to share their practices publicly.

Do we require that charities be running “priority programs” in order to receive further investigation and/or high ratings?

No. The two heuristics we use are “or”, not “and.” We don’t require charities to share our program priorities. Rather, we investigate charities that do share these priorities even if they don’t pass the other heuristic. We do this because we have enough capacity to deeply investigate some “extra” charities, but not all 300+.

Why do we issue ratings to charities that don’t pass our heuristics, rather than simply marking them as “Not examined?”

We feel it would be misleading to simply say “not examined” for the charities that didn’t pass the heuristics. Given the constraints of what information is available and what’s practical, we feel strongly that there is a better case for the highly-rated charities than for the examined-but-not-rated charities. By contrast, a charity that doesn’t appear at all is one we simply haven’t looked at.

We feel it is accurate and important to call our top-rated charities the standouts (by our criteria) from a field of 300+.

Following up on Thursday’s Alliance for Effective Social Investing meeting, Sean at Tactical Philanthropy writes:

A high performance nonprofit is a very well run organization. It has outstanding leadership, clear goals, an ethic of monitoring performance and making adjustments as needed, and it is financially healthy.

A high impact nonprofit is one whose efforts have been proven to cause sustainable, positive change.

Impact can be seen only in retrospect. Often many years later. Performance can be directly observed.

I think high impact nonprofits are the holy grail of philanthropy. But like any holy grail, it is something to journey towards, not something you demand now.

Sean goes on to argue that funders should put more focus on “high-performing,” as opposed to “high-impact,” nonprofits. At GiveWell, we focus on “high-impact” nonprofits, in that we look for evidence of past impact and not just future promise. Our response to Sean:

1. Assessing “high-performance” is much harder than assessing “high-impact.” This isn’t to say that either is easy. But we feel it’s very doable for charities to take the “form” of a “high-performance” nonprofit - collecting large amounts of data, executing activities competently, and describing those activities in a compelling and money-raising way - without actually being on a path toward impact (which requires that the data be the right data and that the activities be the right activities for the goal).

We see many charities with impressive-looking evaluation systems; far fewer with actual past outcomes to show. If anything, this makes us suspect that other funders are looking for the form and appearance of good evaluation, without holding charities accountable for actual results.

2. Because of this, funding “high-performance nonprofits” is not something that casual donors (as opposed to subject matter experts) are well positioned to do. This point parallels our argument that casual donors aren’t well positioned to fund the unproven and innovative. Like funding a small and unproven charity, funding a “high-performance” but not “high-impact” charity means trying to do something that hasn’t been done before, and introduces a greater need for understanding the full context of a program.

3. “High-impact” nonprofits might be rare - but that doesn’t make them overfunded. We believe that our top-rated charities can productively use more funds than they’re currently getting. As long as that’s the case, why should a casual donor give to a charity without past impact rather than a charity with past impact?

Our reviews have a tendency to discount stories of individuals, in favor of quantitative evidence about measurable outcomes. There is a reason for this, and it’s not that we only value quantitative evidence - it’s that (in our experience) qualitative evidence is almost never provided in a systematic and transparent way.

If a charity selected 100 of its clients in a reasonable and transparent way, asked them all the same set of open-ended questions, and published their unedited answers in a single booklet, I would find this booklet to be extremely valuable information about their impact. The problem is that from what we’ve seen, what charities call “qualitative evidence” almost never takes this form - instead, charities share a small number of stories without being clear about how these stories were selected, which implies to me that charities select the best and most favorable stories from among the many stories they could be telling. (Examples: Heifer International, Grameen Foundation, nearly any major charity’s annual report.)

A semi-exception is the Interplast Blog, which, while selective rather than systematic in what it includes, has such a constant flow of stories that I feel it has assisted my understanding of Interplast’s activities. (Our review of Interplast is here.)

I don’t see many blogs like this one, and I can’t think of a particularly good reason why that should be the case. A charity that was clear, systematic and transparent before-the-fact about which videos, pictures and stories it intended to capture (or that simply posted so many of them as to partly alleviate concerns about selection) would likely be providing meaningful evidence. If I could (virtually) look at five random clients and see their lives following the same pattern as the carefully selected “success stories” I hear, I’d be quite impressed.

But this sort of evidence seems to be even more rare than quantitative studies, which are at least clear about how data was collected and selected.