The GiveWell Blog

Our process: Narrowing the field

(updated on: ) | by Holden

One of the aspects of our research process that has generated some objections is our use of “heuristics,” i.e., shortcuts to winnow the field of recommended charities from 300+ to a manageable number for closer investigation. The heuristics we use are described here. A good statement of the objections comes this comment at Hatrack forums:

I don’t care if a charity’s evaluation and monitoring reports are on their website, as long as they are publically available in some way. And while I agree with many of thier priorities, 75% of funding or more matching a list of specific programs is not vetting cost-effectiveness, it’s vetting whether or not the organization has the same priorities as Givewell does.

This post briefly explains and defends our approach. It does not discuss our criteria (proven, cost-effective, scalable, transparent), but rather the shortcuts we use to identify the charities most likely to meet those criteria.

The most important thing to know is that we are always ready to look at charities that don’t pass these heuristics, if they meet our broader criteria. If you know of such charities, please alert us using our submission form. If it appears that the information we require is available – whether or not it’s available on the charity’s website – we will change a charity’s status to “Pending” until we have reviewed it more thoroughly.

Why do we look at what information is available on a charity’s website, instead of searching more comprehensively and contacting them directly?

We have found that going back-and-forth with charities to see what they have internally is extremely difficult and time-consuming for both us and them. We are generally first connected to fundraising staff, and it takes a lot of communication and waiting just to end up talking to someone who knows what information is available. Repeating this process for all 300+ charities we have examined would not be practical, so we use a heuristic to identify the most promising candidates for further investigation.

We are explicit that our research is constrained by practical considerations. Our goal is not to be “perfect” in our assessments but rather to provide better information than donors can find anywhere else.

We do contact all rated charities to let them know about their status and how they can change it if they feel we are in error.

Is there independent evidence that “what information is on the website?” is a reasonable proxy for “what information is available at all?”

Yes. We have also used alternate research methods that involve much more back-and-forth with charities, and feel that the results give support to the “website scanning” heuristic as an imperfect but pretty good predictor of which charities actually have the information we require (particularly evidence of impact).

  • Our first-year research process involved applications for grants of $25,000-$40,000. All non-confidential application materials have been publicly posted. For all five charities that earned a 2 star or better rating through this process, the primary evidence of impact we used is available on or via their website.
  • We’re currently conducting a grant application process for $250,000 and will be publishing the full details of what it turns up in early 2010.

Also note that our “website scanning” heuristic is similar to the method used by William Easterly and Tobias Pfutze to rate aid agencies (PDF). Our aim is similar in that we seek to reward organizations that have both good practices and the transparency to share their practices publicly.

Do we require that charities be running “priority programs” in order to receive further investigation and/or high ratings?

No. The two heuristics we use are “or”, not “and.” We don’t require charities to share our program priorities. Rather, we investigate charities that do share these priorities even if they don’t pass the other heuristic. We do this because we have enough capacity to deeply investigate some “extra” charities, but not all 300+.

Why do we issue ratings to charities that don’t pass our heuristics, rather than simply marking them as “Not examined?”

We feel it would be misleading to simply say “not examined” for the charities that didn’t pass the heuristics. Given the constraints of what information is available and what’s practical, we feel strongly that there is a better case for the highly-rated charities than for the examined-but-not-rated charities. By contrast, a charity that doesn’t appear at all is one we simply haven’t looked at.

We feel it is accurate and important to call our top-rated charities the standouts (by our criteria) from a field of 300+.

Donor illusions

(updated on: ) | by Holden

There’s an persistent conflict in international charity:

  • It feels great to be able to say, “My donation helped THIS person.”
  • But it’s rarely – if ever – practical for that sort of connection to be real.

As a result, international charities tend to create “donor illusions” by implying that donations can be attributed more tangibly, reliably and specifically than they really are. Some charities are more purposefully misleading than others, and some have more prominent and clear disclosures than others, but we feel that all of the cases below end up misleading many donors.

Kiva.org: making a loan
Illusion: You pick a developing-world entrepreneur who needs a loan. You lend money to that entrepreneur, interest-free. With you, s/he wouldn’t have gotten the loan.
Reality: The entrepreneur you’re viewing in your browser probably already got his/her loan and is probably paying significant interest on it. What you’re really doing is sending money to a microfinance institution that uses it as it sees fit (including for loans to less creditworthy people).

Details at GiveWell Board member Tim Ogden’s summary of the recent Kiva debate.

Child sponsorship: supporting a child

Illusion: through an organization such as Save the Children, your money supports a specific child.
Reality: as Save the Children now discloses, “Your sponsorship contributions are not given directly to a child. Instead, your contributions are pooled with those of other sponsors to provide community-based programming for all eligible children in the area.” See this David Roodman post (starting with “The Kiva Story”) for the interesting, scandal-ridden history of this practice.

Heifer International: giving livestock

Illusion: your donation pays for a cow for a specific developing-world family, helping it earn a better living.
Reality: as the fine print says, “Gifts made through this catalog represent a gift to the entire mission.” The entire mission generally includes a lot beyond livestock, including difficult projects like rural extension services.

Other donor illusions are more subtle. More on this in a future post.

Evaluating microfinance charities

(updated on: ) | by Holden

When we think about microfinance, we don’t ask “which person” or “which story” to fund; we think about which organization to fund.

As explained at our discussion of microfinance myths, we don’t think the traditional story donors are told is accurate. We do think microfinance could be helping people in other ways – or hurting them. Here are the questions we’re asking as we consider granting one or more microfinance charities:

1. Are the customers getting “handouts” or “services”?

Even if an microfinance charity can’t directly show changes in standards of living, it seems like a good sign of “empowerment” if people are choosing to participate in a program that has concrete costs (i.e., interest).

On the other hand, a microfinance charity that’s heavily subsidizing loans could be essentially giving out cash, at which point the mere fact of participation becomes less meaningful, and it becomes more important to ask whom the handouts are going to.

Some ways to get at this question:

  1. Is the charity creating self-sustaining institutions? If so, it is (a) multipling the impact of donations; (b) ultimately creating services that people are willing to pay for.
  2. If not, is the charity seeing high participation (per dollar of operating expenses) with high repayment rates and reasonable interest rates? Together, these seem to indicate “real participation”; an operation with high default rates (10%+ per year) or depressed interest rates (relative to, for example, local bank rates) is closer to giving out cash. Note that the “headline” repayment rate is not necessarily the right one to look at.

2. Who are the customers?

This is an especially important question if the answers to the above questions are “no” (implying that the charity is making gifts more than providing services). If a charity is giving out gifts, we want strong evidence that those gifts are going to those in need.

As a side note, many charities argue that low average loan sizes prove that they are serving the very poor; we don’t find this convincing.

3. Are customers better or worse off for participating?

As we wrote in Microloans vs. Payday Loans, we aren’t convinced that mere borrowing and repayment means positive impact.

Impact studies would be ideal, but based on what we’ve seen, we doubt that we will be able to find a microfinance charity that provides strong impact studies. That leaves us with measures of the general popularity/client satisfaction of programs, which won’t tell the full story (as in the case of payday loans).

  1. Data on dropouts. How many people each year drop out of the program? Are they dropping out for positive reasons (like now having the creditworthiness to get traditional loans) or negative ones (failing to benefit from costly loans; paying back under undue pressure)?
  2. Trends in participants per branch. If impact studies and good data on dropouts aren’t available (and they often aren’t), we may try this measure as a proxy for the general “popularity” of services. The idea is that growth in total clients may simply be due to aggressive marketing and geographic expansion, but if more clients continue to come in in the same area, the product is probably popular and people probably perceive themselves to be benefiting. We’d like to see that most branches have positive growth and that many have significant (10%+) growth in clients.
  3. Monitoring of client satisfaction, client overindebtedness, harassment by loan officers, etc. To be confident that meaningful monitoring is happening, we’d want examples of and figures on actual complaints filed and actions taken, not just a manual laying out how monitoring is supposed to happen.

Beyond loans

This post has focused on “microfinance” as most people know it, i.e., loans. However, we find “microsavings” more promising in some ways, and have a different set of questions to assess microsavings programs. More in a future post.

How would a malaria vaccine affect charity?

(updated on: ) | by Holden

What happens if a malaria vaccine becomes available – and is far more effective than medicine and perhaps insecticide-treated nets?

I hope that all malaria charities will transition, as smoothly as they can, to assisting with immunization programs. But I’m not sure.

Immunization-focused charities include the GAVI Alliance, the Measles Initiative and VillageReach. Malaria-focused charities are a different set entirely including Nothing But Nets and the Against Malaria Foundation. Are malaria charities going to be happy to direct their resources toward immunization charities, or will they end up duplicating their work so they can run things themselves?

One particular thing that worries me is the extent to which malaria charities have constructed brands, stories and even names around a specific approach to the problem (bednets). For example, see the Rick Reilly column that launched Nothing But Nets.

Is it possible that charities could overemphasize the wrong solution to their problem, for the sake of a pun?

I’d like to think it’s impossible. But as long as most donors give based on stories, not facts, I can’t be sure.

Microfinance where access already exists

(updated on: ) | by Holden

A key argument for supporting microfinance is that “Millions of families are … without access” to financial services.

Families may often lack access to credit, but they don’t always – and microfinance institutions may not always be clear on which situation they’re dealing with.

A 1999 paper by Brett Coleman (PDF) aims to examine the impact of two microfinance nonprofits in Thailand. Prior to a recent wave of strong studies, this paper was one of the most rigorous impact evaluations available, and it found no positive effects (and some negative ones) for access to the nonprofits’ services. But to me, its most surprising finding was that

many households … have access to low-interest institutional credit, the most frequently used source being the state-run Bank for Agriculture and Agricultural Cooperatives (BAAC), which serves 4 million Thai farm households (84.5% of all farm households in the country) with subsidized low-interest … loans.

In other words, nonprofit lenders were working in an area where there was already an enormous, state-run source of credit. I used to be under the impression that microfinance charities wouldn’t possibly go into an area where a quality substitute was already available, but I’m no longer sure. And when we ask microfinance charities for information about what other sources of credit are available where they operate (and whether their clients are using them), they often have little (or no) such information.

There’s a parallel to the case of Village Phone. Unfortunately, just because a charity is selling something doesn’t mean they’ve established a true need for it.

Can a donor give “new life for the pariahs” through the World Fistula Fund?

(updated on: ) | by Holden

Nicholas Kristof’s excellent column describes the problem of obstetric fistula and the World Fistula Foundation.

We strongly share Mr. Kristof’s view that obstetric fistula is a pressing problem, and have written before that it’s one of the problems we’d most like to help donors address.

However, we still have not identified a fistula charity that we can be highly confident in. Here’s what we do know about the Worldwide Fistula Fund (WFF), the charity discussed in the column:

We’re confident that WFF is a legitimate operation working hard to address the problem of obstetric fistula. We’ve had significant back-and-forth with Dr. Lewis Wall, the founder and President of the Worldwide Fistula Fund.

Obstetric fistula is correctable, but the surgery is complex and may have a high failure rate. More at our discussion of the condition of obstetric fisula.

WFF’s current initiative is building specialized fistula facilities. The proposed plan (PDF) states that “cases such as fistula repair are chronically “bumped” from the surgical schedule by a continuous string of emergencies,” and thus that “without dedicated
facilities for fistula surgery, these operations will never get done.”

Is it the case that regular hospitals are unwilling or unable to treat fistulas reliably? Or might this approach represent an unnecessary diversion of resources from other pressing health problems (for example, pneumonia)? Especially if WFF’s focus is on lobbying the government to reallocate funding rather than using donations to increase total aid flows? At this point we don’t have the information to say.

We don’t know:

  • The skill level of the surgeons that will be working at WFF’s new facilities, or the success/complication rate of their surgeries. Hopefully this information will become available in the future.
  • The full story of what happens to the women who get corrective surgery. We have some information about the medical benefits of surgery, but very little about the social implications. To what extent can a “pariah” be re-accepted into her community?