The GiveWell Blog

CARE evaluations

(updated on: ) | by Holden

How transparent is CARE?

On one hand, it maintains a site at www.careevaluations.org that currently lists 448 project evaluation documents (352 of which are in English). We haven’t found anything comparable for any other of what we call the “household name” charities – enormous, well-known, aggressively fundraising international aid charities (usually members of the InterAction network) that conduct a huge array of different programs in different places.

On the other hand, it does not appear to link to this website anywhere from its main website – in fact, there appear to be only four external links to the site anywhere on the Web.

Looking through the evaluations provides an interesting example of what one of these “household name charities’” operations and impact evaluation look like. The variety of the projects and of the evaluations is huge. Some evaluations examine measures quite relevant to “impact,” such as reported behavior change and children vaccinated (example); others are looser, mentioning regional trends in disease burden but focusing on qualitative generalizations (example); others do not examine life outcomes at all, but simply make qualitative observations about strengths and weaknesses of the program evaluated (example). The quality and tone of the studies varies considerably as well. The use of “control groups” to assess impact is rare but occasional; none that we examined have what we consider to be a high level of rigor, but many appear encouragingly honest about program weaknesses as well as strengths.

Note that this set of evaluations appears to be far from comprehensive: CARE currently lists 845 active projects, whereas the database (which in some cases includes more than one evaluation per project, and goes back to 1991) contains only 448 evaluations as of this writing.

This isn’t the level of impact evidence that we see from our recommended charities, but some evaluation is better than no evaluation and non-publicized disclosure is better than no disclosure.

As a side note, CARE appears to be the only “household name charity” that turned down government funds during the debate over US-provided food aid. We aren’t sure whether they have the right side of this debate, but the turning away substantial money is unusual among charities, and suggests that CARE’s staff aren’t always putting fundraising first.

Bottom line: we’d recommend these charities over CARE, but we’d recommend CARE over other “household name” charities.

GiveWell grant: Open application

(updated on: ) | by Holden

We welcome applications for $250,000 in funding for economic empowerment in sub-Saharan Africa, to be disbursed by 12/31/2009. Interested charities should read the full details of our application process and then submit our first-round application.

Why we are making this grant: in 2008 we received $250,000 earmarked specifically for regranting to a top organization working on “economic empowerment” (I.e., raising incomes directly as opposed to focusing on education or health outcomes) in sub-Saharan Africa. This grant was associated with an institutional donor that prefers anonymity.

Our recent work on international aid has concluded that economic empowerment is not a particularly promising area for a donor, and we have found no charities in this area that are as promising as our top charities or that have met either of our criteria for further investigation. More at our current writeup on economic empowerment.

We are committed to honoring our donor’s intentions, and with $250,000 to grant, we feel it is possible that we will get access to information we haven’t been able to get access to before. (This serves as something of a “reality check” on the approach we used in our recent research report, which used the information available on charities’ websites as a key indicator of how promising they were – details here.) Thus, we are conducting an application process for this area and this funding in particular.

What we hope to accomplish with this grant: we are

  • Looking to expand a proven, cost-effective, scalable program rather than to fund an “experimental” proposal with no empirical supporting evidence to date.
  • Looking to help people in sub-Saharan Africa go from extreme poverty to moderate self-sufficiency. (See our definition of these terms.)
  • Planning to publicly post as much as possible of the materials, and reasoning, behind our decision so that other donors, particularly individual donors, can benefit from the work we do. Because of the larger-than-usual (for us) grant, we recognize that we may be sent more confidential materials than usual, but have a preference for charities that also want to share as much information as possible. All applicants are being strongly encouraged to (and must be explicit if they don’t want to) share their application materials publicly.
  • Trying to minimize the application/reporting burden on all but the top contenders. Our first-round application does not ask everything we need to know; it is intended to filter out those charities that cannot or will not provide fairly basic, but in our experience fairly rare, information about (a) income/standard of living of clients; (b) details (particularly financials) of past successes in creating self-sustaining operations. Our framework for economic empowerment explains why these two pieces of information are so vital.
  • Open to funding new research, rather than a program, if we feel it’s necessary. We will most likely award all of the grant money to one or more charities, and are doing what we can to maximize our odds of finding a strong one. If, however, we feel that we truly cannot have confidence in any applicants, we have permission from our funder to use the funds on a formal study of the effectiveness of an existing program (to be carried out by an external evaluator such as the Poverty Action Lab). Note that if we do go this route, we will still be granting all of the $250,000 to one or more other organizations by 12/31/2009 (i.e., any research project we fund will be carried out by other organizations, not by us).

We make grants and recommendations based on substantive and (whenever possible) shareable information, not based on personal relationships, and so we are casting the net as wide as possible. If you know of any great organizations in this area, please make sure they know about our grant.

High-impact nonprofits are rare, but worth funding

(updated on: ) | by Holden

Following up on Thursday’s Alliance for Effective Social Investing meeting, Sean at Tactical Philanthropy writes:

A high performance nonprofit is a very well run organization. It has outstanding leadership, clear goals, an ethic of monitoring performance and making adjustments as needed, and it is financially healthy.

A high impact nonprofit is one whose efforts have been proven to cause sustainable, positive change.

Impact can be seen only in retrospect. Often many years later. Performance can be directly observed.

I think high impact nonprofits are the holy grail of philanthropy. But like any holy grail, it is something to journey towards, not something you demand now.

Sean goes on to argue that funders should put more focus on “high-performing,” as opposed to “high-impact,” nonprofits. At GiveWell, we focus on “high-impact” nonprofits, in that we look for evidence of past impact and not just future promise. Our response to Sean:

1. Assessing “high-performance” is much harder than assessing “high-impact.” This isn’t to say that either is easy. But we feel it’s very doable for charities to take the “form” of a “high-performance” nonprofit – collecting large amounts of data, executing activities competently, and describing those activities in a compelling and money-raising way – without actually being on a path toward impact (which requires that the data be the right data and that the activities be the right activities for the goal).

We see many charities with impressive-looking evaluation systems; far fewer with actual past outcomes to show. If anything, this makes us suspect that other funders are looking for the form and appearance of good evaluation, without holding charities accountable for actual results.

2. Because of this, funding “high-performance nonprofits” is not something that casual donors (as opposed to subject matter experts) are well positioned to do. This point parallels our argument that casual donors aren’t well positioned to fund the unproven and innovative. Like funding a small and unproven charity, funding a “high-performance” but not “high-impact” charity means trying to do something that hasn’t been done before, and introduces a greater need for understanding the full context of a program.

3. “High-impact” nonprofits might be rare – but that doesn’t make them overfunded. We believe that our top-rated charities can productively use more funds than they’re currently getting. As long as that’s the case, why should a casual donor give to a charity without past impact rather than a charity with past impact?

A small charity that meets our criteria

(updated on: ) | by Holden

As we’ve written before, we tend – deliberately – not to focus on charities that are small and/or “experimental” in nature. From what we’ve seen, these charities rarely can demonstrate that their program has “worked” (in the sense of changing lives) before, and so the only way to evaluate them is to have a deep understanding of the environment they’re entering, the history of similar projects, etc. Our basic response to such charities is: “Get funding from people who are better positioned to evaluate whether this risk is worth taking; evaluate yourself over time; once you can demonstrate impact, a GiveWell recommendation becomes possible.”

However, there’s nothing about our requirements that’s incompatible with relatively small, innovation-focused charities, and VillageReach is a case in point. VillageReach is not a global conglomerate working toward universal coverage of tuberculosis drugs or insecticide-treated nets; it is aiming to transform health system logistics, and to date has completed one pilot project in one district of Mozambique. But it has emphasized monitoring and evaluation from the start, and its pilot project has impressive (if not fully conclusive) results. That distinguishes it from any other “transformative” or “experimental” charity we’ve seen.

In this case, we feel that enough information is available to show us – and individual donors – that this particular risk is worth taking. We feel that VillageReach’s approach has most likely “worked” (impressively and cost-effectively) in the first area where it was tried, and that future attempts are likely to be thoroughly evaluated as well.

There are important differences between VillageReach and the larger charities we recommend. As with any small/innovation-focused charity, giving to VillageReach is a high-risk, high-upside proposition, and we don’t mean our endorsement to indicate otherwise. To date it only has results from one project in one area, and future settings could differ in any number of ways. (The flip side of this risk is hope – hope that if a still-experimental approach proves to be repeatedly and demonstrably successful, it will impact the way other organizations operate, creating effects far beyond its own expenses.)

The choice between VillageReach and a larger charity like the Stop Tuberculosis Partnership is not a straightforward one. There are inherent concerns both with small charities (limited track record; no clear “pattern of success”) and with larger ones (so many activities in so many places that it’s hard to be confident in the organization as a whole), and neither of our top-rated charities has a case ironclad enough to dismiss these concerns. But we feel that on balance, both are extremely “good bets” (and better than any other we’re aware of) for a donor.

This post is based on a discussion of these issues on our public email list.

Development experiments (randomized controlled trials) as a counterpoint to marketing materials

(updated on: ) | by Holden

There’s been a minor flurry of recent blog posts about randomized controlled trials (RCTs) in international aid, including William Easterly’s take and responses by Chris Blattman and GiveWell Board member Tim Ogden.

A central theme has been the difficulty of generalizing from one experiment to whether something “works in general.” There seems to be a loose consensus that while RCTs can highlight interesting behavior patterns, they shouldn’t be expected to stamp programs with “effective” or “ineffective” to the extent that a page like this one suggests.

To me, this concern is another example of “two worlds” in philanthropy. Scholars naturally consider it an open question whether a program “works,” and rightfully point out that a single experiment (or even many experiments) can’t provide a general answer; within this context, the cautions about RCTs are good to note. But charities constantly “sell” the logic of a single approach, independent of time and place; within this context, RCTs provide a meaningful and much-needed reality check.

Microfinance is an excellent example. The Grameen Foundation states, “Microfinance helps people to escape poverty by giving them collateral-free loans and other financial services to support income-generating businesses.” Though it works in four continents, its success stories all follow the same basic pattern, in which a loan is used to expand a small business. Messaging from other microfinance organizations is generally similar, leading to what I perceive as a highly unified impression among donors that microfinance fights poverty by enabling small business expansion.

In this context, the argument of a recent paper (PDF) is highly relevant. It asserts (and we have reviewed it only superficially at this point) that, among other things,

  • The impact of microloans in this case (an expansion of credit in Manila) was generally to make businesses shrink (“treated businesses shed unproductive employees. One explanation is that increased access to credit reduces the need for favor-trading within family or community networks” – pg 3)
  • Loans were more beneficial for “male, and relatively high-income, borrowers” (pg 4)

These claims substantially challenge (a) the story that microfinance institutions have been using to explain and “sell” their operations; (b) the way they have been carrying these operations out; and (c) the assumptions that donors normally bring and the questions they normally ask.

From what we’ve seen, charities don’t proceed with the caution of researchers, trying to fully understand what happened in one area before moving on to the next. They tend to run with a single theory of how an intervention changes people’s lives, scaling up the intervention substantially – often in many countries – with little or no evaluation.

Hence the value of even a single truly rigorous test, in a single time and place, of whether reality was consistent with the theory. We recognize the limitations of RCTs (and have not relied on them much in generating our current recommendations), but we feel strongly that a charity is far more credible when it is testing its claims with RCTs, and that the existence of more RCTs would greatly help donors.

Can donors fund “sustainable” projects?

(updated on: ) | by Holden

It’s one thing to pay for children’s immunizations. It would be quite another to pay for a project that increased immunization rates over the long term, without continued donor support (either thanks to improvement in private-sector or government operations).

Aiming for the latter – or more broadly, aiming to use donations as “startup funds” for lasting and self-sustaining improvements (rather than as indefinite subsidies) – is often referred to as “sustainable development.” Pursuing “sustainability” is often very important to funders. (For example, the World Bank’s latest Independent Evaluation Group report aggregates figures on the likelihood of sustainability for all projects evaluated, and lists these aggregates as one of its key indicators – see Table A.4).

Yet while we would far prefer sustainable programs to unsustainable programs, we don’t currently list “sustainability” as one of our key criteria.

The reason is that while we’ve seen many charities aiming for sustainability, we’ve yet to see any that can demonstrate both future plans and past success in this area.

Sustainability is hard to come by

Most of our recommended international aid charities are aiming at least partially for lasting impact, but evidence that they’ve achieved it is murky at best. Two examples:

Non-recommended organizations rarely appear to be documenting long-term impact of any kind.

Independent literature reinforces the concern that achieving sustainability is difficult and rare.

  • A 2004 paper entitled “The Illusion of Sustainability” states, “Some anecdotal evidence suggests that sustainability has often been a chimera – and sometimes a costly one.”
  • A 2008 paper by William Easterly adds, “This hope [for sustainable projects] has turned out to be an illusion, as the failure to cover recurrent costs has been nearly universal.”
  • A book reviewing aid effectiveness (and generally positive on it) states that “The bulk of the NGO evaluation material which looks into the issue of sustainability has concluded that most NGO projects are not financially sustainable without the continued injection of external funds … Exceptionally few studies have provided a long-term view of [nonprofit] projects and programmes. Most views on the sustainability of NGO projects are based on forward extrapolation of short-term assessments” (281).

We do believe that sustainability in aid is possible, and has been achieved in the past. For example, Aravind Eye Care System appears to be a self-supporting humanitarian organization, using revenue from paying customers to perform cataract surgery on those with lower incomes. This program, in fact, appears so sustainable that there is little donors can add to it, and donations support tangentially related activities. (More broadly, we were recently pointed to family planning as an example of an area where aid has been associated with lasting behavior change.)

But as with impact, we believe that sustainability needs to be demonstrated, not just claimed. In fact, the burden of proof for demonstrating sustainability is even higher than for demonstrating impact.

The importance of long-term evaluation

We constantly stress the importance of systematic, long-term impact evaluation, and note how rarely it is seen in the world of charity. It seems to us that if one is pursuing sustainable impact, the challenge and importance of evaluation become more significant. In order to assess sustainability, a charity must be systematically examining an area (with all the challenges that involves) even after it has withdrawn support.

VillageReach appears to be doing exactly this, and is therefore at least in a position to identify and respond to disappointing results (according to VillageReach representatives, the local government appears to have lapsed back into its former, less efficient logistics approach, but has agreed to renew its focus on sustaining the new model).

In our view, any project that does not include rigorous evaluation can’t make any credible claim that it has achieved sustainability – only that it hopes to.

A desirable goal, but not a reasonable requirement

We think it may often be wise for experts and major funders to focus on sustainability – doing so could make sense even with a high failure rate, because successes are so significant.

But in trying to help individual donors – who aren’t well positioned to innovate – we focus on finding programs that have worked before and are likely to work again. If we required recommended charities to show evidence of “sustainability,” we simply would have no recommended charities, at least at this point. As such, we’ve evaluated our strongest charities by what they can accomplish even assuming that their hopes of sustainability don’t pan out. We recommend VillageReach, despite concerns about sustainability, because its program appears to do a lot of good for relatively little money even if its improvements are never maintained (more here).

If VillageReach’s program does end up having a lasting impact, it will be an even better deal. If we later find a charity that can convincingly demonstrate an ability (not just a plan) to create sustainable and demonstrable impact, we will recommend it strongly. But in the meantime, it doesn’t seem reasonable to criticize a charity on the grounds that its work isn’t sustainable. Simply having real, demonstrable impact is hard enough.