This post is more than 10 years old

GiveWell’s traditional work (the work behind our current top charities) and our work on GiveWell Labs reflect two very different visions of giving.

• The first, giving as consumption, sees giving as analogous to making a purchase. For every $X one spends, one gets some desirable outcome (such as a life saved), and the goal is to find giving opportunities that can deliver these outcomes approximately linearly and with good value-for-money.
• The second, giving as investment, sees giving as analogous to investing in a company. Risk is known to be high, and outcomes hard to foresee in detail (particularly for earlier-stage investments). Rather than asking “What will each $X buy?” one tends to focus more on questions like “Does this organization have a good team and model?” and “Is this organization positioned to have a huge upside, even if one can’t say in advance just what this would look like?”

These don’t represent strict categories – any giving opportunity can be described in the language of either – but they reflect different ways of thinking about giving, and tend toward different ways of evaluating (and describing) the pros and cons of different giving opportunities.

Over the years, we’ve come to believe that “giving as consumption” is an apt framework only for a tiny minority of giving opportunities today – particularly if one seeks strong evidence and cost-effectiveness – and “giving as investment” is likely to be the more broadly useful framework for evaluating giving opportunities. Recent developments with our top charities (see recent posts on AMF and SCI) have moved us further in this direction.

Our audience has expressed (and we have felt) confusion over how hard it seems to be to find proven, cost-effective giving opportunities along the lines of “$X saves a life.” Many of the major funders we talk to – who tend to prefer the “giving as investment” model – have expressed views along the lines of “It’s easy to spend some money, buy some bednets and save some lives; we’re being more ambitious.” Yet we believe that proven, cost-effective “giving as consumption” is a very hard thing to find, and we think we can explain why.

In brief:

• There are a relatively small number of programs that are strongly evidence-backed, in terms of their ability to improve lives. Evidence about the impact of a charitable intervention on life outcomes is inherently difficult and expensive to collect; it faces a large number of inherent challenges in terms of reliability and generalizability. More
• Of evidence-backed programs, many are delivered widely via governments and the global aid community. That leaves quite a small number of interventions that both (a) are strongly evidence-backed (b) have room for more funding. In many cases, these are interventions for which the strongest evidence is relatively recent, such as cash transfers and deworming. More
• Charities often aren’t structured to focus on directly delivering a small number of programs. Charities that do deliver evidence-backed interventions often present a further challenge to the “giving as consumption” model for a couple of reasons: (a) they often seek to promote their programs to governments (in addition to, or instead of, delivering the programs themselves), and (b) they often work on a broad variety of activities (making it difficult to understand the marginal impact of one’s donation). More
• “Rollout” is rarely straightforward. Because collecting rigorous evidence is expensive, difficult and slow to bear results (along the lines of our first bullet point), there’s often great uncertainty in how a program is adapting – and how relevant its evidence base remains – in new contexts and changing times. More than one of our recommended charities has struggled to reach a new level of scale, and with all our priority programs there are major questions about how closely the context in which they were studied matches the context in which they’re executed today. More
• Transparency and accountability present a further challenge. The current culture around charities is one in which nuance, substance and criticism are extraordinarily rare. Even if a charity has relatively strong evidence of impact, it may hesitate to share the technical details and subject itself to the sort of scrutiny – and honest discussion of strengths, weaknesses, successes and shortcomings – that we seek to provide. Doing so is also a major time investment, so the incentive to do so has to be sufficient. As our money moved has grown, we have had more success in getting engagement from charities on this front. More

Below, we expand on each of these obstacles to finding good “giving as consumption” opportunities, and discuss how (if at all) one might work to overcome them. We believe that GiveWell has played a role in improving the quality of “giving as consumption” opportunities, and will continue to do so. At the same time, we think that the “giving as consumption” framework will likely always apply to only a small fraction of giving opportunities, and broadening our horizons (via GiveWell Labs) is essential.

In the meantime, we note that GiveDirectly is the only giving opportunity we can name today that seems to strongly and closely fit the full “giving as consumption” model: focus on an evidence-backed program, using funds to directly implement the program, and producing and publicly sharing strong monitoring and evaluation data from its implementation. This doesn’t mean that GiveDirectly is the best of our top charities: as discussed in our announcement, the others have major potential advantages. But it does mean that our traditional criteria – “proven, cost-effective, scalable” – no longer describe our full set of top charities. Accordingly, we have modified our criteria page and our process page so that they accurately describe what distinguishes our top charities from others – and list what we see as the pros and cons of supporting them. (More details below.)

Obstacle 1: a relatively small number of programs are evidence-backedWe’ve searched extensively over the years for proven, cost-effective programs that charities can deliver. At this point, we believe it’s likely that we’ve identified all or nearly all such programs.

We believe the problems of selection bias and publication bias are serious ones. As such, we’ve become skeptical of much of the research that is published, and we’ve developed principles for how to identify strong evidence. (See Our Principles for Assessing Evidence, How We Evaluate a Study, Surveying the Research on a Topic).

With these principles in mind, strong studies are relatively rare, especially in areas outside of health. A strong study often involves using randomization to create a “treatment group” (receiving a program) and a “control group” (not receiving the program, and thus serving as a basis for comparison); following both the treatment and control group over a long period of time; and seeking objective measures of whether the program improved lives. Such evaluation can be very expensive and take a long time to execute, and thus it can be important to put a great deal of care and planning into it so as not to waste time and resources. Such evaluation generally manifests as academic literature, and we’ve never encountered a case in which a charity continually builds randomized (or other rigorous) evaluation into a substantial portion of its day-to-day programming.

Health programs are generally easier to study than other programs, for a couple of reasons. Appropriate measures to evaluate health programs can be relatively straightforward – death rates, prevalence of various markers and symptoms of disease, etc. – and it often appears that results can be generalized fairly well. By contrast, examining the long-run impact of e.g. an education or job training program on quality of life can involve questionable measures, very long time horizons, and substantial questions over whether the results in the study will generalize to new contexts.

We discuss the proven cost-effective programs we’ve found, and our process for finding them, at Priority Programs.

Prospects for progress on this obstacle. Developing new health technologies and approaches could lead to new priority programs. In addition, there are cases of existing health-based programs that we feel could use stronger evidence than they have (these include deworming and clean water interventions).

Conducting more rigorous studies of non-health programs – as Innovations for Poverty Action and others do – could lead to strong evidence bases for non-health programs. However, creating a strong evidence base for a non-health program appears to be extremely difficult, and we’ve seen it happen only in one case to date (the relatively, though not extremely, strong evidence base for cash transfers). Part of the reason for this may be the inherent difficulty, high expense and long time horizon of assessing non-health impacts in a generalizable way; there may be some more complex contributing factors as well (it seems to us that many of the people conducting these studies are economists, who are often more interested in exploring a general principle and/or demonstrating a new idea than in conducting the second, third, fourth, etc. study on a program that has already been evaluated once).

We’re hopeful that by creating and demonstrating demand for evidence-backed programs, GiveWell can improve incentives for people to carry out more such studies.

Obstacle 2: many evidence-backed programs are delivered widely via governments and the global aid communityWe wrote at length about this issue early this year. At the time, one of the major exceptions we saw was that of bednets, but even on that front the funding picture appears to be improving dramatically. Total global health aid is in the range of $30 billion per year and much of the “lowest-hanging fruit” is already picked.

This dynamic is easy to underestimate, because charities may advertise themselves as delivering certain interventions (such as vaccines). However, donations toward this end may ultimately be fungible with other donations. There’s a big difference between “having your donation allocated to paying for vaccine delivery” and “having your donation cause more vaccines to be delivered.”

It’s a good thing that governments and multilateral organizations play a major role in rolling out proven, cost-effective programs. However, it also means that the opportunities for donors to “buy” more of such programs are less common than they may appear.

Deworming and cash transfers are both programs whose strongest evidence is relatively recent. Because of this, they may currently be in a “sweet spot” between becoming widely accepted as good uses of funds and becoming widely rolled out – creating opportunities for private donors to fill gaps before larger funders do, and in some cases to leverage larger funders via advocacy and technical assistance.

Obstacle 3: charities often aren’t structured to focus on directly delivering a small number of programsThe Deworm the World Initiative (DtWI) is an example of trying to take advantage of the kind of “sweet spot” described in the previous section. It focuses on advocacy and technical assistance in order to encourage, and assist, governments to execute large-scale deworming programs. Schistosomiasis Control Initiative (SCI) appears to have a mix of activities: in some cases it uses funds to carry out programs itself, and in others it uses them to fill gaps, do advocacy, and create pilot programs in the hopes of influencing governments and other funders. (For example, see our most recent update on SCI.)

These are very logical activities for a charity – arguably more logical (if they can successfully influence larger funders) than direct delivery. However, these activities pose serious challenges from an evaluation perspective: whereas GiveDirectly can show documentation on programs it has directly carried out, it’s much harder to say just what donations to DtWI have accomplished (and will accomplish).

Another structural issue is that many charities work on a broad variety of activities. We discuss some such charities in a 2011 post. When dealing with such charities, it can be extremely challenging to trace the impact of additional donations.

Prospects for progress on this obstacle. We don’t believe it’s necessarily a bad thing for charities to focus on harder-to-evaluate activities such as technical assistance and advocacy, and accordingly, we’ve modified our criteria so that they no longer require organization-specific evidence of effectiveness. Regarding large broad charities working on a variety of activities, we believe that it may become easier to trace the impact of marginal dollars as the dollar amounts become larger (at a high enough level of funding, one can fund an entire project that wouldn’t have been on the agenda otherwise).

We see hope for progress on being able to evaluate sub-programs of charities that work on multiple activities. Our recommendation of DtWI represents the first time that we’ve recommended an initiative within a larger organization; given the amounts of money moved we anticipate this year, we are comfortable with this and anticipate being able to assess the impact of GiveWell-directed funds on deworming activities relatively well.

Obstacle 4: “rollout” is rarely straightforwardThe context in which a program was studied is often very different from the context in which it is now being carried out:

• The strongest evidence we’ve seen for the positive impact of deworming comes from two studies: one on a hookworm eradication campaign in the American South in the early 20th century, and another from a randomized controlled trial in Kenya during El Nino (in which El Nino led to greatly elevated levels of worm infection, and thus likely greatly elevated impact of treatment). In examining Deworm the World Initiative, we’ve concluded that the areas it works in have far less prevalence of worm infections than was seen in the Kenya study (even prior to the impact of El Nino).
• The evidence on cash transfers also comes mostly from programs that are importantly different (in terms of the amount of cash given and the time period over which it was given, among other things) from the program run by GiveDirectly, although this issue has been mitigated this year by the release of a randomized controlled trial that directly examines GiveDirectly’s work.
• Most of the evidence for the effectiveness of insecticide-treated nets (for preventing malaria) is from before 2000, and we’ve noted that improving general health since then may have decreased the marginal impact of insecticide-treated nets. We’ve also noted that these studies of bednets involved more intensive promotion, and measures to ensure that nets were hung properly, than a typical large-scale campaign might involve.

In addition, a charity can hit many unanticipated obstacles as it expands from one area to another:

• From 2009-2011, we recommended donations to VillageReach, which we believed showed strong evidence that its pilot logistics program had improved vaccination rates in the Cabo Delgado province of Mozambique, and which had the approval of the national-level Ministry of Health to roll out the program to the rest of the country. However, the rollout ran into a variety of obstacles detailed in a 2012 update, mostly related to getting buy-in from other funders to support the transition to VillageReach’s model. (VillageReach continues to move forward and recently received ~$3 million from the Gates Foundation.)
• We recently wrote about the Against Malaria Foundation (AMF) and its delays in committing its funds to a distribution of insecticide-treated nets. AMF’s increase in scale led to new challenges in negotiating with governments and other partners on distributions.

We generally seek as much monitoring and evaluation as we can from the organizations we review, so that we have a basis for comparing their programs as they are executed today with the programs that studies have demonstrated the effectiveness of. However, as discussed above, we have never seen a case in which highly rigorous evaluation is integrated into a charity’s general activities; the data that is available generally has major limitations (many of which aren’t immediately apparent – for example, see our recent post about SCI’s evidence of impact).

Prospects for progress on this obstacle. Changing contexts will always be a real challenge, but we are optimistic that we will see more, and better-designed, monitoring and evaluation over time to act as a diagnostic. We feel that Against Malaria Foundation has dramatically improved its approach to monitoring and evaluation, adding measures to assess whether the insecticide-treated nets it distributes remain in use and are effective over time. GiveDirectly, which is the recommended charity with the strongest overall monitoring and evaluation from our perspective, has only been in existence since 2011. In both cases, we feel that GiveWell’s existence provided encouragement – and funding – for strong monitoring and evaluation, though we can’t make a firm statement about the impact we’ve had (it’s possible that in both cases the same outcome would have occurred if GiveWell had not existed). If we continue to demonstrate strong and growing money moved, and continue to be public about what we see as the strengths and weaknesses of the evidence collected by recommended charities, we expect to see more improvement over time.

Obstacle 5: transparency and accountabilityIn our experience, it’s almost never possible to understand the case for a charity’s impact based on publicly available materials. Monitoring and evaluation reports are generally internal (and require assistance to interpret); the sort of information necessary to understand room for more funding often requires extensive back-and-forth to obtain. Thus, if we seek to thoroughly understand a charity’s work, it’s generally necessary that the charity voluntarily participate in our extensive evaluation process.

This has been a significant issue throughout our existence. We’ve tried to make participation in our process as attractive to charities as possible, which has generally included sending them previews of content we’d like to publish about them, and the discussions we’ve had around preview content have consistently given us the impression that charities are uncomfortable (more so than other organizations) being discussed publicly in anything but glowing terms. In addition, it has often been surprisingly difficult to get charities to share substantive information even when they have it. We’ve often been connected to fundraising-oriented staff who have sent promotional materials in response to our substantive requests. (For some flavor of this, see the list of grant application responses from our 2007 process; the page is out of date and has problematic formatting but still contains the relevant materials.)

At this time, we believe that the idea of public, thorough evaluations of charities – discussing weaknesses as well as strengths, shortcomings as well as successes – remains a very unusual (and, to many charities, discomfiting) idea. It is a major factor in why there are so few GiveWell-recommended charities.

With that said, there has been real progress.

• We have seen dramatic improvement over the years from SCI in terms of sending substantive information to help us understand its activities.
• This year (accompanied by a change in leadership), DtWI changed from an organization that we knew very little about to an organization that participated eagerly and thoroughly in our process, leading to a recommendation.
• GiveDirectly has been highly transparent since its inception in 2011, when it allowed us to publish a pre-analysis plan for a study on its work. We believe that GiveWell may have played a role in causing GiveDirectly to share more information publicly than it would have otherwise (although its founders have shown what we perceive as a genuine openness to and enthusiasm about extreme transparency).
• We have also recently seen interest from other charities that previously declined to participate in our process, and we plan to follow these up in the coming year.

Prospects for progress on this obstacle. We are highly optimistic that if we can continue to demonstrate strong and rising money moved, we will see more and more interest from charities in participating in our process and being thoroughly and publicly evaluated.

With that said, we also recognize that participation in our process is a major ask, and this year we have begun providing a list of charities that focuses on evidence-backed programs, regardless of whether they participate in our process, so that donors can support these if they prefer.

Bottom line

There are many reasons it’s hard to find a proven, cost-effective giving opportunity, along the lines of “Every $X I spend accomplishes Y.” We have seen – and perhaps played a role in – substantial progress on some fronts, particularly with regard to (a) improved monitoring and evaluation and (b) improved willingness of charities to share substantive information and be thoroughly, publicly evaluated and followed up. We expect more progress, and we believe that our offerings for donors interested in “giving as consumption” will improve. GiveWell’s money moved provides an incentive for evidence-backed charities to share information, and an incentive for younger charities to build evidence and transparency in from the beginning. We are also contemplating other measures we can take (aside from our charity recommendations) to contribute to progress, such as providing seed grants to help potential future top charities get started.

With that said, some of the obstacles to “giving as consumption” are fundamental, and we now believe that this framework will – for the foreseeable future – apply to only a very small subset of giving opportunities. GiveWell Labs is exploring giving opportunities using a very different framework.

In the meantime, we find ourselves with very little in the way of giving opportunities that we can call “proven,” even though we do have giving opportunities that we consider outstanding (partly due to more “investment” type factors, which we discuss at our top charities announcement).

Accordingly, we have modified our criteria page and our process page so that they accurately describe what distinguishes our top charities from others – and list what we see as the pros and cons of supporting them. In brief, we have removed language implying that charities are “proven” in the sense of having demonstrated their own impact, or “scalable” in the sense that we can precisely predict and quantify the impact of additional donations. Instead, we emphasize that our top charities (a) focus on evidence-backed programs serving the global poor; (b) have been thoroughly vetted, such that even if we haven’t pinned down the answer to each of our key questions, we’ve gathered as much information as we can and can make informed statements about our top charities’ strengths and weaknesses. For those who are concerned that openness to our process is not a desirable quality in a charity, we also provide a more extensive list of charities focused on proven cost-effective programs.

We also now prominently discuss the pros and cons of giving to our top charities:

We think the principal advantages of our current top charities are that:

• They represent the best opportunities we’re aware of to help low-income people with relatively high confidence and relatively short time horizons. If you’re looking to give this year and you don’t know where to start, we’d strongly recommend supporting our top charities.
• Due to the emphasis on thorough vetting, transparency, and following up, our top charities represent excellent learning opportunities, and we feel that one of the most desirable outcomes of giving is learning more that will inform later giving. Supporting our top charities helps GiveWell demonstrate impact and improves our ability to learn, and we are dedicated to sharing what we learn publicly.

Some counter-considerations:

• There is an argument for saving money rather than giving, and giving at the point where better information on top giving opportunities is available. We do expect to make substantial progress on GiveWell Labs over the next few years.
• If you have access to other giving opportunities that you understand well, have a great deal of context on and have high confidence in — whether these consist of supporting an established organization or helping a newer one get off the ground — it may make more sense to take advantage of your unusual position and “fund what others won’t,” since GiveWell’s research is available to (and influences) large numbers of people.