The GiveWell Blog

Errors in DCP2 cost-effectiveness estimate for deworming

(updated on: ) | by Alexander

Two notes on this post:

  • This post discusses flaws in a particular published cost-effectiveness estimate for deworming. It should not be taken as a general argument against deworming as a promising intervention, and it does not address various other publications on deworming including the 2003 paper by Edward Miguel and Michael Kremer.
  • Prior to publication, we sent a draft of this post to several relevant scholars including the authors of the estimate. They have reviewed our work and confirmed the major errors we point out.

Over the past few months, GiveWell has undertaken an in-depth investigation of the cost-effectiveness of deworming, a treatment for parasitic worms that are very common in some parts of the developing world. While our investigation is ongoing, we now believe that one of the key cost-effectiveness estimates for deworming is flawed, and contains several errors that overstate the cost-effectiveness of deworming by a factor of about 100. This finding has implications not just for deworming, but for cost-effectiveness analysis in general: we are now rethinking how we use published cost-effectiveness estimates for which the full calculations and methods are not public.

The cost-effectiveness estimate in question comes from the Disease Control Priorities in Developing Countries (DCP2), a major report funded by the Gates Foundation. This report provides an estimate of $3.41 per disability-adjusted life-year (DALY) for the cost-effectiveness of soil-transmitted-helminth (STH) treatment, implying that STH treatment is one of the most cost-effective interventions for global health. In investigating this figure, we have corresponded, over a period of months, with six scholars who had been directly or indirectly involved in the production of the estimate. Eventually, we were able to obtain the spreadsheet that was used to generate the $3.41/DALY estimate. That spreadsheet contains five separate errors that, when corrected, shift the estimated cost effectiveness of deworming from $3.41 to $326.43. We came to this conclusion a year after learning that the DCP2’s published cost-effectiveness estimate for schistosomiasis treatment – another kind of deworming – contained a crucial typo: the published figure was $3.36-$6.92 per DALY, but the correct figure is $336-$692 per DALY. (This figure appears, correctly, on page 46 of the DCP2.)

We do believe that the corrected DCP2 calculations are too harsh on deworming; our best estimate of the cost-effectiveness of deworming is in between the corrected and uncorrected DCP2 figures, at $30-$80 per DALY. In addition, there are strong arguments for deworming as an excellent intervention that do not depend on these figures. Overall we consider deworming a highly promising (though not the single most promising) intervention; we will be discussing our thoughts on this intervention further in the future. This post focuses not on deworming in general, but on the DCP2 figures and what lessons we should take from the flaws in them.

  • The estimates on deworming are the only DCP2 figures we’ve gotten enough information on to examine in-depth. Getting to this point took a lot of work and communication with a number of different scholars, so we aren’t sure of the extent to which other estimates might also turn out to be flawed if examined closely.
  • We believe that the errors we’ve found in the estimate would have been caught by a helminth expert independently examining the estimate. Therefore, the presence of these errors implies to us that there has been no such examination. If this is the case, it would argue against the reliability of the DCP2’s estimates in general.
  • We’ve previously argued for a limited role for cost-effectiveness estimates; we now think that the appropriate role may be even more limited, at least for opaque estimates (e.g., estimates published without the details necessary for others to independently examine them) like the DCP2’s.
  • More generally, we see this case as a general argument for expecting transparency, rather than taking recommendations on trust – no matter how pedigreed the people making the recommendations. Note that the DCP2 was published by the Disease Control Priorities Project, a joint enterprise of The World Bank, the National Institutes of Health, the World Health Organization, and the Population Reference Bureau, which was funded primarily by a $3.5 million grant from the Gates Foundation. The DCP2 chapter on helminth infections, which contains the $3.41/DALY estimate, has 18 authors, including many of the world’s foremost experts on soil-transmitted helminths.
  • It is possible that we have made errors in our corrections to the calculation. One of the reasons we go to great lengths to be transparent is because we want our errors to be caught as quickly as possible.

Outline for the remainder of this post:

About the DCP2’s estimate

The DCP2 was published by the Disease Control Priorities Project, a joint enterprise of The World Bank, the National Institutes of Health, the World Health Organization, and the Population Reference Bureau, which was funded primarily by a $3.5 million grant from the Gates Foundation.

The Gates Foundation also appears to have invested substantially in the dissemination of the DCP2’s findings, including a $4.4 million grant to the Population Reference Bureau to “disseminate key messages from [the DCP2].”

The DCP2 aims to estimate the cost-effectiveness of different health interventions, in terms of dollars per disability-adjusted life-year (DALY) saved, in order to prioritize the most cost-effective interventions–the ones that will have the largest effects in reducing mortality and morbidity for a given amount of funding. The DCP2’s published estimates imply that soil-transmitted helminth (STH) treatment is one of the cheapest ways to improve health: the same “amount of health” could be provided by spending $1 on STH deworming or roughly $34 on family planning programs or more than $90 on treating drug-resistant tuberculosis. In fact, it appears that the DCP2 rates STH treatment as the second most cost-effective health intervention of all, behind only hygiene promotion (p. 41).

The DCP2’s cost-effectiveness estimates for deworming have been cited widely to advocate a greater focus on treating STH infections, including in:

  • an article (PDF) in The Lancet
  • a report (PDF) by REACH, a consortium of large international NGOs and other organizations working to end child hunger, which labeled deworming one of 11 “promoted interventions”
  • the most-cited paper (PDF) published in the journal International Health
  • an editorial by Peter Hotez, a co-founder of the Global Network for Neglected Tropical Diseases, which has received more than $40 million in funding from the Gates Foundation
  • work by charity evaluators, such as GiveWell, Giving What We Can, and the University of Pennsylvania’s Center for High Impact Philanthropy.

Why we decided to look into the DCP2’s deworming estimates

We undertook this research because:

  • We wanted to do a case study of a cost-effectiveness estimate from the DCP2, understanding the full details of what goes into it and where the room for error is.
  • We were particularly curious about the estimate for treatment of soil-transmitted helminths since the published $3.41 per DALY averted figure didn’t seem to sync with what we knew about the costs and effectiveness of STH treatment (or the independent estimate of $280/DALY given by another study, as we’ve mentioned previously).
  • We also wanted to focus on STH treatment since the DCP2 rates it as the second most cost-effective health intervention of all, behind only hygiene promotion.
  • Finally, we wanted to learn more about deworming after Elie visited the Schistosomiasis Control Initiative in London and we became more optimistic about this organization than we had been.

Our process for investigating the estimate

GiveWell took the following steps to investigate the DCP2’s estimate for the cost effectiveness of STH deworming:

  • We initially contacted Peter Hotez, the lead author of the DCP2 chapter on intestinal nematode infections; he sent us several papers on the costs and effectiveness of deworming and referred us to another scholar to explain the calculation that the DCP2 had published.
  • This scholar, in turn, referred us to two more, who sent us further references in response to our questions.
  • At this point we had an extended back-and-forth trying to understand the details of the calculation that had been done, and since we weren’t sure we would reach a conclusion on this, we asked volunteer Jonah Sinick to use all the references we’d been sent to create his own best guess estimate for the cost-effectiveness estimate of deworming. This estimate implied a significantly higher cost per DALY than the published figure, which seemed strange since we were now using the references and inputs suggested to us by the chapter authors.
  • The scholars we had been corresponding with sent us a spreadsheet with the full details of the calculation, as well as an accompanying table, which we will call Table 9, that had been used to input some of the figures in the spreadsheet. Here is the PDF of Table 9 that we were sent.
  • However, the interpretation of the numbers from Table 9 was still unclear to us. Table 9 is not clearly labeled; the scholars involved in the calculation appeared to have conflicting interpretations of what the numbers meant, and both meanings were highly counterintuitive to us (details below).
  • So we contacted another scholar who had worked on Table 9 to get her help in interpreting it. She sent us the full paper from which Table 9 was taken, Intestinal Nematode Infections, and this paper appeared to have a different interpretation of Table 9 than the spreadsheet’s. We confirmed this with her.
  • We also found the disability weights being used counterintuitive, and after some investigation we received confirmation that they were erroneous (details below).
  • All in all, we found five errors in the estimate, not all of which were attributable to the creator of the spreadsheet.

Problems with the official estimate of the cost-effectiveness of deworming

The basic approach of the estimate is to:

  • Calculate the benefits of deworming by
    • Starting from a population of schoolchildren being dewormed;
    • Estimating the percentage of these children suffering from different symptoms of infection;
    • Using the above, estimate the number of children cured of these symptoms (the estimate assumes that they are cured for exactly one year, since reinfection can occur after deworming)
    • Incorporating the severity of symptoms to arrive at DALYs saved by the deworming
  • Separately calculate the costs of deworming this population of schoolchildren, and divide costs by DALYs to obtain the cost per DALY.

When we examined the details of the official estimate, it struck us that nearly all of the DALYs saved (i.e., nearly all of the benefit) were coming from the reduction of a single symptom of a single worm infection: cognitive impairment due to ascariasis (we abbreviate this as CIDTA). Specifically, the figures going into the estimate implied that:

  • In a hypothetical population of 208,530 children (age 5-14 in Latin America) treated, 45,060 suffer from CIDTA. (Cells C44 and L44 in “ascariasis” sheet). That’s about 22%.
  • The disability weight of CIDTA is 0.463 (cell E8). While these figures are difficult to interpret, this implies that having CIDTA is about half as bad as being dead (disability weight 1.0), and only slightly less debilitating than being blind (disability weight 0.6). (See the official list of disability weights published alongside the DCP2.) These figures implied (to us) that CIDTA was not a matter of subtle cognitive impairment, but of mental handicap so severe as to truly prevent normal functioning.
  • The intervention in question – a single dose of albendazole – could completely restore normal mental functioning (i.e., completely eliminate disability associated with CIDTA) for one year.

These implications didn’t sync with the information we had from other sources, such as the Global Burden of Disease (GBD) report published alongside the DCP2.

  • If ascariasis caused this sort of symptom, we’d expect to see much more focus on ascariasis (relative to other helminth infections) in the global health and deworming communities.
  • In addition (as we observed when trying to reconcile the official estimate with our own estimate), if 22% of the 110 million 5-14 year olds in Latin America (GBD, 198-199) had a disability with weight 0.463, then this – alone – would result in 11.2 million DALYs lost to ascariasis per year in this region (22% * 110 million * 0.463). However, the official DALY burden for this ascariasis (all symptoms) among this population is only 31,000 (GBD, 198-199) – in fact, the worldwide DALY burden for ascariasis is only 915,000 (GBD, 180-181).

We therefore did further investigation on the CIDTA symptom – both how prevalent it is and how severe it is. It turns out that the official calculation significantly overstates both. For example, among 5-14 year olds in Latin America, CIDTA affects about 0.23% of the population – not 22.6% as the official calculation suggests – and its correct disability weight is 0.024 (the same severity as anemia), not 0.463.

Specifics of these errors:

  • Prevalence of CIDTA. The official calculation starts from a hypothetical population of 1 million people of all ages, then calculates the number of 5-14 year olds (per million people) using demographic data, then takes the number of CIDTA cases directly from Table 9 (this figure is multiplied by 10 before being put in the official spreadsheet). For example, for 5-14 year olds in Latin America, Table 9’s “A/B” column has the figure, “4506”; the official calculation records “45060” for the number of CIDTA cases among 5-14 year olds.

    The labeling of Table 9 is ambiguous and doesn’t make it clear whether this is the intended meaning of the figures. We contacted one of the original authors who wrote the paper from which Table 9 is taken, received a copy of the (unpublished) paper from her, discussed it with her, and found that this figure’s intended interpretation is different from the official calculations, in two ways:

    • The figure in the “A/B” column refers number of people at risk for a given symptom, not the number of people suffering from that symptom. These are equivalent for Type A and Type C symptoms, but not for Type B symptoms including CIDTA. Intestinal Nematode Infections (PDF), the working paper that contains Table 9, says that “in any annual cohort of heavily infected children some 5% suffer [Type B symptoms, which are the only symptoms that have life-long effects]” (p. 26). Using the figures as the official calculation did would therefore lead to a 20x overstatement in the prevalence of CIDTA.

      This mistake applies not just to cognitive impairment due to ascariasis, but also to cognitive impairment due to trichuriasis and hookworms, similarly leading to a 20x overstatement of the prevalence of cognitive impairment due to those infections as well.

    • The figures in Table 9 refer to the number of children at risk, per 100,000 children of the age group indicated in the row. For 5-14 year olds in Latin America, the figure (for symptoms “A/B”) is “4506”; this means that 4506 out of 100,000 5-14 year olds are at risk for CIDTA. This in turn means that 45060 of every million 5-14 year olds are at risk. However, the official calculation assumes 45060 cases not for one million 5-14 year olds, but for only 208,530 5-14 year olds (which is the number of 5-14 year olds one would expect in a population of 1 million people across the three age groups). Thus, this difference results in overstating the prevalence of CIDTA by about 5x.

      This mistake applies to each of the symptoms of all three soil-transmitted helminths, not just to CIDTA, and therefore leads to an overstate of the prevalence of every symptom of STHs by about 5x.

    Bottom line – the correct interpretation of Table 9 (for 5-14 year olds in Latin America) is that 45060 out of every million 5-14 year olds are at risk for CIDTA, and 5% of these actually have it – so 2253 out of every million 5-14 year olds have CIDTA. The official calculation assumes that in a population of 208,530 5-14 year olds, 45060 have CIDTA. The same types of errors apply to the other regions and conditions as well.

  • Severity of CIDTA. The disability weight of 0.463 is correctly transcribed from the Global Burden of Disease official disability weights, which in turn takes the figure from the earlier 1996 edition (which we examined in a library). However, we still found this figure odd because of the contrast with the other two kinds of helminth infections:
    Helminth type Symptom A – disability weight Symptom A – description Symptom B – disability weight Symptom B – description Symptom C – disability weight Symptom C – description
    Ascariasis 0.006 Reduction in cognitive ability in school-age children, which occurs only while infection persists 0.463 Delayed psychomotor development and impaired performance in language skills, motor skills, and coordination equivalent to a 5- to 10-point deficit in IQ 0.024 Blockage of the intestines due to worm mass
    Trichuriasis 0.006 Reduction in cognitive ability in school-age children, which occurs only while infection persists 0.024 Delayed psychomotor development and impaired performance in language skills, motor skills, and coordination equivalent to a 5- to 10-point deficit in IQ 0.114-0.138 Rectal prolapse and/or tenesmus and/or bloody mucoid stools due to carpeting of intestinal mucosa by worms
    Hookworm NA NA 0.024 Delayed psychomotor development and impaired performance in language skills, motor skills, and coordination equivalent to a 5- to 10-point deficit in IQ 0.024 Anemia due to hookworm infection

    It looked to us as though the weights may have been switched, in the case of ascariasis, for symptoms B and C. We contacted Colin Mathers, the second-listed author on the Global Burden of Disease publication, and he confirmed to us that the weights are in fact switched, stating, “We also noticed this and corrected it in the spreadsheets for WHO estimates, but possibly it has remained uncorrected in some of the summary tables of weights.” Thus, CIDTA’s correct disability weight is 0.024, but the published disability weight in both editions of the GBD – and the weight used in the official cost-effectiveness calculation – is 0.463.

We created a version of the official calculation that corrected for the above errors, as well as two other errors that we found in the process of checking the calculation as thoroughly as we could. (See Footnote 1 below.) Our version is here (XLS).

This calculation leads to a revised cost-effectiveness estimate of $326.43 per DALY, rather than the $3.41 per DALY in the original.

The DCP cost-effectiveness estimates only took into account short term effects of the three diseases, even though they have some long term effects. This seems to have been an intentional decision rather than an error, but our feeling is that a best estimate of the true cost-effectiveness of deworming would likely take these long-term effects into account. We therefore created another version of the estimate that does so, as best as we can. (See Footnote 2 below.) Taking these long-term effects into account, our cost-effectiveness estimate for STH treatment moves to $138.28 per DALY.

These corrections also have implications for the cost-effectiveness estimate for combination deworming (simultaneously addressing both STH and schistosomiasis, another type of infection). The DCP2 reports a cost-effectiveness estimate of $8-$19/DALY averted for combined treatment, depending on whether generic or brand-name drugs are used for schistosomiasis treatment. Using our overall best guess for the revised DCP2 estimate for STH of $138.28/DALY and the DCP2’s estimate for generic schistosomiasis drugs of $336/DALY (note that this is incorrectly presented as “$3.36/DALY” on page 476, but the correct figure – without the erroneous decimal point – appears on page 46), we estimate the cost-effectiveness of a combined program, according to the DCP2, as $177/DALY. Ignoring the long-term effects of STH treatment, as the DCP2 does, changes that figure to $272/DALY.

In our first email to the author of the spreadsheet, we had only caught the first four of the five errors mentioned above, and made substantial mistakes in our attempts to take long-term effects into account. It was only when we checked the figures later that we noticed both of these mistakes. Mistakes are easy to make in this type of situation (for an interesting study on spreadsheet mistakes, see here). Transparency is the best way we can think of to avoid such mistakes. Now that we’ve published the spreadsheets, we look forward to hearing about any other mistakes you find – in the original or ours.

Our independent estimate of the cost-effectiveness of STH treatment

At the same time we were working through the DCP cost-effectiveness estimate for STH deworming, Jonah Sinick, a GiveWell volunteer, was working on an independent set of cost-effectiveness estimates for deworming, separately for both STH and a second type of worm-based disease, schistosomiasis. His report on the results is now available here. His bottom-line best guess for the cost-effectiveness of STH deworming is $82.54/DALY. Jonah’s calculation implicitly takes long-term effects into account, as we do in our more optimistic version of the calculation (the one that comes to $138.28 per DALY). Most of the discrepancy between Jonah’s $82.54/DALY figure and our $138.28 figure can be explained by the DCP’s use of a much higher cost-per-child treated ($0.225 vs. $0.085), though Jonah also finds different levels of disease burden and treatment effectiveness. (See footnote 3 below.)

Jonah also found more promising results for schistosomiasis treatment, another form of deworming that (as mentioned above) can be combined with STH treatment. His estimate ranges from $28.19-$70.48/DALY for schistosomiasis deworming. This is much more optimistic than the DCP’s estimate of $336-$692/DALY because Jonah finds, following the current consensus in the literature, a much higher disability weight for schistosomiasis than the DCP used (0.02-0.05 vs. 0.005-0.006). The DCP’s higher cost-effectiveness estimate also assumes using much more expensive brand-name drugs, while the lower estimate, like Jonah’s, assumes generics.

Conservatively combining Jonah’s estimates for the cost-effectiveness of schistosomiasis and STH deworming (by assuming that no delivery costs are saved), we reach an estimate of $32-72/DALY, depending on the disability weight of schistosomiasis. More liberally assuming that a combined program would eliminate delivery costs equal to half the per-child cost of STH treatment, Jonah’s estimate of the cost-effectiveness of a combined program ranges from $29/DALY to $66/DALY, depending on the disability weight of schistosomiasis.

Implications for donors interested in deworming

These estimates are only a small part of the picture, in our view, regarding how promising deworming is as an intervention. We will be writing more about this in the future.

However, we think it is important to note that the DCP2’s original published figures implied that deworming is among the most cost-effective interventions listed in the publication; with errors corrected, it appears comparable to treating drug-resistant tuberculosis; taking into account long-term effects, it seems comparable to providing family planning services. Neither of those interventions are traditionally considered especially cost-effective. (Note that that according to the DCP2’s original estimate, STH deworming is 30-100X more cost-effective than those interventions.)

Whether or not the long-term effects are taken into account, the corrected DCP2 estimate of STH treatment falls outside of the $100/DALY range that the World Bank initially labeled as highly cost-effective (see page 36 of the DCP2.) With the corrections, a variety of interventions, including vaccinations and insecticide-treated bednets, become substantially more cost-effective than deworming.

The more important takeaway, for us, concerns the DCP2’s cost-effectiveness estimates in general. We believe that the errors we’ve found in the estimate – described above – would have been caught by a helminth expert independently examining the estimate. Therefore, the presence of these errors implies to us that there has been no such examination. If this is the case, it would argue against the reliability of the DCP2’s estimates in general. We have not done similar investigations of other DCP2 estimates, and given the process it took to get the details of this one, we are not planning to do many more until and unless the details of estimates become available publicly.

Our takeaways

  • We’re now much more hesitant to place any weight on DCP2 cost-effectiveness figures except where we can fully understand and check the calculations.
  • More generally, we feel this case illustrates how opaque, formal calculations can obscure important information and demonstrate high sensitivity to minor errors. We see this as support for our position that formalized cost-effectiveness analysis can do more harm than good in trying to maximize actual cost-effectiveness.
  • Explicit cost-effectiveness estimates will continue to play a relatively small role in our decisions between top charities, though we will still use them in deciding which charities are potential top candidates.
  • We’re continuing to investigate deworming as a promising intervention, but one of the most encouraging figures widely cited in its favor appears deeply flawed.
  • Transparency is crucial. Had the scholars we discussed these issues with been less willing to engage with us, or had we been unable to find Intestinal Nematode Infections or the spreadsheet, these substantial errors would not have come to light.

Footnote 1: The other two problems we found in the calculation both have to do with the burden of trichuriasis:

  • The spreadsheet swaps the disability weights for Type B and C symptoms of trichuriasis. In the Global Burden of Disease and Risk Factors (GBD) 1990, which the spreadsheet cites, the Type B symptom of trichuriasis is cognitive impairment, which has a disability weight of 0.024, while the Type C symptom is massive dysentery syndrome, with disability weights ranging from 0.116 to 0.138. In the ‘trichuriasis’ sheet of the spreadsheet, Type B morbidity has disability weights ranging from 0.116 to 0.138 while Type C morbidity has the lower disability weight of 0.024. In the original calculation, this leads to an overestimate of the burden of trichuriasis by nearly 4x, but once the main errors described above are corrected, correcting this error actually makes STH treatment appear more cost-effective.
  • The spreadsheet uses a duration of .05 years for trichuriasis symptom Type C, while Intestinal Nematode Infections suggests that the duration for trichuriasis symptom Type C should be 12 months (pg. 24). This mistake likely occurred because the duration for ascariasis symptom Type C is .05 years.

In the corrected spreadsheet, sheets ‘a.3’, ‘t.5’, and ‘h.3’ contain our corrections to all five of the issues we have identified (for ascariasis, trichuriasis, and hookworm respectively). Most of the corrections should be fairly self-explanatory, but please don’t hesitate to email us or comment here if you have questions. We corrected the second main error above by changing the population of 5-14 year olds treated to 1,000,000 (see, e.g., sheet ‘a.3’ cell C23).

Footnote 2: The Type B symptom of all three diseases treated by STH deworming is called “cognitive impairment,” has a disability weight of 0.024, and lasts a lifetime once it develops. Intestinal Nematode Infections implies that 3% of the population at risk for symptom B (that is, 3% of the population listed in the A/B columns in Table 9) newly acquires a lifelong disability each year (pg. 26). We therefore altered the calculation to reflect lifelong (not just 1-year) benefits for these 3% (replacing the 5% listed in #2 above because that 5% is the total proportion infected during a given year, not the total proportion newly infected). At the same time, we also changed DALYs saved due to prevented mortality to compound to the end of life, rather than just counting the one year of life saved during the treatment. (This, arguably, is an actual error in the DCP2 process, not just a disagreement about how to take long term effects into account. When an intervention prevents someone from dying, it does not seem reasonable to count just one extra year of life saved.)

Footnote 3: We also looked into the possibility that the disability weights for helminth infections are “too low,” as implied by a passage in the DCP2:

The Disease Control Priorities Project helminth working group has determined that the WHO global burden of disease estimates are low because they do not incorporate the full clinical spectrum of helminth-associated morbidity and chronic disability, including anemia, chronic pain, diarrhea, exercise intolerance, and undernutrition (King, Dickman, and Tisch 2005). (DCP2, pg. 471)

Based on our review of the literature and correspondence with relevant scholars, we believe this argument has never been raised specifically in respect to STHs; most of the papers about it are about schistosomiasis, another type of worm infection. There is one paper (Chan 1997) that appears to imply a higher disability burden for STHs than the standard burden, which gives rise to Jonah’s more optimistic STH cost-effectiveness estimate of $11.25/DALY. We think the data from that paper is no longer credible: it appears to have been based on a lower worm threshold for experiencing morbidity than further research has found appropriate (Brooker 2010). Furthermore, the cited source of the relevant data is a working paper, the published version of which does not contain the data cited.

GiveWell Labs: Plans for our process and transparency

(updated on: ) | by Holden

[Added August 27, 2014: GiveWell Labs is now known as the Open Philanthropy Project.]

We’re starting a new initiative, GiveWell Labs, an arm of our research process that will be open to any giving opportunity, no matter what form and what sector.

One of the major challenges of this initiative (as mentioned in the previous post) will be remaining systematic and transparent despite the very broad mandate of GiveWell labs. It’s core to GiveWell that the thinking behind our recommendations

  • Comes from reasoning and principles that are applied consistently, not from whims.
  • Is transparent, i.e., interested people can read up on why we made the decisions we did and judge our thinking for themselves, with as little need as possible to have trust in us.

This post lays out our plan for accomplishing these.

Process
While we reserve the right to change plans mid-stream, our basic planned process is:

  1. Sourcing general ideas. We plan to cast a wide net initially, looking pretty much anywhere we can for general funding ideas – and/or organizations – that might be promising. Key sources will include:
  2. Going from general ideas to specific proposals. We will maintain a ranked list of the most promising ideas from step 1, and for high-ranked ideas, we will attempt to find the people and/or organizations who can make (and, potentially, execute on) specific proposals. At this stage we’ll just be looking, in each proposal, for rough ideas of (a) costs; (b) what people/organizations will do the execution; (c) what the basic plan is.
  3. Detailed investigation of proposals. We will maintain a ranked list of the most promising proposals from step 2, and for the most promising ones, we will conduct in-depth investigations similar to those we’ve always conducted for promising charities. These will include in-depth conversations with the relevant people/organizations; conversations with others in their space, particularly those who have funded them or chosen not to fund them; site visits when applicable; and requesting technical reports, budgets, and other materials when applicable.
  4. Recommending and funding proposals. We will attempt to get any outstanding submissions from step 3 funded. We will have $1 million to spend at our discretion if we can raise no other funds, but we expect to be able to raise more if we succeed in finding great giving opportunities.

At this point we are most interested in funding others’ ideas, and have a preference for cases where the implementing organization is the same as the organization that hatched the idea and strategy. We have the impression that much philanthropy works differently, as foundation staff design their own strategies and treat grantees to some extent as “contractors” for carrying it out; this model does not currently appeal to us, but we plan on further investigating the history of philanthropy (particularly success stories) to see whether there is more promise in this approach than we’d guess.

Transparency
With a project as broad and open-ended as GiveWell Labs, we expect to make a lot of guesses and judgment calls regarding promising areas/ideas/projects, and we expect to use heuristics and take shortcuts in narrowing the field. We don’t commit to detailed reviews of every idea or every proposal, or to the use of objective formulas to decide between them. (The same applies, and always has applied, to our existing research on top charities.)

However, a core value of ours is that interested parties – no matter who they are – ought to be able to understand as much as possible of (a) which options we considered; (b) why we chose the ones we chose. To this end, we plan on publishing:

  • Extensive discussion of the values and beliefs that are relevant to which sorts of sources we use and which areas we focus on investigating. This discussion will take place via future blog posts. We hope that anyone who reads these posts will understand why we look at the areas and sources we do, and if we aren’t accomplishing this we hope our readers will comment to let us know.
  • A list of the sources we use to generate ideas (step 1), along with detailed notes from particularly noteworthy conversations. Our goal here is to cast the net wide, so if you know of promising sources of ideas that fit with the values/principles we’re expressing and you don’t see us using them, we encourage you to comment.
  • Discussion of the general beliefs (and relevant facts) that lead us to discard certain ideas from step #1 while moving forward to step #2 on other ideas, again via the blog.
  • A full list of the proposals we consider (step #2), along with notes from discussing these proposals.
  • Discussion of the general beliefs (and relevant facts) that guide our choice of particular proposals (step #2) to move to the “deep investigation” phase (step #2), again via the blog.
  • Full details of the materials we collect via deep investigation (step #3) and our notes on the strengths and weaknesses of each giving option that makes it to this stage.

We will withhold information when necessary to respect confidentiality agreements. However, we will make our best effort to obtain clearance for – and share – all important/relevant information. This is the same policy we’ve used in charity investigations, and while some information remains confidential, we’ve still published the vast majority of the information we have (enough so that our views generally don’t need to be taken on trust).

Preserving GiveWell’s core values
GiveWell Labs is different in substantial ways from our existing research (which continues). However, we feel that we will be able to preserve the most important aspects of GiveWell:

  • A focus on finding the best giving opportunities in terms of positive impact, rather than in terms of telling compelling stories or making donors feel good.
  • Recommendations that are transparent enough to allow outsiders to draw their own conclusions and give meaningful feedback.

If we can preserve these things while working in a more open-ended way, we’ll be able to find better giving opportunities and to demonstrate our principles’ broad applicability. This means there will be fewer reasons than ever for other large givers to be keeping their own processes opaque.

Why GiveWell Labs?

(updated on: ) | by Holden

[Added August 27, 2014: GiveWell Labs is now known as the Open Philanthropy Project.]

We previously announced GiveWell Labs, a new arm of our research process that will be open to any giving opportunity, no matter what form and what sector. Here I share a bit more of the thinking behind why we’re doing this.

What we’re trying to accomplish with this initiative

Our goals are twofold:

Find better giving opportunities. When we laid out our main goals for 2011, #1 was finding more great giving opportunities, and our possible strategies for doing so involved (a) broadening our scope (b) considering project-based funding. With GiveWell Labs, we are doing both simultaneously.

  • We’ve previously come across groups that might have been able to offer great giving opportunities, if we had selected a specific project and provided all the funding necessary to carry it out. However, we couldn’t recommend them to individual donors, not knowing whether $1000 or $1 million would come in as a result. Now, we’ll plan to go back to these groups, open to anything. If we do end up wanting to raise specific amounts of money, this will be a more complex endeavor than simply publishing a recommendation on our website and saying “Give here,” but we now have enough connections to major donors and enough sense of our audience of smaller donors that we think it will be worthwhile to try.
  • We’ve previously come across interesting funding opportunities that didn’t fit neatly into the causes we had chosen to focus on. This won’t be an issue for GiveWell Labs.
  • Examining opportunities with the above qualities (project-based and/or outside the sectors we’re experienced in) will be hard to do systematically, and will be by nature a bit experimental. That’s why we’re allocating only 25% of our research time to GiveWell Labs, with the remainder allocated to carrying out our existing research process (which has some restrictions but is more established and systemized). However, we expect the things we learn through GiveWell Labs to eventually shape the evolution of our more systemized research process.

Position ourselves to advise seven-figure donors.

When analyzing our own impact, we’ve noted that it comes disproportionately from large donors. (We influence more $100 donors than $10,000 donors, but the ratio is far under 100:1, so the $10,000 donors end up accounting for the lion’s share of our money moved.)

This seems logical to us, when considering that GiveWell is a “niche product” – we don’t appeal to large interconnected groups of people, but the rare people who do resonate with our work resonate very strongly with it, and give a lot based on it. The logical implication is that our greatest potential for impact may come from very large donors – and we need to be positioned to be useful to these donors.

The research we’ve done to date – recommending direct-aid charities that can absorb arbitrary amounts of funding – seems best suited to those giving under $1 million per year. When we encounter people who give more, they generally are interested in funding whole projects at once, which gives them options that simply aren’t open to our standard research process. That means our current product is a poor fit with the people who may represent our most potentially impactful audience.

We need to address this issue, and GiveWell Labs will allow us to do so. The $1 million in pre-committed funding is coming from large donors who will be able to give more if we find them great opportunities. More importantly, GiveWell Labs will allow us to move closer to having the same universe of options that seven-figure donors have, which will hopefully improve our ability to connect with and influence seven-figure donors.

Pros and cons of issue-agnostic giving

GiveWell Labs is issue-agnostic, i.e., we are not restricting our work to particular areas of philanthropy (such as international aid, climate change, etc.) We will focus on what we consider the most promising areas, but we will be potentially open to anything.

There are clear disadvantages to issue-agnostic giving:

  • The more different sorts of projects we allow ourselves to consider, the greater the challenge of sorting through them in a coherent, principled, systematic way. It will be particularly challenging to make sure we are applying principles consistently, rather than giving based on whims.
  • There are conceptual advantages to “specializing” in particular sectors over time. Doing so means having the ability to
    • Learn from past successes and failures in an area.
    • Make contacts in an area.
    • Gather evidence about the most promising approaches in an area, particularly informal/qualitative evidence (e.g., site visits).

However, issue-agnostic giving has advantages as well.

  • First and foremost is that when you’re new to giving, you can’t tell where the best opportunities are going to be. Picking a “sector” could be the dominant determinant of how effective your giving will be; taking a guess and sticking with it, therefore, seems very dangerous for a donor seeking to maximize impact. (Note that I’ve changed my view of the most promising cause as I’ve learned more about the different causes we’ve studied.)
  • Even when you’re not new to giving, the highest-impact sectors can change rapidly and chaotically as new philanthropists come on the scene. Choosing to focus on developing-world-oriented medical research may have been a great idea before the Gates Foundation came along, but I’m guessing that opportunities in this area have fallen drastically since, as the Gates Foundation has attempted to fund the best ones.
  • There may be outstanding opportunities that get overlooked by other funders because they don’t fit neatly into a particular “sector.” I think this is possible in today’s environment, simply because issue-agnostic giving is so rare.
  • Regarding the above-mentioned advantages of specialization:
    • We are hoping for a relatively “low-touch” approach to funding: we seek people with ideas but not funding, and we seek to provide funding and not other kinds of support. We hope this approach will diminish the need for us to become “experts” in any given sector.
    • We hope to be very communicative with other funders and people with relevant expertise/experience. We won’t recommend a funding opportunity without getting as many relevant opinions as we can. If people with different specialties are open and communicative with each other, it mitigates the need for funders to have all the expertise themselves.
    • In practice, we will probably find ourselves focusing on certain sectors – not out of a pre-commitment, but because these sectors appear particularly promising to us. This is especially true in light of the fact that we prefer (as we always have) to fund things we can understand well – our past experience and established knowledge do matter. So being issue-agnostic doesn’t actually preclude specializing; it just means that any specialization will happen gradually and out of a desire to maximize impact, rather than being driven by up-front choices of particular sectors.

Bottom line – at this point in our development, we think the advantages of issue-agnostic giving outweigh the disadvantages for us.

Announcing GiveWell Labs

(updated on: ) | by Holden

[Added August 27, 2014: GiveWell Labs is now known as the Open Philanthropy Project.]

The research we’ve been doing for the last couple of years has been constrained in a couple of key ways:

  • We’ve pre-declared areas of focus (based on our guesses as to where the most promising charities would be found), and disqualified charities for recommendations on the basis of their being “out of scope” (though we’ve been gradually broadening our scope).
  • We’ve needed to decide which organizations to recommend without being able to say in advance how much money would go to them as a result. This has led to challenges with the question of “room for more funding.” We’ve had to find charities that could essentially use any amount of funding (large or small) productively, and this has drastically narrowed our options.

We’re now launching a new initiative within GiveWell that will not be subject to either of these constraints. We plan to invest about 25% of our research time in what we’re calling GiveWell Labs: an arm of our research process that will be open to any giving opportunity, no matter what form and what sector.

Through GiveWell Labs, we will try to identify outstanding giving opportunities (whether they’re organizations or specific projects), publish rankings of these giving opportunities (separate from the top charities list we maintain using our existing research process) and try to raise money for these opportunities. Donors have pre-committed a minimum of $1 million to the GiveWell Labs initiative, meaning that we will have at least $1 million to commit to our choice of projects even if we are able to raise nothing else. (We expect to raise more if and when we find great giving opportunities; the $1 million has been committed based on donors’ trust in our ability to find such opportunities.)

Our existing work of finding outstanding international aid charities – using a more systematic process – continues. Over the coming year, we expect to spend about 75% of our research time on our existing work of finding outstanding international aid charities, and 25% of our research time on GiveWell Labs. Note that our “standard” process continues to gradually evolve and broaden its scope, and hopefully will come to incorporate insights gained through the work on GiveWell Labs. The distinction between the two may even dissolve over time. But at this time, GiveWell Labs is the arm of our process that is open to any giving opportunity, no matter what form and what sector.

In future blog posts, we’ll be giving a lot more information about this project, including:

  • More on why we’re moving in this direction at this time, and why we think a less-constrained, exploratory arm of our research process will help us find better giving opportunities.
  • Our planned process for finding great giving opportunities through GiveWell Labs, and what you can expect from us in terms of transparency.
  • The main qualities we’re looking for in a funding opportunity (when unconstrained by the form or sector of the opportunity), and why we’re looking for them.
  • The areas we think are most likely to yield great giving opportunities, and why.

In the meantime, if you know of any giving opportunities that are (a) not already funded or likely to be funded by others; (b) outstanding opportunities to have a large positive impact, please let us know.

Somalia famine: Update

(updated on: ) | by Elie

Over the past month, we’ve worked to understand the situation in Somalia and make a recommendation to donors about where they should give. At this point we’re wrapping up our work with the following conclusions:

  • We wouldn’t recommend giving to support Somalia specifically over supporting everyday aid. While the needs are extreme, we aren’t convinced that individual donors can effectively cause more aid to be delivered via their donations.
  • For those who do want to give, we suggest The International Committee of the Red Cross (ICRC), the World Food Programme (WFP) or Doctors Without Borders (MSF), but with serious reservations about each of these.
  • There is a severe lack of transparency on the part of charities and funders, particularly the US government, that has hindered our ability to understand the situation and make a strong recommendation.

A brief note before getting into the details: most of this work was done by our summer intern, Josh Rosenberg. We couldn’t have learned as much as we did about this crisis without his help. Thanks, Josh.

Details follow.

Giving to Somalia relative to giving to everyday international aid

We believe strongly that the needs inside Somalia are great and that, were it possible to send food or medical supplies such that it would reach people in the region, that would accomplish a great deal of good. We recently spoke with an American journalist in the region, and he said:

I went to a hospital in Somalia recently, and we saw kids in very bad shape. There are just no resources there. They don’t have medicine, IV bags, solution. There are dozens, if not hundreds of people arriving each day that need hospitalization. It’s the same situation with camps inside Somalia.

Nevertheless, we don’t have confidence that it is in fact possible for donors to help get more food and supplies to those who need it. There are numerous reports of World Food Programme food aid being stolen by al-Shabaab, a group classified as a terrorist organization that governs much of the famine zone, and many Western NGOs have been explicitly banned from the region by al-Shabaab or have chosen to leave due to security concerns.

Even for the groups operating in the famine zone, it’s not clear to us that additional funds donated will lead to additional services provided, for reasons of donations’ fungibility. Our guess is that given the dire circumstances, aid organizations are likely to spend whatever resources they can, unrestricted or otherwise, to reach those in need, and individuals’ providing additional funds to the few organizations active in the famine zone may make little difference to this specific relief effort.

That said, we have still had limited contact with organizations operating inside famine zones, and we would be interested in hearing from any if they feel they would reach additional people with additional funding. Were we to gain confidence that an organization could do this, we could plausibly view the donation opportunity as on par with our highest rated organizations.

Assuming one wants to give to Somalia, which organization will be most effective?

We struggled to obtain substantive credible information to help us answer this question well. In our previous Somalia update, we listed the questions we sought to answer, and in most cases, we received vague information from charities such that we weren’t able to answer our questions well. For example, we often received budget proposals such as “$5 million for water and sanitation projects” with no detail regarding (a) the type of projects (e.g., digging wells vs trucking water vs purifying water) to be implemented, (b) the cost of each project component, or (c) the location where the projects would be implemented.

Given the lack of substantive, credible information, the three factors we focused most on are:

  1. Where is the organization operating? While we believe that there are great needs throughout the region–inside the famine zone, in the rest of Somalia, and in refugee camps and mainland areas of Ethiopia and Kenya–the greatest needs are in the famine zone. In our conversation with a journalist in the region, we asked “How do the conditions in refugee camps in Kenya and Ethiopia compare to conditions inside Somalia?” and he responded:

    My impression is that the refugee camps are pretty well taken care of right now. Even though they’re burgeoning with people, they’re doing OK. There’ve been some disease outbreaks in Ethiopia. They could use more help but there’s already a huge infrastructure there. In Dadaab there’s a huge compound for western aid workers. There’s a bar and restaurant. In Somalia, the people are near death and have no access to resources… I went to Dadaab, and I saw the same thing and saw starving kids and poor families, but there were people driving CARE cars and wearing MSF badges or Save the Children hats, so there are NGOs in the camps, but there’s no help inside Somalia.

    That is, while the people who reach the refugee camps need assistance, they are being served and we don’t have enough information to say that there is room for more funding in the camps.

  2. How transparent is the organization about its activities and spending? Regarding Somalia, but also other disasters, we’ve found limited information about the impacts or results of charities’ programs. We’ve therefore focused on organizations’ transparency and openness to being held accountable for their activities as a proxy for the organizations that are likely to be most effective.
  3. What other information do we have about the organization that would inform our conclusion about where to give now? To the extent we’ve considered the organization in other contexts, we’ve incorporated any additional information into our views here.

Having completed our analysis, three organizations stood out.

  • The International Committee of the Red Cross (ICRC). The ICRC is appealing for funds solely for use in the famine zone, and our understanding from them and from the journalist we spoke to is that they are active in famine areas. ICRC gave us a uniquely detailed plan for scaling up and using funds. Unfortunately, we aren’t cleared to share this plan publicly, but it was a more comprehensive and detailed plan than we received from other charities.However, the plan did not allow us to easily connect what ICRC plans to do with how it would spend money. Also, the journalist we spoke with told us:

    They are working in South Somalia in the al-Shabbab areas where no one else is. But, I’ve been told by some people that they screwed it up for other aid groups because they paid al-Shabbab a tax/bribe to work in those areas, and then al-Shabbab demanded it from other groups. Because al-Shabbab is a designated a terrorist organization by the US government, aid groups had to leave because it wasn’t legal for them to pay money to al-Shabbab. So, while ICRC is doing good work, there’s some resistance to them from other NGOs.

    We have not verified this claim or questioned the ICRC about it.

  • The World Food Programme (WFP). WFP is the only organization we spoke with that makes its detailed reports publicly available on its website. The reports include detailed budgets as well as quantities of food to be delivered and targeted locations. In addition, WFP is one of the largest entities (if not the largest entity) operating in the region, and they have been criticized in the media for mistakes they’ve made. Other things equal, we feel donors are well served to support the groups that will ultimately be seen as “responsible for” the response because they are most likely to be held accountable by donors and the public. Note that al-Shabbab has denied access to WFP in areas it controls.The criticisms that have been made raise room for concern as well, particularly regarding reports of World Food Programme food aid being stolen by al-Shabaab.
  • Doctors without Borders (MSF). We’ve spoken several times with MSF, but have received limited information from them. MSF is operating in the famine zone. We maintain our generally good feelings about the organization, but this is based largely on MSF’s transparency about their activities and needs for donations in past disasters. We are disappointed in MSF’s lack of transparency in this case.

The journalist we spoke with also mentioned some other organizations. We don’t endorse these but include his comments here for those interested:

I interviewed one guy and came across one NGO I thought was doing decent work and one of the few that had American people on the ground. It’s the American Refugee Committee. They’re pretty small and maybe their smallness has helped them be more nimble. He has gone to Mogadishu, and come out and gone back in. A few people have asked me whom to give to and I said that I had seen the American Refugee Committee. That was one of the few western organizations working on the ground. I also saw IRC, the International Rescue Committee. I have some friends in the aid business, and they’ve told me that IRC is there. They’re trying to provide help at camps and hospitals….There is a local NGO in Somalia that is doing good work. It’s called the Hawa Abdi Foundation. There’s a Somali woman doctor who set up a clinic in a camp, and she’s helping a lot of people. She was named one of Glamour Magazine’s top ten women of the year. If you steer any donors to local groups, it’s a good one. She has a track record of doing good work and reaching people. World Vision has done some good work inside Somalia. They were run out but are now starting again. In North Kenya, they’ve done good famine prevention work and set up agricultural projects that are helping people in these drought areas become farmers and less reliant just on cattle. I’ve looked closely at the World Vision and they’re pretty brave. They’re working in areas no one wants to go to.

Our struggles and the lack of transparency

One of the most disappointing aspects of our Somalia research has been the opacity of charities and the US government. We are particularly disheartened by USAID’s consistent position that they cannot help us, or by extension, individual donors in any meaningful way.

Over the past month, we have spoken with several representatives at USAID, all of whom have told us the same thing:

  • We cannot comment on or off the record about specific aid organizations.
  • We cannot offer any advice about which organizations are likely most effective or have the greatest need for funds.
  • We cannot share any of the information we’ve received from organizations about what they’re doing or how they’ll spend money.

While there may be specific cases of documents that must be kept private for safety or privacy concerns, we feel that most information can and should be shared. We’ve seen USAID documents in some cases because charities have voluntarily sent them to us, and we haven’t seen anything about these documents that would clearly cause harm if shared more widely. Generally, when charities have asked us to keep information confidential (which we’ve honored), we’ve seen little that seems it would cause harm or danger if shared publicly; confidentiality concerns have seemed to have more to do with charities’ not wanting to be judged in certain ways.

USAID is a government agency funded by the public. USAID has significant, detailed information about NGOs’ activities around the world, and sharing this information publicly would provide significant help to donors aiming to give more effectively. USAID has told us that the information they receive from charities is private and confidential and cannot be shared. This conclusion does not seem valid or just to us.

Donors who care about impact should continue to pressure the charities they support and the US government, which provides funding to them, to be more open with their information.

Working for GiveWell

| by Holden

If you’re interested in working or volunteering for GiveWell, now would be a good time to let us know. We’ve been a 7-person team for the last couple of months, but since two of the hires were temporary, we’re soon going to be back down to 5. We were happy with our productivity when we had 7 people; we have the funds, the management capacity and the desire to get back up to that size if the right people come along.

About the role

We’re looking primarily for Research Analysts – people who will provide support to the goal of finding the best giving opportunities. Research Analyst duties mostly consist of:

  • Reviewing independent research on the best ways to help people and on other issues relevant to giving
  • Reviewing particularly promising charities – including speaking with their representatives and asking critical questions, reviewing and evaluating documents they send, and writing up their answers to critical questions, strengths, and weaknesses
  • Taking part in discussions of which giving opportunities are most promising and of general GiveWell strategy
  • Miscellaneous duties depending on individual preferences, including networking, outreach, writing (e.g., for the blog), and original analysis on research questions
  • We encourage analysts to push their abilities to the limit and take on as much responsibility as they can. An analyst can grow into a major role at GiveWell.

A few practical details on the role:

  • We are located in New York City and currently work in the Tribeca/Chinatown area. Hours are flexible and some telecommuting is allowed, though overall expectations for productivity are high.
  • The general environment is one of intense discussion and debate. We change course and rethink things frequently, and analysts are encouraged to challenge, question and critique their managers.

What we’re looking for

We believe the most important qualities for a Research Analyst are

  • Passion for finding great giving opportunities. There’s little precedent for the kind of work we do, and we can’t train people to the point where little of their own judgment is required. As a result, analysts end up making a lot of judgment calls and it’s important that these judgment calls be oriented toward finding great giving opportunities.

    In the past, we’ve found that the best employees are the ones who come to us looking to volunteer or work for us, demonstrating pre-existing interest in and passion for the project. That’s why we’re starting this search via our own blog – we think the most promising candidates are likely to be among our readers.

  • Critical thinking/analysis skills. Analysts need not have existing proficience with data analysis (though it’s a plus), but they do need to be able to approach claims about charities’ impact with skepticism and good critical questions – whether those claims are made by charities, scholars, or myself and Elie.
  • Attention to detail. Analysts need to do careful, reliable work whose conclusions we can trust.

How to apply

Email us with a resume and a note on why you’d like to work for GiveWell. We will most likely write back asking you to enter our volunteer process; we generally ask people to volunteer for us before being hired, so that we can get a strong sense of their fit with the organization.

We also appreciate referrals to people who might be a good fit.