Update 12/5/2014: we update our cost-effectiveness models annually. The most up-to-date versions can be found here. 

This post discusses how we see the relative “bang-for-the-buck” – good accomplished per dollar spent – of three interventions:

• Distribution of insecticide-treated nets to fight malaria, the intervention carried out by our #1 charity (the Against Malaria Foundation).
• Unconditional direct cash transfers, the intervention carried out by our #2 charity (GiveDirectly)
• Treating children for parasites (soil-transmitted helminths and schistosomiasis), the intervention carried out by our #3 charity (the Schistosomiasis Control Initiative)

We discuss:

• Our general philosophy of cost-effectiveness. We find the exercise of cost-effectiveness useful, and we care greatly about large robust differences, but we also believe that cost-effectiveness estimates involve substantial judgment calls and shouldn’t be taken literally.
• Simple “cost per person per year” measures. Deworming is the cheapest of the three interventions in terms of “total costs to treat one person for one year,” so other interventions must be more beneficial (on a per-person-per-year basis) in order to be an equally good buy from a donor’s perspective. Deworming and nets are in the same ballpark on this basis, and cash transfers that are spent on metal roofs could be seen as being in the same ballpark as well.
• Cost per “equivalent life saved.” We can try to convert all the benefits of the different interventions into “equivalent lives saved” (or, alternatively but similarly, into DALYs). We have not attempted this for cash transfers; when doing so for deworming, we run into many judgment calls and an extremely wide range of values, from $31 to $25,000 per “equivalent life saved.” Estimating “lives saved” for bednets is more straightforward, though it involves judgment calls as well. Different people have significantly different intuitions about the right inputs into these figures, even within GiveWell staff. We provide estimates based on the guesses of the three staff members who have worked most intensively on cost-effectiveness, and may be providing other staff members’ estimates in the future. These estimates diverge widely from each other, though in all cases, that nets and deworming are estimated to be in the same ballpark, with a slight edge for nets.
• Financial returns. The primary benefit we’ve seen evidence for, for deworming, is improved earnings for people dewormed in childhood. We can estimate the net present value of these benefits to arrive at a figure along the lines of “For each $1 spent on deworming, recipients receive total benefits equivalent to $X.” Following this approach allows a comparison to cash transfers. Again, the outcome of the comparison depends on many significant judgment calls, and different staff members produce very different figures. The general picture here is that deworming is between 2 and 5 times as cost-effective as cash-transfers.

We encourage readers who find formal cost-effectiveness analysis important to examine the details of our calculations and assumptions, and to try putting in their own. To the extent that we have intuitive preferences and biases, these could easily be creeping into the assumption- and judgment-call-laden work we’ve done in generating our cost-effectiveness figures, and we’re not entirely confident that the figures themselves are adding substantial information beyond the intuitions we have from examining the details of them.

We have put a great deal of time into our formal cost-effectiveness analysis, and we think our results simultaneously illustrate (a) the value of cost-effectiveness analysis, in terms of highlighting and causing us to examine key assumptions and judgment calls (as well as potentially clarifying what our intuitions imply about which interventions do the most good); (b) the limitations of cost-effectiveness analysis, in that it cannot practically be made robust for many comparisons (including the comparisons in this post), and is only as good as the intuitions and judgment calls that go into it.

• The driving aim of our charity recommendations is to help donors accomplish as much good as possible, on a per-dollar basis.

• Defining “good” necessarily involves judgment calls, and measuring/estimating it involves more judgment calls. We generally look at cost-effectiveness from multiple angles and publish multiple versions of our estimates, and we encourage donors to do their own critical thinking.
• We think an appropriate role for our analysis is to help clarify the decisions donors are making, without necessarily quantifying every aspect of the decision. An analogy we sometimes use is that of deciding whether to buy an apple or an orange: ultimately the decision is subjective, but having certain facts – namely, the price of each – can clarify the decision and make it more informed. In this case providing a “cost per apple” and/or “cost per orange” would be helpful, while providing a “cost per unit of food-enjoyment” (with the calculation of “food-enjoyment per fruit” based on guesswork) would likely be less helpful.
• The less robust a cost-effectiveness estimate, the less weight we should place on the estimate in giving decisions. The conceptual goal of accomplishing the most good per dollar spent does not necessarily entail giving to the charity with the highest explicitly estimated “good accomplished per dollar spent”; there may be factors that are best dealt with outside of the “explicit expected value” framework. (More at our post on the subject and the comments that followed, particularly this comment).
• We see two potential sources of value in doing explicit cost-effectiveness analysis: (a) finding relatively large and robust differences between different charitable interventions; (b) using cost-effectiveness analysis as a way to ensure that we have carefully thought through the relevant issues.
  Simple cost-effectiveness measuresOne way to start thinking about the relative cost-effectiveness of the three interventions is to think about the “cost per person-year of coverage,” i.e., the cost of serving one person with the intervention for one year.
  • For LLINs, we estimate a cost of $1.36 per “person protected by an LLIN per year.” (This is based on $5.15 per LLIN distributed, with each LLIN covering an average of 1.8 people for an average of 2.22 years; details at the spreadsheet linked from our cost-effectiveness analysis of LLINs, column H).
  • For deworming, we estimate a cost of $0.51 per person treated (details). This $0.51 figure incorporates costs that SCI, our top-rated deworming charity, reported to us. We have not fully investigated all costs associated with deworming programs (as we have for bednets and cash transfers) and believe the cost per person treated could be ~10-25% higher, leading to a cost per person treated of $.51-.64.In high-prevalence areas, deworming is annual, and all of the information we have about the impacts of deworming is based on high-prevalence areas. There is an additional question of what percentage of people treated are children, since the main case for deworming (in our view) is its developmental impacts when applied to children; we have little ability to estimate this figure and generally assume that 50% of those treated are children, leading to a figure of $1.02-$1.28 per child treated.
  • Cash transfers are much more difficult to model on a “per person per year” basis, since they are structured as one-time “wealth transfers.” One way of coming up with a partially informative “per person per year” figure is to look at the cost of a metal roof, which is a commonly reported use of cash transfers. GiveDirectly estimates that such a roof costs about $250 for a household and lasts about 20 years. During our visit to GiveDirectly’s operations in Kenya, recipients reported spending closer to $500 for a roof. At an average household size of 4.7 (from the “Household size analysis” document in our GiveDirectly review) and 20 years per roof, that implies about $2.66-$5.32 in recipient expenditures per “person-year of roof coverage”; at a rate of 90c transferred for each $1 in expenses, that implies about $3-$6 in donor expenses per “person-year of roof coverage.”

  The obvious problem with these figures is that we don’t know how beneficial a year of LLIN coverage is relative to a year of deworming or a year of coverage by a roof. However, estimating such benefits involves a lot more judgment calls than estimating the costs. We think the figures listed above – while inconclusive and unsatisfying – are much less likely to be wildly inaccurate on their own terms than the figures that follow, so we think it’s appropriate to keep them in mind.

  Humanitarian outcomesA much more satisfying comparison, but one that is much more difficult to do with precision, is to estimate the “humanitarian value per dollar” created by each intervention. In order to do this, we need a unit of “humanitarian value” that we can standardize on. One such unit is the disability-adjusted life-year (DALY). A similar unit, which we find easier to think about but which is fundamentally similar to the DALY (and relatively easily convertible to it), is the “life saved equivalent.” For some interventions (those with direct impacts on mortality) we can estimate a “cost per life saved”; we can then try to estimate non-mortality-related benefits of interventions, decide how much we value these benefits relative to “lives saved,” and then convert them into “lives saved equivalent.”

  The cost per life saved for LLIN distribution

  Our current best estimate of the “cost per life saved” for LLIN distribution is about $2300 (details at the spreadsheet linked from our cost-effectiveness analysis of LLINs, column H). Some notes on this figure:

  • This figure is only including direct lives saved for children under five.
    • It is also possible that LLINs save adult lives, but figures on adult malaria deaths are disputed and we have not seen studies addressing whether LLINs save adult lives.
    • As discussed previously, we believe the case that LLINs have non-mortality-related “developmental benefits” is somewhat comparable to the case that deworming has such benefits.
    • Finally, LLIN distribution may reduce the burden of malaria, on LLIN users and on the health system, in ways not captured in the above considerations.
  • Our spreadsheet also estimates the cost per life saved under different assumptions about questions like “How long do LLINs last in the field?” and its overall range comes out at ~$1700-$5500.
  • There are several potential major sources of uncertainty in our estimate that are not captured in our spreadsheet, including the possibility of insecticide resistance, the possibility that today’s conditions differ in other ways from the conditions under which the original studies of mortality effects were done, and more. These are listed in our discussion of LLIN cost-effectiveness.

  The cost per “equivalent life saved” for deworming

  We see a lot of uncertainty around the benefits of mass deworming. We have not seen evidence that it directly saves lives, and any such lives saved are likely to be quite rare (not competitive with how often LLINs save lives). However, there is some evidence suggesting that deworming has developmental impacts: that deworming someone in childhood can cause them to earn more money later in life (among other benefits).

  When attempting to estimate a “cost per life saved equivalent” for deworming, one must take a view on multiple difficult-to-quantify factors, all of which are listed in the “assumptions” sheet of our spreadsheet on “cost per life saved equivalent” comparisons:

  • The relative value of saving a life vs. realizing the developmental benefits associated with deworming. The value assigned by the corrected Disease Control Priorities Report to averting “cognitive impairment” is 2.4% of the value it would assign to averting death, but many might consider improving a life to be more than 2.4% as valuable as saving a life. (Assumption 9 on “Assumptions” sheet)
  • The proportion of people dewormed by SCI who are children. (Assumption 2)
  • Whether the benefits of deworming are likely to scale linearly with repeated treatments. The Kenya study on developmental benefitslooks at the benefits of receiving ~2.5 years of additional treatment; it’s possible that people treated by SCI receive fewer or more years of treatment. (SCI generally aims for repeated treatment throughout childhood.) (See Assumption 3; we formalized this issue via a multiplier that captures “how helpful years of treatment in SCI’s program are, relative to the years of treatment in the key study.”)
  • Whether to make a “replicability adjustment” due to the fact that developmental benefits for deworming are not as robustly established as mortality effects for LLINs. (We have discussed this issue previously.) We have sought a reference point for how likely it is that a given study result will hold up under replication, and have seen some potentially relevant figures in John Ioannidis’s analysis of biomedical literature (here and here), though the analogy between biomedical studies and the studies in question has substantial limitations. We have formalized this adjustment as Assumption 4 in the sheet.
  • Whether to make an “external validity” adjustment, to account for the fact that infection prevalence rates were unusually high due to El Nino in the Kenya study on developmental benefits (the study that provides the most helpful information for quantifying the benefits of deworming). We have formalized this adjustment as Assumption 1 in the sheet.
  • How to account for the shorter-term health benefits of deworming, including both rare severe health effects (such as intestinal obstruction due to ascariasis) and subtle general health effects. Given recent developments, we think it is reasonable to consider the subtle general health benefits negligible, but we also provide the option to use our previous estimate of such benefits, which is based on the Disease Control Priorities Report.

  Our spreadsheet on “cost per life saved equivalent” comparisons shows the “cost per life saved equivalent” under different assumptions. We have provided many possible assumptions in the “assumptions” sheet and calculated the “cost per life saved equivalent” for deworming under each possible combination of inputs; it ranges from $31 in the most optimistic scenario to over $2 million in the most pessimistic. It also includes scenarios that represent different GiveWell staff members’ best guesses, discussed further below.

  Comparing LLIN distribution and deworming

  Recall that LLIN distribution may also have developmental benefits. Generally, the more optimistic one is about the case that health in childhood affects quality of life in adulthood, the more optimistic one ought to be about both LLIN distribution and deworming. Personally, I would guess that LLIN distribution has stronger developmental benefits than deworming on a “per-person-per-year” basis (for reasons outlined previously) though one could easily argue either side of this question.

  If one assumes that LLIN distribution and deworming have equal benefits on a “per-person-per-year” basis, and that the proportion of people treated who are children is similar for the two, then this implies that when considering developmental benefits alone, deworming accomplishes about 2.5x as much good per dollar as LLIN distribution (based simply on the “cost per person treated per year” from the previous section).

  One may also try to combine the mortality benefits and the developmental benefits of LLIN distribution under these assumptions, as we do in our spreadsheet on “cost per life saved equivalent” comparisons. Below, three of us – myself, Elie, and Alexander – explain our assumptions and give the resulting cost-effectiveness comparison. We also encourage readers to use the spreadsheet to enter their own assumptions. You can do so by going to the “Master” sheet in our spreadsheet on “cost per life saved equivalent” comparisons, manually editing columns M through V for any row, and watching the output in columns X and Y.

  Elie’s assumptions: $0.56 per person dewormed (assumes that our $0.51 estimate is too low by 10%, based on an underestimate we previously made of AMF’s costs before analyzing more deeply); 50% of deworming goes to children; child-years of deworming by SCI are on average 1/3 as effective as child-years in the key study; developmental benefits should be valued 20% as much as lives saved; the external validity of the key deworming study is 30.25% (based on the change in moderate-heavy infections experienced during the course of the study); 40% chance that the study would hold up under replication; standard (Disease Control Priorities Report-based) minor health benefits of deworming. $1,668 per life saved equivalent for deworming (83% from developmental benefits); $1,564 for nets.

  Holden’s assumptions: $0.51 per person dewormed (uses our current estimate); 50% of deworming goes to children; child-years of deworming by SCI are on average (2.41/10) as effective as child-years in the key study (this is equivalent to assuming that SCI is treating all children throughout childhood, and that benefits do not increase beyond the 2.41 years point); the external validity of the key deworming study is 30.25%; 10 people with developmental benefits are morally equivalent to 1 life saved (this figure comes from regressing my own intuitions about what’s valuable – which put very low value on saving lives relative to improving them – somewhat toward “normality,” since I’m not confident that my intuitions are appropriate on this point); 30% chance that the study would hold up under replication; standard (Disease Control Priorities Report) based minor health benefits of deworming. $3,813 per life saved equivalent for deworming (56% from developmental benefits); $2,004 per life saved equivalent for nets.

  Alexander’s assumptions: $0.51 per person dewormed (uses our current estimate); 50% of deworming goes to children; child-years of deworming by SCI are on average exactly as effective as child-years in the key study (this is based on ignorance about where in the distribution of child-years of deworming additional funds might be spent, and takes into account the upside potential of later years conceivably helping to eliminate schistosomiasis from an area); the external validity of the key deworming study is 30.25%; a 2.4% quasi-disability weight for developmental effects (following the Disease Control Priorities Report, not due to confidence that it is the correct estimate but because it roughly maps to the intuition that a permanent 25% increase in income [derived from Baird et al 2012] for ~40 people starting in adulthood would be roughly as valuable as saving a life of a young person); 50% chance that study would hold up under replication; standard (Disease Control Priorities Report 3% discount rate DALY-based) minor health benefits of deworming (with no minor health benefits for bednets). $2,981 per life saved equivalent for deworming (73% from developmental benefits); $2,027 per life saved equivalent for nets. (Alexander comes up with broadly similar figures because he’s more confident in the representativeness and quality of the deworming studies than Elie or I, but places much less weight on developmental effects than either of us.)

  All of these figures can be converted straightforwardly to DALYs if one prefers these units. The DALYs per “life saved equivalent,” for three different versions of DALYs, are given in column AC of our spreadsheet on “cost per life saved equivalent” comparisons.

  Cost per “equivalent life saved” for cash?

  We have not attempted to use the “equivalent life saved” framework to quantify the benefits of cash transfers, since we know so little about how they are spent and what the likely results are. Instead, we compare cash transfers to the other interventions using a different framework, discussed immediately below.

  Financial returnsThe primary benefit we’ve seen evidence for, regarding deworming, is improved earnings for people dewormed in childhood. We can estimate the net present value of these benefits to arrive at a figure along the lines of “For each $1 spent on deworming, recipients receive total benefits equivalent to $X.” We can also do a modified version of this: estimating $X as a percentage of recipients’ annual income (in the case of deworming, we would guess that the percentage increase in income is a more robust figure across different settings than the dollar increase).

  We can estimate a similar figure for cash transfers, using what we know about the longer-term returns to cash transfers, and compare the two. Finally, we can do a similar calculation for LLIN distribution, assuming that LLINs have similar developmental benefits to deworming (on a per-person-per-year basis), though this calculation excludes other benefits of LLINs such as direct mortality benefits.

  Note: we have updated this section (and the linked spreadsheets) since initially posting to incorporate the short-term health benefits of deworming; across all scenarios we assume that each dewormed person experiences current health benefits of value equivalent to a 0.51% increase in income. This value was chosen because it results in 71% of the total benefits of deworming in our preferred scenarios (and 60% in the full set of considered scenarios) arising from developmental effects, the same as in the “lives saved equivalent” framework discussed above (assuming the short term health effects of deworming from the DCP2).

  Our spreadsheet on financial returns performs this comparison. As in the previous section, many assumptions and judgment calls must be made. Regarding deworming, we have the same questions as in the previous section regarding the reliability and external validity of the studies on developmental benefits. Regarding cash transfers, there are additional questions: how much of the transfers are likely to be invested, and what rate of return are they likely to earn? What discount rate should be used to capture the fact that (a) recipients prefer present consumption to future consumption; (b) donors can invest their money and give later rather than giving today and letting recipients invest it?

  Our spreadsheet on financial returns (XLS) addresses these questions.

  Again, because this calculation is extremely sensitive to small changes in inputs, we provide the inputs for myself, Elie and Alexander. We also encourage readers to use the spreadsheet to enter their own assumptions. You can do so by going to the “Overview” sheet in our spreadsheet on financial returns, manually editing columns J through Q for any row, and watching the output in columns S through Z.

  Elie’s assumptions: 50% of deworming goes to children; child-years of deworming by SCI are on average 1/3 as effective as child-years in the key study; 30.25% adjustment for external validity concerns; cash transfers are 75% invested, returning 10% a year with a 5% discount rate; 40% replicability adjustment for deworming and a 95% replicability adjustment for cash. (Elie assigns a much lower likely rate of return than seen in the studies on cash transfers, and as such does not do another large “replicability adjustment” for this.) This results in deworming being 2.3x as cost-effective as cash, with each dollar spent leading to total benefits equal to 1.25% of annual income for deworming, vs. 0.55% for cash. (In a family with the sort of income reported for GiveDirectly’s clients, this would translate to the equivalent of $2.98 in benefits for every dollar donated for deworming; $1.30 for cash.)

  Holden’s assumptions: I did two different estimates, one representing low skepticism about results found in studies and another representing high skepticism. In both, I assume 50% of deworming goes to children; that child-years of deworming by SCI are on average 50% as effective as child-years in the key study; a 30.25% adjustment for external validity concerns (based on before-and-after rates of heavy-to-moderate worm infections in the key study); and a 75% investment rate (i.e., cash transfers are 75% invested vs. 25% consumed) with a 5% discount rate. However, in one case I give both the cash and deworming studies a 50% chance of holding up under replication, and use a rate of return (25%) that is triangulated from studies; in the other case I use a 5% rate of return on investment (which effectively assumes simply keeping up with the discount rate, i.e., not earning any substantive return on investment) while applying a 30% “replicability adjustment” to the deworming study. These assumptions result in deworming being ~2.9-4.2x as cost-effective as cash. In the less skeptical version, each dollar spent on deworming leads to total benefits equal to 1.93% of annual income, as opposed to 0.66% for cash; in the more skeptical version, each dollar spent on deworming leads to total benefits equal to 1.36% of annual income, as opposed to 0.32% for cash. (In a family with the sort of income reported for GiveDirectly’s clients, this would translate to the equivalent in $4.58 (optimistic)/$3.22 (skeptical) in benefits for every dollar donated for deworming; $1.57 (optimistic)/$0.76 (skeptical) for cash.)

  Alexander’s assumptions are similar to those in my “optimistic” scenario except that he (as described above) assumes that 100% of the child-years of deworming are as effective as the child-years from the key study and takes the average of the reported ranges for cash transfer ROI across the two longer-term studies (54%). These go in opposite directions, leading to the conclusion that each dollar spent on deworming translates to total benefits worth 3.36% of annual income ($7.97 in a family with the sort of income reported for GiveDirectly’s clients) while each dollar spent on cash transfers translates to total benefits worth 1.21% of annual income ($2.87), which implies that deworming is ~2.8x as cost-effective as cash.

  Relative to the ranges of assumptions we consider, our individual estimates come out more positive for cash transfers. The mean across the 6,075 scenarios representing each combination of our assumption set is that each dollar spent on deworming increases annual consumption by 3.37% and each dollar spent on cash transfers increases annual consumption by 0.65%, implying that deworming is ~5x as cost-effective as cash transfers.

  ConclusionOur best guess is that bednet distribution is somewhere between one and two times as cost-effective as deworming, while deworming is between 2 and 5 times as cost-effective as cash-transfers. However, we believe that there is ample room for disagreement around these figures, and we plan to write more about the way these figures should be used to guide giving decisions.