GiveWell seeks to constantly revisit and rethink our content, to make sure that it continues to represent a reasonable interpretation of the currently available facts. To this end, we assigned Jonah Sinick, a Research Analyst, to perform a thorough review of the content most relevant to our #1 charity (the Against Malaria Foundation): the AMF review page and the page on distribution of insecticide-treated nets. We asked Jonah to check our footnotes, think critically about our reasoning, and list as many potential problems with our content as he could find; we committed to publishing a list of all such potential problems along with our responses.
The full output of this exercise is here; a summary of the major issues raised and our responses is here. This post focuses on one particular issue Jonah raised – a question about the extent to which small-scale studies of ITN efficacy are representative of real-world conditions (in particular, real-world malaria burdens) – and describes the steps we took to investigate the matter. In brief:
- Jonah found some initial reason to suspect that our “cost per life saved” for ITN distribution was unrealistically low. He noted that based on available country-level estimates for malaria mortality, ITNs could not save lives with the frequency we were claiming, even if they prevented all malaria deaths.
- In response, I did some initial estimates that implied the correct “cost per life saved” figure might be 5-10x higher than our current figure, and accordingly marked the issue as a high priority.
- After further investigation, we concluded that, in fact, no major revision to our “cost per life saved” figure is called for. The deaths averted by ITNs appear to include deaths that are not strictly classified as “malaria deaths,” and available data does not give us reason to believe that the ITN efficacy studies are unrepresentative.
- However, a smaller issue – the decline in all-cause child mortality between the time that ITN efficacy studies took place and today – does potentially call for a relatively small adjustment, and also adds a new source of uncertainty regarding the cost-effectiveness of ITN distributions.
- There are multiple issues that we continue not to account for in our cost-effectiveness estimate, including other positive impacts of nets: prevention of non-lethal malaria cases, reduction of malaria deaths among those over the age of 5, and contribution to the prevention of other diseases transmitted by mosquitoes such as lymphatic filariasis.
Our bottom line after going through this exercise remains that insecticide-treated net distribution remains one of the most cost-effective programs we know of and we continue to rank the Against Malaria Foundation as our #1-ranked charity.
The rest of this post describes our investigation in greater detail than we usually do; since we are publishing the results of the vetting, we thought we’d take this opportunity to give interested parties a more-in-depth-than-usual look at the kind of back-and-forth and analysis we are constantly engaging in to get a better understanding of our top charities.
Overall, we take this episode as another example of how difficult it is to have high confidence in a “cost per life saved” type figure. Our estimate represents the figure that we come to when we use the best evidence we can to estimate each parameter as fairly as we can, but it does not include a Bayesian adjustment to account for the substantial uncertainty, both on account of issues we’ve thought of and issues we haven’t thought of; as such, it should not be taken literally.
That said, we also think the issues discussed in this post illustrate a good reason to do cost-effectiveness estimation: it is a way of disciplining ourselves to make sure we’ve addressed every input and question that matters on the causal chain between interventions (e.g., nets) and morally relevant outcomes (e.g., lives saved). In this case, Jonah’s examination of our cost-effectiveness estimate led to many new questions and observations, and we feel that our overall understanding of the strengths and weaknesses of the case for ITN distribution has improved. This benefit of cost-effectiveness estimation is only realized when estimates are continually revisited and rechecked, which in turn requires that they be fully transparent to those checking them.
Note that a future post will discuss another concern we have investigated this year: the question of whether mosquitoes develop resistance and/or adapt their behavior to ITNs over time.