This post is more than 12 years old

GiveWell seeks to constantly revisit and rethink our content, to make sure that it continues to represent a reasonable interpretation of the currently available facts. To this end, we assigned Jonah Sinick, a Research Analyst, to perform a thorough review of the content most relevant to our #1 charity (the Against Malaria Foundation): the AMF review page and the page on distribution of insecticide-treated nets. We asked Jonah to check our footnotes, think critically about our reasoning, and list as many potential problems with our content as he could find; we committed to publishing a list of all such potential problems along with our responses.

The full output of this exercise is here; a summary of the major issues raised and our responses is here. This post focuses on one particular issue Jonah raised – a question about the extent to which small-scale studies of ITN efficacy are representative of real-world conditions (in particular, real-world malaria burdens) – and describes the steps we took to investigate the matter. In brief:

• Jonah found some initial reason to suspect that our “cost per life saved” for ITN distribution was unrealistically low. He noted that based on available country-level estimates for malaria mortality, ITNs could not save lives with the frequency we were claiming, even if they prevented all malaria deaths.
• In response, I did some initial estimates that implied the correct “cost per life saved” figure might be 5-10x higher than our current figure, and accordingly marked the issue as a high priority.
• After further investigation, we concluded that, in fact, no major revision to our “cost per life saved” figure is called for. The deaths averted by ITNs appear to include deaths that are not strictly classified as “malaria deaths,” and available data does not give us reason to believe that the ITN efficacy studies are unrepresentative.
• However, a smaller issue – the decline in all-cause child mortality between the time that ITN efficacy studies took place and today – does potentially call for a relatively small adjustment, and also adds a new source of uncertainty regarding the cost-effectiveness of ITN distributions.
• There are multiple issues that we continue not to account for in our cost-effectiveness estimate, including other positive impacts of nets: prevention of non-lethal malaria cases, reduction of malaria deaths among those over the age of 5, and contribution to the prevention of other diseases transmitted by mosquitoes such as lymphatic filariasis.

Our bottom line after going through this exercise remains that insecticide-treated net distribution remains one of the most cost-effective programs we know of and we continue to rank the Against Malaria Foundation as our #1-ranked charity.

The rest of this post describes our investigation in greater detail than we usually do; since we are publishing the results of the vetting, we thought we’d take this opportunity to give interested parties a more-in-depth-than-usual look at the kind of back-and-forth and analysis we are constantly engaging in to get a better understanding of our top charities.

Overall, we take this episode as another example of how difficult it is to have high confidence in a “cost per life saved” type figure. Our estimate represents the figure that we come to when we use the best evidence we can to estimate each parameter as fairly as we can, but it does not include a Bayesian adjustment to account for the substantial uncertainty, both on account of issues we’ve thought of and issues we haven’t thought of; as such, it should not be taken literally.

That said, we also think the issues discussed in this post illustrate a good reason to do cost-effectiveness estimation: it is a way of disciplining ourselves to make sure we’ve addressed every input and question that matters on the causal chain between interventions (e.g., nets) and morally relevant outcomes (e.g., lives saved). In this case, Jonah’s examination of our cost-effectiveness estimate led to many new questions and observations, and we feel that our overall understanding of the strengths and weaknesses of the case for ITN distribution has improved. This benefit of cost-effectiveness estimation is only realized when estimates are continually revisited and rechecked, which in turn requires that they be fully transparent to those checking them.

Note that a future post will discuss another concern we have investigated this year: the question of whether mosquitoes develop resistance and/or adapt their behavior to ITNs over time.

Initial cause for concern: comparing “deaths averted” rates to “total malaria deaths” estimates

Jonah noticed that “GiveWell assumes that ‘each effective year of protection for children under five results in 0.00553 [lives saved].’ In order for this to be the case, the death rate due to malaria among under-5’s must be at least 0.005.” He then collected a series of malaria mortality estimates by country and found that the general picture was that very few countries (among those that AMF has told us it is considering working in) have under-5 malaria mortality rates of .005 or higher.

Our spreadsheet on malaria and all-cause mortality has this data, though it is not well-organized (due to being compiled for an internal vetting assignment); the easiest column to examine is AO, which show estimates from the source that appeared to have the highest malaria mortality estimates. For 2010, only one country on AMF’s list (Mali) had a malaria mortality rate greater than .00553, and most of them were in the range of .002-.003.

We briefly explored a few possible interpretations in the emails that followed. I pointed out that current malaria mortality rates may be lower than they otherwise would be because of the presence of ITNs; after all, if (hypothetically) ITNs reduced malaria mortality from a rate of .007 to a rate of .002, then it could simultaneously be the case that (a) ITNs reduce malaria mortality by .005 (b) the current malaria mortality rate is only .002. But Jonah responded by referencing the trend in malaria mortality rates since 1990. In 1990, before any of the studies on ITNs were carried out (and, we believe though we haven’t directly verified, before there were significant ITN distribution campaigns), the under-5 malaria mortality rate was still under .00553 in about half of AMF’s target countries.

At this point my suspicion was that the ITN efficacy studies had taken place in unrepresentative regions: places with higher malaria mortality rates than the average for their countries.

Potential implications for cost-effectiveness

The review of ITN efficacy studies we have been using (the same one that gives the .00553 figure mentioned above) states that, regarding under-5 mortality, “ITNs provided 17% protective efficacy (PE) compared to no nets, and 23% PE compared to untreated nets.” To take a rough look at the possible implications of Jonah’s observation for our cost-effectiveness estimate, I

• Started with the most generous estimate of under-5 malaria mortality I could find (the rates from 2000, the year with the highest under-5 malaria mortality, according to the Lancet paper which appeared from Jonah’s analysis to have the highest overall estimates of malaria mortality).
• Multiplied the rates by 20% (a rough estimate of the proportion of malaria deaths averted by ITNs; 20% was based on a statement in the review that we used as our main source on the efficacy of ITNs (PDF), “ITNs provided 17% protective efficacy (PE) compared to no nets, and 23% PE compared to untreated nets”) to estimate how many deaths would be averted, on average, for each country, per person-year of protection with ITNs.
• Compared this “deaths averted per person-year of protection” figure with .00553, which was the figure we had been using.

The implication is in column of AT of our spreadsheet on malaria and all-cause mortality: cost-effectiveness would drop by 50-99% depending on the country. In most cases it would drop by 70-80%, implying that our cost-effectiveness estimate for ITNs would be off (too optimistic in terms of impact) by a factor of 3-5 even in this optimistic scenario.

Resolving the issue and assessing the representativeness of efficacy studies

I returned to the original ITN efficacy studies, seeking to get more concrete about just how unrepresentative they were, making notes in a modified version of our summary of ITN efficacy studies. I sought to compare the malaria mortality rates observed in the control groups of these studies to the malaria mortality rates estimated, at around the same time, for the countries in which the studies took place. I wanted to know whether the study areas were unrepresentative of their countries, whether the countries the studies took place in were unrepresentative of the sorts of countries AMF seeks to work in, or whether there was something else I had missed.

As it turned out, out of five studies that had addressed mortality, only two reported figures for malaria mortality specifically – and their figures were not out of line with the national averages at the time. (See columns Z, AG and AH of the modified summary of ITN efficacy studies.) Furthermore, the national figures for these countries didn’t look very different from the national figures for AMF’s countries (see columns AM-AO of our spreadsheet on malaria and all-cause mortality). Contrary to my expectation, there was no evidence that the ITN efficacy studies were highly unrepresentative in terms of malaria mortality. One study had a malaria mortality rate that was about 2x the country average (still a smaller discrepancy than what I had expected based on the above reasoning); the other had a malaria mortality rate that was lower than the country average.

So how did the studies find such big impacts on childhood mortality? The headline impacts for all five studies were for all-cause mortality, not malaria mortality – a point noted in the Cochrane review:

The impact of ITNs on malaria-specific death rates was looked at only briefly because of the problems using verbal autopsies in determining malaria deaths. In the two trials for which the data were available, the percentage reduction in malaria-specific mortality was similar or smaller than the percentage reduction in all-cause mortality: 14% (versus 23%) for Gambia (D’Alessandro), and 22% (versus 18%) for Ghana (Binka). One interpretation is that malaria-specific death rates were not reflecting the true impact of ITNs on mortality (since a much higher specific impact would have been expected).

This raises a couple of questions:

• Is it plausible that ITNs avert deaths that would be classified as non-malaria deaths? I believe so. In addition to mortality reduction, the efficacy studies show other “general health” effects such as reduced anemia, reduced splenomegaly, and more. And we believe that improved general health in children under 5 could prevent deaths from multiple causes (some support for this idea comes from the fact that vitamin A supplementation has been associated with substantial drops in all-cause mortality in children under 5).

  I checked the Lancet paper that estimates total deaths from malaria and verified that it does not seek to estimate deaths that malaria may have contributed to; it simply attempts to estimate deaths that were or should have been classified as “deaths from malaria.”

• Were the ITN efficacy studies unrepresentative in terms of having unusually high all-cause under-5 mortality relative to country averages? I examined World Bank data on all-cause mortality in order to investigate this question. As columns W and AN of the modified summary of ITN efficacy studies show, there was again no reason to believe that ITN studies were unrepresentative: in some cases the mortality rate seen in the study resembled that seen in the country as a whole, and in some cases it diverged in one direction or the other, but never by more than a factor of ~2.
• What about comparing the countries of the ITN efficacy studies to the countries AMF plans to work in? I did this in columns AL and AP of the spreadsheet on malaria and all-cause mortality, and again found that (speaking very broadly) the countries the studies took place in didn’t look very different from the countries AMF is considering working in, in terms of all-cause mortality or in terms of the share of mortality accounted for by malaria.

Bottom line – I didn’t see reason to believe that the studies were unrepresentative of their countries, or that the countries the studies were done in were importantly different form the countries AMF now targets. As column AR shows, assuming a 20% reduction in all-cause mortality and using the country averages from 1995 (I picked this year because it’s fairly representative of when the studies were done) could leave us with a very similar “lives saved per person-year-of-protection” estimate to the one we’re already using. (A simple average of this rates for the countries featured in the studies is .00567, very close to the figure we’ve been using of .00553; the AMF target countries have a slightly higher average rate, .006.)

All-cause childhood mortality has declined; what does this mean for ITN distribution?

However, there’s one more issue that I noticed in doing this calculation: all-cause mortality has fallen significantly since the time the studies were done. Using 2011 levels of mortality, rather than 1995 levels, would lead to a lower “lives saved per person-year-of-protection” estimate, more in the range of .004 as opposed to .00553. In other words, while the studies may not have been geographically unrepresentative, the time they were carried out looks different from today.

There are a few ways to interpret this finding:

1. One could assume that ITNs cause a relatively stable (percentage) drop in all-cause mortality. Thus, since baseline all-cause under-5 mortality is (very roughly speaking) about 30% lower in the relative countries than at the time the studies were done, one could believe that ITNs save 30% fewer lives per dollar than they did at the time the studies were done. This assumption would change our “cost per life saved” estimate from ~$1600 to ~$2300.
2. However, some of the observed drop in all-cause mortality may be because of ITN distribution, and gains may be reversed if ITN distribution were to stop. This argues for a smaller adjustment than the one mentioned in the previous point.

   On the “Trend since 1995” sheet of our spreadsheet on malaria and all-cause mortality, I charted the decline in all-cause mortality since 1995 against the rise in malaria funding; based on this, it seems possible that malaria control has not been a substantial part of the decline in all-cause mortality, and if it has been it this phenomenon probably started in 2004, meaning that the ~20% drop between 1995-2004 can be attributed to other causes. So even with this point in mind, there should be at least a ~20% reduction in “lives saved per dollar” (taking our estimate from ~$1600 to ~$2000).

3. It’s highly possible that the reduction in mortality due to ITNs is not a simple function of baseline mortality. Perhaps the deaths averted by ITNs are the same deaths that could be averted by, for example, vitamin A supplementation (recall that many of the deaths averted by ITNs are not specifically attributable to malaria). So perhaps other improvements in general health are independently averting all the deaths that ITNs could avert in their absence; under this model it’s possible that ITNs don’t avert any deaths at all. On the flipside, there may be increasing returns to improved general health: perhaps there are children who previously would have been in such poor health that they would have died even with ITNs, but now can have their deaths averted by ITNs.

We are planning to add an adjustment into our cost-effectiveness calculation, lowering the cost-effectiveness by 20-30%.

Conclusions

We haven’t reached a final conclusion on how to adjust our malaria mortality estimate. We have determined that any adjustment won’t be as large as initially anticipated. That said, I believe the issues discussed in this post are a good illustration of a point that we made in a previous post:

Our own attempts to do cost-effectiveness analysis have turned out to be very sensitive to small variations in basic assumptions. Such sensitivity is directly relevant to how much weight we should put on such estimates in decision-making.

We have uncovered substantial sources of uncertainty in a figure (the cost per life saved for ITN distribution) that was already a very rough approximation. The number relies heavily on 5 efficacy studies carried out in the 1990s; in addition to questions about whether the studies were carried out in representative environments, there are also major potential questions about how the developing world of today compares to the developing world of the 1990s, and what that means for the impact of ITNs.

Our estimate represents the figure that we come to when we use the best evidence we can to estimate each parameter as fairly as we can, but it should not be taken literally. That said, we are glad to have done the analysis necessary to arrive at this estimate, to have published the full details of our analysis, and to have had these details vetted, as they have raised important new questions and observations about our current top-rated charity.