If we were to look for confirming evidence, then perhaps we’d find that severe errors (currently undiscovered) in the DCP2 calculation means that, by their own lights, they should have given a much more favourable estimate of STH’s cost-effectiveness than GiveWell’s corrected estimate … it’s worth bearing in mind that we should also be looking for evidence in favour of higher cost-effectiveness of deworming as well.

are you saying that GiveWell went into this investigation intending to look for errors that overstated the cost-effectiveness of deworming, but not for errors that understated the cost-effectiveness of deworming? If so, what leads you to believe that?

As Alexander says, I think the errors are very worrying - they seem to reflect a process that has no systematic double-checks or reality-checks. However, the errors aren’t the only thing about the estimate that I find worth noting. I think it’s also worth noting the simplicity of the calculation.

Among other things, the calculation uses broad regional prevalence figures (and the final result is implicitly based on an equal-weighted average of regional prevalence, an incorrect way to take the average; Alexander didn’t mention this as an error), ignores externalities, ignores long-term considerations both positive and negative, ignores drug side-effects, and makes no apparent attempt to address the fact that its key figures (regarding delivery costs and effectiveness) are based on a small number of studies (i.e., there is no apparent attempt to address the question of how wastage and other execution issues would likely differ in larger-scale campaigns). There are no factors in the calculation that would make the estimate rise as evidence quality rises or fall as it falls. No sensitivity analysis is present.

We have conjectured that the estimates are this simple before, but now we know (for at least one estimate), and I don’t want to lose sight of this in the discussion about errors.

My impression is that the DCP2 estimate - far from trying to compensate for the complexity of deworming with sophistication of analysis - is basically a back-of-the-envelope figure that incorporates only a fraction of the information that we have at our disposal (if not in a way that can be effectively formalized) when investigating a specific charity. My impression is that other DCP2 estimates follow the same basic approach. Based on this alone, I would see little argument for giving them a substantial role in giving decisions. Adding in the high sensitivity to error that we’ve observed, and the apparent lack of a process for double-checks/reality-checks, makes me lean against giving them any role.

Given the amount of work it would take to further investigate this impression of the estimates, I’d rather focus our efforts on other work. Of course, if someone else came to us with evidence that other DCP2 estimates are more sophisticated, we’d be interested.

Thanks for the comments. A few responses:

The main thing I’m worried about is inferring, from the fact that we should decrease reliance on DCP2, that we should decrease reliance on cost-effectiveness estimates in general. What we ultimately care about is how much benefit we can give people per $ invested in a charity. We shouldn’t lose sight of that. Your research also shows, in terms of what charity evaluators should be doing, that we need to spend *more* time looking into the literature of economic estimates of cost-effectiveness, because we don’t have one authoritative source.

As Holden and I have been saying, we agree with the goal of maximizing expected value. It is becoming less clear whether explicit cost-effectiveness estimates, even “good” ones, are a good way to maximize expected value. I don’t think anyone is losing sight of the goal of doing as much good as possible with money. But I also don’t think that, say, using the Copenhagen Consensus or WHO-Choice is the best strategy for getting better estimates. We’ll be discussing this more in future posts.

GiveWell’s research shows that the DCP2 estimates are extremely noisy. If we were to look for confirming evidence, then perhaps we’d find that severe errors (currently undiscovered) in the DCP2 calculation means that, by their own lights, they should have given a much more favourable estimate of STH’s cost-effectiveness than GiveWell’s corrected estimate. Having said that, I think that the errors discovered by GiveWell do provide a greater amount of disconfirming evidence than I would have expected prior to investigation; and that this isn’t just attributable to noise. But it’s worth bearing in mind that we should also be looking for evidence in favour of higher cost-effectiveness of deworming as well.

As I mentioned above, we had a number of reasons for looking into the DCP2 estimate; trying to debunk the DCP2 (or deworming) wasn’t one of them. Like you, we started off thinking that the DCP2 was a relatively credible source of cost-effectiveness estimates. In a 2008 blog post, for instance, we said that we were trying to get in touch with the DCP2 authors and lauded them for their thoroughness, while expressing frustration that the inputs to the calculation weren’t public. We’re also seriously considering recommending SCI, so understanding deworming is very important to us. We were very surprised that there were these mistakes in the DCP2 cost-effectiveness estimate.

I want to note that the mistakes we found point both directions. If you look at our corrected spreadsheet, you’ll see that the estimated cost-effectiveness when just errors 1-3 are fixed is $515/DALY. Once the two trichuriasis errors are fixed, it drops to $326.43, and once long term effects are taken into account, it goes to $138. There were numerous errors pointing in both directions, but the errors that overestimated the cost-effectiveness were larger.

In addition, we think Jonah’s calculation, which is much more optimistic about deworming, seems superior to the DCP2’s.

As to the points you raise in favor of deworming, we have written about many of these in the past (example). Many apply to the landscape of cost-effectiveness estimates in general. Our blog post is not intended as a summary of all possible issues with the DCP2 calc, but as an explanation of a few previously unknown issues.

When you say that “The above research should, I think, make one update in favour of ‘no’ to [the question, “should we think that deworming charities are the expectedly best charities to fund?”],” are you saying that you now think the answer is no, or just that there is less evidence for a yes answer?

We agree that the (cost-)effectiveness of particular deworming charities is up in the air right now. We’re continuing our research in this area.

Nick:

A couple more points:

When I said it was one estimate, I suppose I should have said that it was one DCP2 chapter, all done by the same group. Different groups may have been run significantly differently, so it is not transparent how much to downgrade other estimates by the DCP2.

We don’t know how the other cost-effectiveness estimates were made; they may have been better. As far as we can tell, the people who did the deworming cost-effectiveness estimate were also the ones who wrote the cost-effectiveness chapter in the DCP2 and were involved in many of the other chapters.

I currently have a 90% subjective probability that if someone looked at another DCP2 estimate for a highly-rated intervention, we would not find that the estimate was off by a factor of 100 due to dimple computational errors, and a 70% subjective probability that we would not find it to be off by a factor of 10 due to simple computational errors. (Previously, I might have had much higher probabilities in these claims, like 97% and 90%.) For me, this is enough to make the use of such estimates an important tool for analysis. I would be curious what your subjective probabilities are for these claims, since I might update toward your position.

As we’ve discussed with GWWC elsewhere, the DCP2 estimates appear to be approximately log-normally distributed. More than 97% of the DCP2 estimates fall within two orders of magnitude of the mean, and 69% fall within one order of magnitude (a factor of 10) of the mean. I’ve attached the spreadsheet we used for this calculation here if you’re curious. I think we’ve sent a variant of it to you before.

Thus, according to your subjective probabilities, one can chalk the entire variance in DCP2 cost-effectiveness estimates up to “simple computational errors.” If it turned out that the error-free version of each DCP2 calculation came out to exactly the same value (the median cost-effectiveness figure for the distribution as a whole), that would be consistent with your subjective probabilities. Saying that the DCP2 is about as reliable as a repeated calculation with variation coming entirely from mistakes seems to me like a vote of extremely low confidence.

I find your comment above abstract to the point that I don’t know what you have in mind; are there particular interventions that you think that people interested in efficient philanthropy should focus more on or less on based on the DCP2 cost-effectiveness estimates?

As long as we agree that it would be irrational to give no weight to DCP2 estimates, that’s all I wanted to establish with the hypothetical example.

When I said it was one estimate, I suppose I should have said that it was one DCP2 chapter, all done by the same group. Different groups may have been run significantly differently, so it is not transparent how much to downgrade other estimates by the DCP2.

I’m not sure what to make of your point that the chapter errors are not indicative of “a process that ensures correct results” or the claim about who has the “burden of proof.” I don’t think you’d have found me, or anyone else, really, ready to sign up for the view that the DCP2 process “ensures correct results”, though your research has significantly decreased my confidence in this process. However, I don’t think that treating some estimate as decision-relevant requires thinking that the estimate was produced by a perfectly or extremely reliable process. I just do my best to assign a subjective probability to these kinds of things and act on the basis of that assignment. I currently have a 90% subjective probability that if someone looked at another DCP2 estimate for a highly-rated intervention, we would not find that the estimate was off by a factor of 100 due to dimple computational errors, and a 70% subjective probability that we would not find it to be off by a factor of 10 due to simple computational errors. (Previously, I might have had much higher probabilities in these claims, like 97% and 90%.) For me, this is enough to make the use of such estimates an important tool for analysis. I would be curious what your subjective probabilities are for these claims, since I might update toward your position.

One important decision-making upshot of this research for me is that before I make recommendations on the basis of cost-effectiveness research like this, I will make every reasonable attempt to obtain the original calculations or reproduce them myself. However, if I cannot do this, I will not act as if the estimate had negligible weight.

Having said that, I just want to make some clarifications, so that we draw the right conclusions from the above.

In particular, we should distinguish three different questions:

 1.    How heavily should we rely on the stated cost effectiveness from the DCP2, as listed in the table of interventions?

As you’ve shown, I think we should be wary of relying solely on them. DCP2, of course, still provides evidence, but the value of checking that evidence against other estimates is very high indeed.

2. How heavily should we rely on cost-effectiveness estimates in general?

The main thing I’m worried about is inferring, from the fact that we should decrease reliance on DCP2, that we should decrease reliance on cost-effectiveness estimates in general. What we ultimately care about is how much benefit we can give people per $ invested in a charity. We shouldn’t lose sight of that.

In particular, your research shows, in terms of the global allocation of funds, that there should be *more* funding of cost-effectiveness research; because there are such great and easy gains to be had by simple improvements of the estimates. Your research also shows, in terms of what charity evaluators should be doing, that we need to spend *more* time looking into the literature of economic estimates of cost-effectiveness, because we don’t have one authoritative source.

In general, we should be wary of inferring from the fact the one source of X is less trustworthy than we thought to the conclusion that X, in general, isn’t to be trusted.

 
3.      Should we think that deworming charities are the expectedly best charities to fund?

The above research should, I think, make one update in favour of ‘no’ to this question.

But it should be borne in mind that, somewhat counterintuitively, it’s not always true that gaining new evidence that disconfirms a proposition should lower one’s credence in that claim. If one is setting out to look for disconfirming evidence for a proposition, and one is not also looking for confirming evidence, and one then finds less disconfirming evidence than one would expect, then one should update in favour of that proposition.

This is relevant because I get the sense that GiveWell takes a deliberately critical stance when evaluating any intervention type or charity.  So, at the end of GiveWell’s investigations, we need to ask not “How much disconfirming evidence is there?” but rather “How much disconfirming evidence is there, relative to what I would have expected prior to investigation?”

Even this question doesn’t give us quite what we want. Because it could just be that the noise of the data is greater than we would have expected (i.e. charity evaluation is harder than we thought!). This would mean that one would find a greater amount of disconfirming evidence, relative to what one would have expected prior to investigation, even when the expected cost-effectiveness is just the same.

I think that this could explain why, after GiveWell’s corrections, the DCP2 estimates for STHs and Schisto are so low. GiveWell’s research shows that the DCP2 estimates are extremely noisy. If we were to look for confirming evidence, then perhaps we’d find that severe errors (currently undiscovered) in the DCP2 calculation means that, by their own lights, they should have given a much more favourable estimate of STH’s cost-effectiveness than GiveWell’s corrected estimate.

Having said that, I think that the errors discovered by GiveWell do provide a greater amount of disconfirming evidence than I would have expected prior to investigation; and that this isn’t just attributable to noise. But it’s worth bearing in mind that we should also be looking for evidence in favour of higher cost-effectiveness of deworming as well. This could significantly increase our credence in deworming’s high cost-effectiveness.

To illustrate:

Reasons why deworming might be even better than the DCP2 estimates make out (this is without deliberately looking for good things; and there might be more I’ve currently forgotten about, as these are off the top of my head):

Fewer treatments per child (i.e. 2-3) in long run
Economic and educational benefits of deworming.
Effect on other illnesses.  E.g. HIV/AIDS.  Not accounted for in contemporary cost-effectiveness studies.
Extra economic benefits of improving quality of life rather than extending lives
Because of the way that DALYs are calculated, Years of Life Lost are are inflated in importance relative to Years Lived with Disability.

Reasons why specific deworming charities like SCI might be even more cost-effective than cost-effectiveness of STH treatment in general (again off the top of my head):

SCI administers a combination package of drugs, to treat the 7 most prevalent NTDs. The drug costs are very low or nonexistent, and so I’d expect the combined cost-effectiveness to be much higher than the cost-effectiveness of one intervention on its own.
There are some elimination programs (e.g. Zanzibar)
Benefits from government handover
Leverage of governmental resources (which would have been spent on something less cost-effective).

This isn’t the place to get into the nitty gritty of deworming charities; but the important lesson is that we should be impartial in our search for confirming and disconfirming evidence. If we don’t do this, then we shouldn’t automatically decrease our credence in a proposition given that new disconfirming evidence comes to light.

Like I say, this is extremely important research that you’ve done, and it should make us more wary of relying on the DCP2.

Thanks for the comments.

A few points in reply:

• Your thought experiment of “funding a charity with an intervention sampled from the bottom of the DCP2 rankings and funding a charity with an intervention sampled from the middle of the DCP2 rankings, given no other information” doesn’t strike me as realistic, because we have many other sources of information. Of course, I don’t think it contains no information, so I agree that, all else equal, an intervention rated more cost-effective is likely to be so. But if the only information I had to recommend an intervention was the DCP2 rankings, my top priority would be getting more and better information.
• I don’t agree that this is “only one DCP2 estimate”–the typo in the schistosomiasis cost-effectiveness estimate also puts it off by two orders of magnitude. And while I agree that typos happen, typos plus five other separate errors do not seem to reflect a process that ensures correct results. We’d be happy if others want to do a full scale investigation of a large sample of the DCP2 calculations, but such an investigation wouldn’t be practical for us. We now think the burden of proof is on anyone who wants to take DCP2 estimates on trust; for our part, we do not. If we can independently verify a DCP2 calculation, then we’ll be making recommendations based on our independent verification, not the original estimate; if we can’t verify it, we won’t trust it.

  As I point out above, these errors would almost certainly have been caught by a helminth expert double checking the calculation. The DCP2’s failure to ensure that happened seriously hampers my assessment of its credibility.

• I think you’re right that the deworming estimate was unusually likely to be wrong, but my reasons differ from yours. My main reason is that it was something of an outlier: the $3.41/DALY figure made STH deworming the second most cost-effective intervention in the DCP2. All else equal, we’d expect more extreme estimates to suffer from more extreme measurement error. This point goes back to Holden’s previous observations about applying Bayesianism to charity evaluation.
• I also think you’re right to conclude that “the fact that deworming was more poorly understood (if true) would not have made this kind of error much more likely in the case of deworming than any other intervention.” We agree that deworming is rather poorly understood, but if you look at the errors made in the calculations, that wasn’t the problem. These mistakes had relatively little to do with the particularities of deworming, and much to do with the (apparent) carelessness of the cost-effectiveness estimate. That’s part of the reason we’re so worried about the other DCP2 estimates.
• The case in which “the cost-effectiveness is antecedently well-understood” is the case where the DCP2 is doing the least epistemic work. If there are other cost-effectiveness estimates in credible sources to double check the DCP2 against, I agree that errors are less likely. I also think that those are cases where we are less called upon to trust the DCP2, and more likely to be able to gather useful information from other sources. Basically, I think we agree about this: the DCP2 is most trustworthy when we need to trust it the least.
• Jonah’s estimate did take into account long-term effects for STH deworming. It appears not to have for schistosomiasis; it remains unclear whether it should have. Basically, the official GBD estimates for the DALY burden (for STH, at least) include the future effects of present illness. By dividing the DALY burden by population infected to get the disability weight, he created a disability weight that incorporates future effects.

Bottom line: we have many sources of information about charities. Given the proven possibility for large errors in opaque estimates like the DCP2’s, we don’t see much reason to rely on them in the future.

However, such an error could still be somewhat more likely. After all, if the cost-effectiveness of some intervention were antecedently well-understood, one would have a better idea of what to expect regarding the intervention’s cost-effectiveness, and be more likely to detect an error that was off by a factor of 100x. In that case, the huge difference between one’s estimate and previous estimates would be grounds for checking the estimate again.

This was a significant update for me toward taking the DCP2 calculations less seriously. At the same time, I fear the conclusion in this post goes too far. I definitely would not be “hesitant to place any weight on DCP2 cost-effectiveness figures except where we can fully understand and check the calculations.” I imagine you don’t mean this literally. (You would not, I take it, hesitate between funding a charity with an intervention sampled from the bottom of the DCP2 rankings and funding a charity with an intervention sampled from the middle of the DCP2 rankings, given no other information.)

Some reasons not to go so far:
(i) This is only one DCP2 estimate.
(ii) It is an estimate of an intervention whose cost-effectiveness was more poorly understood than the cost-effectiveness of some of the other highly-rated health interventions. (I have limited information about claim (ii), but Holden expressed it to me in conversation recently. I also heard Chris Murray express some skepticism about the deworming estimates.)

In addition, this report mentions that Jonah’s estimate implicitly accounted for the long-term effects of deworming. But I see no term for this in Jonah’s equations. (Moreover, in a previous conversation with Jonah, I mentioned to him that his model did not take account of these effects, and he agreed that it did not.) Are the long-term effects implicit in Jonah’s calculation in some non-obvious way, was that claim a mistake, or am I somehow confused about this?