• I agree with #1 and #2.

  Note that we are currently putting a good deal of effort into cost-effectiveness estimation as an important input into our upcoming refreshed charity rankings. Because we’re evaluating specific charities, we’re able to make the estimates a bit more context-specific than what the DCP2 provides (though the estimates are still extremely rough).

  I wasn’t arguing that we should stop doing any cost-effectiveness analysis, but rather that it may be more promising to continue with back-of-the-envelope estimates - and give them a limited role commensurate with their status as back-of-the-envelope estimates - than to try to get cost-effectiveness analysis “right” and use estimates as the primary factor in decision making.

• While the cost-effectiveness estimates aren’t the only content of the DCP2, they are central enough to its overall purpose (see the Foreword for example) that I think it’s appropriate to point to how well-resourced this project was as evidence regarding the feasibility of a radical improvement on its methodology.
• I do think cost-effectiveness estimation could be improved with more resources, particularly through more intensive data collection, but I think that to get it to the point where estimates could play a substantially larger role in decisionmaking (as opposed to the role we currently advocate) may be impossible.

Carl, I think the measures you propose would largely eliminate the sorts of errors we’ve written about recently, but I don’t see how they would address the dilemma discussed in this post. There would still be the problem of making estimates very simple and sensitive or of making them opaque to subject-matter experts; either way would create substantial uncertainty over the applicability of a generalized cost-effectiveness estimate to a specific context.

The resources that have already been invested in these cost-effectiveness estimates are significant.

$3.5 MM seems like a tiny, rather than an excessive, expenditure on cost-effectiveness research in the context of many billions in aid spending. Applying a similar sum to back institutionally independent “red teams” in critiquing existing estimates would probably greatly improve data quality, based on past successes. That money could go further by first critiquing currently top-rated interventions and working its way down.

Likewise, Cochrane Reviews-style assessments of data quality could be funded, and data-quality scores combined with cost-effectiveness point estimates or confidence intervals.

I do think the sort of thing you’re describing would be helpful. Whether it would lead to enough reality-checks is an open question.

I recently ran across what strikes me as an example of putting an admirable amount of care into making one’s quantitative work susceptible to reality-checks. The example is a paper on projecting the impact of insecticide-treated nets based on mathematical modeling of malaria transmission. Key quote:

While the analysis outlined here could be implemented with either of the recently developed (and perhaps more elegant) alternative models [6,12], this particular form captures all of the same processes without necessitating the mathematical subtleties of integration, differentiation, equilibrium analysis, or limits. While these are inherently valuable tools for mathematical modelling, they often constitute “black boxes” to nonmathematicians, including several authors of this article. We therefore chose a model that does not require mathematical complexities that might limit accessibility to some of the field biologists and epidemiologists for whom this analysis is most relevant. The model is presented as a downloadable spreadsheet (see Protocol S1) …

I found the linked spreadsheet to be well-labeled, readable, and usable, even though the model itself is relatively complex (much more complex than the cost-per-DALY estimate discussed in our blog post). I’m also glad to see a willingness to make some sacrifices in technical precision in order to improve accessibility and thus susceptibility to reality-checks.

One thought is that they could make a web page which sketches out how the did the calculation. The main page would go through the calculation step-by-step, stating all of their assumptions (both the structural assumptions about how to do the calculation and the quantitative assumptions of all the numbers involved) as concisely and clearly as possible. It would give intermediate calculations/conclusions along the way. Each assumption would be linked to a discussion page where the researchers could say more about their assumption (citing sources and so on), and anyone could comment on the discussion page with questions, criticisms of the assumption, comments about its uncertainty and its impact on the calculation, suggested alternatives, and so on. Each of the quantitative assumptions would be modifiable by the person browsing the web page, so that they could put in their own numbers and the page would automatically recalculate to get their own estimate.

Would something like this be feasible for researchers to make? Would it allow subject matter experts (who lack experience with DALY calculations) to provide reality checks?