[Added August 27, 2014: GiveWell Labs is now known as the Open Philanthropy Project.]
As noted previously, I’ve been investigating the question: How does the “good accomplished per dollar” of life sciences funding compare to that of other giving options (such as LLIN distribution)? At this point, I believe that there is very little in the way of academic literature that can shed light on this question. I’ve put together some very limited and preliminary analysis; to produce something better, I’d want to work with both generalist scientific advisors and at least one economist.
- Examined a review of economic literature on measuring the returns to R&D in general (Hall, Mairesse and Mohnon 2009) and some of the papers it references. The literature discussed in this review tends to focus on private industry R&D, and does not appear highly relevant to the question we’re focused on.
- Examined a review of literature on “the impact of publicly funded and biomedical research” (Sampat 2011) and some of the papers it references, and spoken to the author.
- Read Measuring the Gains from Medical Research: An Economic Approach, spoken briefly with one of the authors, and spoken with another economist he referred us to.
Our takeaways on the relevance of this literature for our purposes:
- Some research focuses on the role that general scientific research and technology innovation has played in reductions in the burden of disease, and generally concludes that it has played a major role (see, in particular, Chapter 4 of Measuring the Gains from Medical Research: An Economic Approach, which discusses heart disease, and The Contributions of Improved Therapy and Earlier Detection to Cancer Survival Gains, 1988-2000 which discusses cancer).
- Some research attempts to disentangle the roles of publicly and privately funded research, and otherwise isolate which kind of research has been responsible for humanitarian gains (and on what timescales); we haven’t found this work compelling.
- Some research attempts to quantify the economic value of past reductions in disease burden, and juxtapose it with the amount the U.S. spends on public research (without directly estimating how much of the reduction is due to public research). This research generally concludes that benefits are significant and far outweigh the costs. For example, chapter 1 of Measuring the Gains from Medical Research: An Economic Approach argues that the value of improved life expectancy over the last ~100 years (monetized using the value of a statistical life framework) is comparable to total measured GDP growth over that time; chapter 2 estimates that gains from 1970-2000 were worth about $46 trillion total, net of increased spending on health care, as against an NIH budget of ~$35 billion per year as of 1995.
- Overall, relatively little has been done in this area, and we haven’t found any research that attempts to (a) make forward-looking estimates of the likely future returns to investments in life sciences; (b) estimate the value of the marginal (as opposed to average) dollar spent on research.
- I chose to focus on cancer because (a) most of the burden is via mortality rather than morbidity, and I believe mortality statistics to be generally easier to access and verify than morbidity statistics; (b) because cancer is a politically popular disease for research funding, it’s likely that the “official” cancer budget of the NIH is a reasonable proxy for – or perhaps overestimate of – how much is being spent on cancer research. (By contrast, if I focused on a less politically salient disease, I’d worry that I was substantially undercounting the funding directed toward it, as funding that was officially earmarked for more popular diseases could effectively be financing highly relevant research. In the case of cancer, while I expect some research outside the National Cancer Institute to be relevant to cancer, I also expect some National Cancer Institute research to be highly relevant to other diseases, and I think it is not unreasonable to think of the National Cancer Institute budget as approximating the “costs of scientific progress on cancer.”)
- I would guess that cancer research produces less benefit per dollar spent than research in many other categories, partly because I perceive cancer as having a relatively large amount of interest from funders.
- This estimate applies only to the average, not marginal, dollar of research funding.
- I’ve estimated the “future reductions in the burden of cancer attributable to publicly financed research” at 50% based on Dario’s input and some checking of whether this would be consistent with past reductions in the burden of cancer (more in the spreadsheet).
- I’ve only considered the National Cancer Institute budget; I haven’t considered private philanthropic funding, overseas funding, private R&D costs or potential increased health care costs (though it’s possible that stronger progress on the basic science of cancer could lead to lower R&D costs and health care costs, if it leads to drugs with stronger and easier-to-prove effects).
- I’ve only considered the U.S. burden of cancer, not the global burden.
- I’ve only considered mortality, not morbidity.
The estimate is ~$2800 per life-year, which is substantially worse than our estimate of ~$80/DALY for LLIN distribution, but not so much worse as to make it implausible that well-directed scientific research (as opposed to “the average dollar spent on cancer research”) could have greater (even substantially greater) benefits. In order to create a better estimate down the line, I would want to:
- Focus in on a particular kind of highly promising research, to the extent possible, and attempt to focus on the future impact of the marginal rather than average dollar. This would require views about the best opportunities within life sciences funding and their possible product, of the kind that would require substantial input from scientists.
- Take private philanthropic, industry, and international funding into account; have a more sophisticated picture both of which funding to count as being “toward” the goal in question.
- Take the international burden of disease into account, and include morbidity as well as mortality.
Comments
“I’ve only considered the U.S. burden of cancer, not the global burden”
U.S. cancer deaths account for about 1 in 13 cancer deaths worldwide, and about 1 in 4 for OECD countries, so this is a pretty important assumption. This is especially so for drug development, where off-patent drugs or special discounts can broadly share the benefits of new treatments (pharmaceuticals tend to be the most effective treatments, too).
http://www.cancer.org/research/cancerfactsstatistics/cancerfactsfigures2013/index
http://www.who.int/mediacentre/factsheets/fs297/en/
Of course, one would also have to take into account foreign R&D spending. Page 26 of the report below says that in 2009 the U.S. government and foundations provided $7 billion of funding for R&D, pharma supplied $4.2 billion, and total funding was $19.2 billion.
The report also has sources for global cost of cancer figures, lost productivity estimates, fatalities by country, spending on treatment, and other info.
https://assets-livestrong-org.s3.amazonaws.com/media/site_proxy/data/c49ced3068f7205319cb1edf653dd91e0baee3ba.pdf
Random fact I once read:
A universal cancer cure would raise U.S. life expectancy by… three years. That’s not nothing, but it’s certainly not going to slay the dragon.
Carl, thanks for the additional info. I didn’t want to put in foreign burden without foreign funding and didn’t have a ready source for the latter; your links are helpful.
I posted comparison information for malaria R&D, burden, and treatment at my blog:
http://reflectivedisequilibrium.blogspot.com/2014/01/cancer-vs-malaria-burden-treatment.html
Thanks, Carl. A couple of comments on your piece:
1. I chose to focus on cancer for the reasons listed above, not because I think it’s the best giving opportunity available. Malaria seems a more difficult disease to analyze in terms of pinpointing spending (particularly “spending that isn’t tagged as malaria research but is implicitly aimed at it”), projecting the impact of R&D on burden (since much of the change in burden will likely depend more on aid and delivery than on new technology; I believe this to be much less true of cancer), and quantifying the burden (since much of it is morbidity in the case of malaria).
2. I don’t think the final section of your piece is well grounded.
I think there are good intuitive reasons to expect that there is more “low-hanging fruit” in malaria R&D than in cancer R&D, and to believe that the former is the better use of funds for global humanitarians. However, I don’t think it’s clear that recent developments support the “low-hanging fruit” idea. It could simultaneously be the case that malaria is more worth targeting than cancer, and that there is little low-hanging fruit on either front (and other approaches entirely are more promising). I remain very uncertain on the question of how much can be achieved by targeting R&D at diseases like malaria.
Re: RTS, S and artemisinin, I was aware of those points, but still see greater progress. Moving from no vaccines to a poor vaccine seems striking even if the first such is not widely deployed because of other good treatments.
Malaria has more existing good treatments, whereas for cancer, we are far from anything like 25%-50% effective cancer vaccines targeting half of cancer incidence. But my sense is that malaria research is also outpacing cancer at the margin in terms of introducing new ones.
Carl, I’m not sure about this. There are a lot of new developments in cancer research along the lines of “curing cancer type ___ in vitro or in mice”; I think these should be seen as importantly similar to developing technically impressive but not highly deployable solutions for malaria. If the criterion is improvements in scientific knowledge or proofs of concept, I think cancer research would have quite a bit of impressive progress to show; if it’s humanitarian outcomes, I think both cancer and malaria R&D haven’t produced much lately.
Again, I would guess that malaria R&D is in fact a better investment, but consider the empirical situation highly ambiguous.
I would just like to warn against those not in the field thinking that they can better direct research dollars. Except at a very high level (ie. cancer vs malaria), the scientists who make those decisions are really the best people to make them. I’ve seen politicians try to do this laughably bad.
Holden,
I don’t know how committed you are on exploring costs and benefits of medical research instead of other forms of research like, say, human capital formation.
There is widespread belief (even if not completely undisputed) that education is a major factor for economic development, which, considering a worldwide path towards convergence of life expectancy, I would guess would bring more benefits than further expanding medical knowledge.
However, the inner workings of education are a great mistery. What subjects should be taught? Having the subjects, what topics within the subject? In what order? What is the role of technology? What role is there for incentives? What to test? How to test?
It is possible to make a long list of unanswered questions in education. Research in the area, however, is still a small fraction of scientific research overall and a really small fraction of education spending in the world.
With medicine, at least, we know a lot more about what works and what doesn’t. Education is still working in the dark, with practitioners answering all of the questions I posed based completely on intuition and personal experience.
Tiago, I would put education research in the category of social sciences rather than hard sciences. The two present very different challenges. I believe it generally takes more background expertise to evaluate hard sciences work; that hard sciences tend to be better-funded and better-understood to begin with; and that hard sciences tend to be more tractable in the sense of “more research is likely to produce more useful, generalizable insights.” We haven’t yet done a serious investigation into social sciences funding, though we will in the future and we have some passing familiarity with them from our traditional work.
Holden,
I’ll be looking forward for your evaluation of social sciences research. With the risk of abusing the space, I would ask if what makes medical research hard science whereas education research is social – or soft – is not any inherent difference in the subjects but the way they have historically been treated.
That is to say, medicine research is not a science of laws, like physics, it is a science of probabilities, arrived at through randomized controlled trials. This method is wholly available to education research. Sure, it is a lot more expensive and time consuming to run randomized controlled trials in education, but if they are in fact conducted, the results are no less generalizable than with medicine.
I only add that the fact that medicine is so much more explored than education should count in favor of education research, which might have lots more low hanging fruit.
But maybe I’m wrong. Maybe the effects of education on growth are grossly overstaded (as the economist Bryan Caplan, for one thinks). If this is the case, as it might as well be, there would still be huge benefits to realizing the waste of resources both material and intellectual towards education policies.
Thank you for the minute attention you give commenters on this blog, and sorry for the long comment.
Tiago, I would put medicine RCTs somewhere in between hard sciences and social sciences, because the difficulty of reaching generalizable and actionable conclusions lies in between. Medicine RCTs are not the focus of what we are calling “life sciences” above.
Tiago, from my lay understanding, social sciences are much harder to generalize due to cultural or social factors. If you think about it, a medical intervention such as a drug or vaccine is likely to work roughly the same in everyone’s body regardless of the economic structure or social history of their country, whereas an educational intervention may be extremely effective in e.g. Confucian countries but totally ineffective in e.g. traditionally Muslim countries.
Comments are closed.