[Added August 27, 2014: GiveWell Labs is now known as the Open Philanthropy Project.]

As noted previously, I’ve been investigating the question: How does the “good accomplished per dollar” of life sciences funding compare to that of other giving options (such as LLIN distribution)? At this point, I believe that there is very little in the way of academic literature that can shed light on this question. I’ve put together some very limited and preliminary analysis; to produce something better, I’d want to work with both generalist scientific advisors and at least one economist.

Existing literatureIn order to assess the existing literature on returns to life sciences funding, we have:

• Examined a review of economic literature on measuring the returns to R&D in general (Hall, Mairesse and Mohnon 2009) and some of the papers it references. The literature discussed in this review tends to focus on private industry R&D, and does not appear highly relevant to the question we’re focused on.
• Examined a review of literature on “the impact of publicly funded and biomedical research” (Sampat 2011) and some of the papers it references, and spoken to the author.
• Read Measuring the Gains from Medical Research: An Economic Approach, spoken briefly with one of the authors, and spoken with another economist he referred us to.

Our takeaways on the relevance of this literature for our purposes:

• Some research focuses on the role that general scientific research and technology innovation has played in reductions in the burden of disease, and generally concludes that it has played a major role (see, in particular, Chapter 4 of Measuring the Gains from Medical Research: An Economic Approach, which discusses heart disease, and The Contributions of Improved Therapy and Earlier Detection to Cancer Survival Gains, 1988-2000 which discusses cancer).
• Some research attempts to disentangle the roles of publicly and privately funded research, and otherwise isolate which kind of research has been responsible for humanitarian gains (and on what timescales); we haven’t found this work compelling.
• Some research attempts to quantify the economic value of past reductions in disease burden, and juxtapose it with the amount the U.S. spends on public research (without directly estimating how much of the reduction is due to public research). This research generally concludes that benefits are significant and far outweigh the costs. For example, chapter 1 of Measuring the Gains from Medical Research: An Economic Approach argues that the value of improved life expectancy over the last ~100 years (monetized using the value of a statistical life framework) is comparable to total measured GDP growth over that time; chapter 2 estimates that gains from 1970-2000 were worth about $46 trillion total, net of increased spending on health care, as against an NIH budget of ~$35 billion per year as of 1995.
• Overall, relatively little has been done in this area, and we haven’t found any research that attempts to (a) make forward-looking estimates of the likely future returns to investments in life sciences; (b) estimate the value of the marginal (as opposed to average) dollar spent on research.

An extremely rough forward-looking cost-effectiveness analysis for cancer researchI created a rough forward-looking cost-effectiveness analysis for cancer research, in order to start thinking through what a more sophisticated analysis would look like. The analysis is available here. Note that:

• I chose to focus on cancer because (a) most of the burden is via mortality rather than morbidity, and I believe mortality statistics to be generally easier to access and verify than morbidity statistics; (b) because cancer is a politically popular disease for research funding, it’s likely that the “official” cancer budget of the NIH is a reasonable proxy for – or perhaps overestimate of – how much is being spent on cancer research. (By contrast, if I focused on a less politically salient disease, I’d worry that I was substantially undercounting the funding directed toward it, as funding that was officially earmarked for more popular diseases could effectively be financing highly relevant research. In the case of cancer, while I expect some research outside the National Cancer Institute to be relevant to cancer, I also expect some National Cancer Institute research to be highly relevant to other diseases, and I think it is not unreasonable to think of the National Cancer Institute budget as approximating the “costs of scientific progress on cancer.”)
• I would guess that cancer research produces less benefit per dollar spent than research in many other categories, partly because I perceive cancer as having a relatively large amount of interest from funders.
• This estimate applies only to the average, not marginal, dollar of research funding.
• I’ve estimated the “future reductions in the burden of cancer attributable to publicly financed research” at 50% based on Dario’s input and some checking of whether this would be consistent with past reductions in the burden of cancer (more in the spreadsheet).
• I’ve only considered the National Cancer Institute budget; I haven’t considered private philanthropic funding, overseas funding, private R&D costs or potential increased health care costs (though it’s possible that stronger progress on the basic science of cancer could lead to lower R&D costs and health care costs, if it leads to drugs with stronger and easier-to-prove effects).
• I’ve only considered the U.S. burden of cancer, not the global burden.
• I’ve only considered mortality, not morbidity.

The estimate is ~$2800 per life-year, which is substantially worse than our estimate of ~$80/DALY for LLIN distribution, but not so much worse as to make it implausible that well-directed scientific research (as opposed to “the average dollar spent on cancer research”) could have greater (even substantially greater) benefits. In order to create a better estimate down the line, I would want to:

• Focus in on a particular kind of highly promising research, to the extent possible, and attempt to focus on the future impact of the marginal rather than average dollar. This would require views about the best opportunities within life sciences funding and their possible product, of the kind that would require substantial input from scientists.
• Take private philanthropic, industry, and international funding into account; have a more sophisticated picture both of which funding to count as being “toward” the goal in question.
• Take the international burden of disease into account, and include morbidity as well as mortality.

[Added August 27, 2014: GiveWell Labs is now known as the Open Philanthropy Project.]

As noted previously, I’ve been exploring the question, “What are the best opportunities for funders aiming to contribute to progress in life sciences (i.e., biology and medicine)?” This post lays out what we’ve done to date and how we plan to move forward.

The basic approach I’ve taken: interviews and immersionBecause this area is so different from anything GiveWell has looked at before, I’ve initially tried to “immerse” myself in it: I’ve taken opportunities to have extended and low-stakes interactions with scientists and learn a little bit of the basic knowledge underlying life sciences, without having specific questions or goals in mind.

The most helpful person in this endeavor has been Dario Amodei. Dario is currently a postdoctoral research fellow at Stanford Medical School studying proteomics; he is a recipient of the Hertz Fellowship and winner of the Hertz Foundation’s 2012 Thesis prize for his Ph.D. dissertation in the field of neuroscience. He is also a longtime GiveWell fan and supporter (see his 2009 guest post on the GiveWell blog), and the two of us have been housemates since GiveWell relocated to San Francisco.

For the early stages of investigation, Dario has served as our informal scientific advisor,* guiding our investigation by recommending readings, offering connections, etc. I feel that he has been a good fit for this role because he combines experience as a scientist with an extremely strong understanding of GiveWell’s values and goals, and is highly accessible to us. We’ve come to believe that in the early stages of an investigation, when we often don’t know which questions to ask, the ability to have extended, repeated, friendly, low-stakes interactions to get “grounded in the basics” is crucial, and we’ve taken a similar approach to exploring policy advocacy (which we wrote about previously).

Under Dario’s guidance, I have:

• Conducted a series of interviews with representatives of major life sciences funders including Howard Hughes Medical Institute (the largest private philanthropic funder of scientific research in the U.S.), Wellcome Trust (the largest private philanthropic funder of life sciences in the U.K., with similar scale to Howard Hughes and to the British government’s research funding), and the American Cancer Society. (We have also had several other off-the-record conversations, including with U.S. government staff.)
• Studied a small amount of the basics of cell biology. Specifically, I read chapters 1, 5, 6, 7, 8, 18 and 20 from Essential Cell Biology; reviewed the more detailed version of chapter 7; and questioned Dario about them at length. The two of us also discussed the abstracts of several NIH grant proposals and Nature papers. The goal here was simply to improve my basic comprehension of common terms used in discussing life sciences; after the calls discussed above, he and I agreed that it was important for me to improve on this front.
• Taken opportunities to attend larger meetings of scientists in appropriate settings: relatively low-stakes settings in which communication in layperson’s terms was encouraged. I attended a meeting at the Banbury Center, intended to bring together people with different areas of expertise to discuss possible approaches to oncology; a meeting organized by Good Ventures (with Dario’s help) to discuss potential prizes in science; and several meetings of Vannevar Group. All of these meetings were organized by Dario or people in his network.

Several perspectives on giving opportunities aiming at furthering progress in life sciencesOne natural way to look for underfunded opportunities is to focus on the question of “what diseases and conditions should be getting more focus and funding?” It’s possible that some diseases and conditions are overlooked because they primarily affect people in the developing world (for example, malaria and tuberculosis); it’s possible that some are overlooked because they don’t have the same sorts of interest groups behind them (for example, chronic pain and other symptoms of aging, which have no analogue to groups like American Cancer Society). Prior to starting this investigation, this was the main framework we were thinking about for finding overlooked giving opportunities, and we believe there may be a lot of potential in thinking in this way.

However, the interviews we’ve done so far have led us to believe that this is far from the only – and may not be the best – framework for thinking about “what sorts of science are underfunded.” Much scientific research – including many of the discoveries that have been associated with the Nobel Prize – is not clearly associated with a particular disease or condition at all. Conversely, it’s possible that funding officially directed toward a particular disease or condition is instead used for work with more general ambitions. Thinking about “which disease and condition should be targeted” is a natural framework for humanitarian-oriented non-scientists, but it isn’t necessarily the most natural way of dividing up different types of science from a scientist’s perspective.

The interviews we’ve done so far have included several criticisms of the primary existing science funding mechanisms, particularly those of the National Institutes of Health (NIH). The NIH provides over half of all U.S. funding in life sciences, and about half of all NIH funding is via a particular grant mechanism, called the R01. The R01, and the NIH more broadly, is perceived as a major factor in researchers’ incentives, and observations about “what the science world needs more of” seem closely connected with critiques of the NIH and the R01. I refer to the NIH, and particularly the R01, as “the existing” system in the bullet points below, listing some of the claims discussed in our interviews:

• The existing system favors researchers with strong track records, and is not good at supporting young investigators. This was the most commonly raised concern, and is mentioned in all three of our public interviews.
• The existing system favors a particular brand of research – generally incremental testing of particular hypotheses – and is less suited to supporting research that doesn’t fit into this mold. Research that doesn’t fit into this mold may include:
  • Very high-risk research representing a small chance of a big breakthrough.
  • Research that focuses on developing improved tools and techniques (for example, better microscopy or better genome sequencing), rather than on directly investigating particular hypotheses.
  • “Translational research” aiming to improve the transition between basic scientific discoveries and clinical applications, and not focused on traditionally “academic” topics (for example, research focusing on predicting drug toxicity).
• The existing system focuses on time-consuming, paperwork-heavy grant applications for individual investigators; more attention to differently structured grants and grant applications would be welcome. These could include mechanisms focused on providing small amounts of funding, along with feedback on ideas, quickly and with minimal paperwork, as well as mechanisms focused on supporting larger-scale projects that require collaboration between multiple investigators.

We don’t yet endorse any of these claims, and we’re aware that in many cases the NIH has mechanisms that intend to address alleged shortcomings. In order to assess these claims, we feel that we would need staff and/or advisors with substantial technical knowledge, as discussed in the next section.

The centrality of scientific advisorsThe basic approach we’re picturing for GiveWell Labs runs roughly as follows:

1. Generalist GiveWell staff (everyone we currently employ should be thought of as a generalist) gather information on many different causes, through methods including interviewing those with relevant expertise, and seek to understand different causes’ importance, tractability, and “crowdedness” at a high level.
2. For particularly promising causes, GiveWell will consider hiring staff to specialize in these causes (or recommending that major philanthropists hire staff for this purpose), since we believe that having the appropriate amount of context to evaluate giving opportunities in a cause may take in excess of a year (even with good access to relevant experts).
3. Generalist staff will be responsible for evaluating cause-specific staff; cause-specific staff will be responsible for synthesizing expert opinion and other information to make giving recommendations. However, when investigating scientific research, we believe that another link in the chain is needed. We believe that we need generalist scientific advisors in order to evaluate and compare different approaches to scientific funding at a high level. This is because, based on the early investigation detailed above, I don’t feel positioned even to understand the meaning – much less the plausibility – of many key claims. For example, in order to evaluate the claim that “high-risk” research is under-invested in, I’d want to be able to distinguish high-risk from lower-risk research and be able to assess how much is invested in each – something that I believe is likely to require substantial technically informed judgment calls. To some degree, this dynamic holds for any cause. In any field, there will be people whose greater experience makes subtle contributions to how they see many debates and evaluate many interventions. But scientific research is distinguished from other fields I’ve investigated by the degree to which expertise (often in the form of understanding a very specific, near-universal knowledge base that requires extended study to absorb) is a prerequisite to understanding the basics of what people are claiming and how one might reasonably go about comparing and drilling down on claims. Therefore, we chose to pause further investigation of scientific research until we recruit an appropriate set of generalist scientific advisors. (We have very recently – as of this week – formed a few new advisory relationships and restarted research.)

What we’re looking for in generalist scientific advisorsWhat follows is a draft spec that we have started to circulate in search of generalist scientific advisors. We plan to distribute it both within our network and more broadly, and initially Dario and I are evaluating candidates based on scientific aptitude and accomplishments (Dario and his network assess this) and fit with GiveWell (I assess this). So far, we have circulated this spec informally and found a few advisors we are working with on a trial basis; yesterday we started distributing the spec more broadly.

About usGiveWell (www.givewell.org) finds outstanding giving opportunities and publishes the full details of its analysis to help donors decide where to give. It has historically focused on direct aid, but is now exploring the idea of funding (via recommendation) scientific research. GiveWell works as an advisor to Good Ventures (www.goodventures.org), a foundation funded by Dustin Moskovitz (co-founder of Facebook and, more recently, Asana) and run by Cari Tuna.

GiveWell’s ultimate goal is to accomplish humanitarian good, but it believes that some of the most beneficial scientific breakthroughs will come via long-term, high-risk fundamental research and is highly open to promoting work in this category.

GiveWell is not limited to any particular field, disease, biological condition, or population. We seek to broadly consider and identify the approaches that are most (a) promising and (b) under-invested in by other funders. We are likely to focus initially on life sciences (though this will include other areas with applications to life sciences, e.g., novel measurement platforms), but intend to explore other areas as well. We are seeking both advisors with backgrounds in life sciences and/or physical sciences who think strategically about future implications of fundamental research, with an eye to long-term, transformative impact.

GiveWell seeks scientists who are deeply experienced and (depending on career stage) accomplished, and who are committed to identifying the most promising science, regardless of how unconventional it may be. GiveWell hopes to find opportunities that are potentially transformative, and appear after substantial scrutiny (by world-class scientists) to represent intelligent (if sometimes risky) philanthropic investments.

The roleGeneralist scientific advisors will help GiveWell find the best opportunities for a funder to accomplish good – whether it takes the form of pursuing short-term, cost-effective research with direct humanitarian applications, or pursuing long-term, fundamental research aiming for breakthroughs with wide-reaching and difficult-to-predict applications. Activities may include:

• Interviewing other funders’ staff and scientific advisors to evaluate different approaches to research funding.
• Finding, interviewing, and evaluating scientists doing particularly promising work.
• Assessing (via literature review, interviews, etc.) the tractability of particular scientific goals and the likely future progress of particular fields
• Seeking out, and presenting arguments regarding, inefficiencies in the current infrastructure for funding scientific research

GiveWell expects advisors to serve on a part-time basis, with significant variability in time commitment from week to week. In our early stages, we will likely seek to work with advisors on a trial basis, without long-term commitments. GiveWell’s staff will provide support (scheduling, documentation, etc.) as appropriate.

The teamWe expect initially to retain 5-10 scientific advisors, at a variety of career stages (potentially ranging from junior faculty or even postdoctoral fellows to very senior scientists). We are interested in advisors from both academia and industry. This team will interact with each other regularly and bring diverse perspectives to the questions we’re asking. These will not be the only scientists we work with, but they will be the team of “generalists” that has primary responsibility for broad, comparative analysis identifying particularly promising areas and people to invest in.

Qualifications

• Strong scientific accomplishments for one’s career stage.
• Technical intelligence, as assessed by both track record and by interviews (with scientists we retain for interviewing).
• Breadth of interests and abilities. Generalist scientific advisors should be interested in – and good at – comparatively assessing science and scientists in a broad variety of fields.
• Willingness and ability to think critically and strategically but unconventionally, in order to find the ways in which existing funding infrastructure for science falls short of its potential.
• Deep and broad network of high-quality scientific colleagues with whom the advisor may consult for specific technical expertise as needed to assess opportunities.
• Willingness and ability to communicate one’s views to non-technical people, and technical experts outside one’s own scientific field.
• Interest in assessing the potential connection between scientific progress and humanitarian progress, both over the short and long run.

This is an opportunity to influence the scientific priorities of a potentially major source of funds whose focus is on humanitarian good accomplished, without pre-commitments to particular fields, diseases, conditions, or populations. We are seeking people who are excited by the opportunity to have this influence, passionate about finding opportunities for scientific funders to accomplish good, and accordingly prepared to accept compensation that is reasonable but below what they could earn consulting for for-profit organizations.

* We’ve recently started paying Dario for work that is focused on GiveWell’s priorities. These expenses have fallen under the heading of “independent research contractors,” which are covered by a specially earmarked grant from Good Ventures.

[Added August 27, 2014: GiveWell Labs is now known as the Open Philanthropy Project.]

One of our priorities over the last few months has been to learn, broadly speaking, about scientific research and the role philanthropy can play in it. Along with political advocacy (which we’ve been writing about recently), we feel that this is one of the major categories of philanthropy that we’re currently least well suited to understand.

This is first in a series of three posts. It discusses:

• Why we believe it’s important to explore scientific research as a philanthropist.
• Why we’ve tentatively chosen to start by focusing on life sciences (biology, medicine).
• The key questions we’ve been focused on.

Why explore scientific research?We expect it to be a difficult and long-term project to gain competence in scientific research funding. The reasons we think it’s worth doing are that:

• Scientific research is extremely prominent in the list of philanthropy’s claimed success stories, and the conceptual fit with philanthropy is strong.
• The “bang for the buck” of funding or influencing scientific research could be extremely high, since new findings become public goods with potentially global effects.
• Much of the Gates Foundation‘s current work is in the category of trying to develop new technologies aimed at helping people in the developing world, an approach that strikes us as potentially promising, and difficult to assess without a basic competence in funding scientific research.
• In addition, even scientific research that is not aimed at helping the worst-off can have potentially huge flow-through effects. We have, in the past, made some preliminary attempts to investigate certain types of scientific research, and haven’t made much progress. We now recognize that – for reasons we will lay out below and in future posts – making serious traction in this area is going to be a very challenging long-term project.

Starting with a focus on life sciences (biology, medicine)

• According to data we’ve pulled from a government survey on R&D funding at universities (details of the query are in the spreadsheet) (XLS), the total amount spent on (university-based) research in “biological sciences” and “medical sciences” is nearly equal to the amount spent on all other categories of research combined. This sort of research is also the most common focus (within scientific research) among today’s philanthropists.One could argue that this means we should focus somewhere else, with the reasoning that we should be looking for “uncrowded” areas. However,
  • “Heavily funded” isn’t synonymous with “overcrowded” – what matters is how much funding is in an area relative to how much room for more funding there is, and at this point we have no way of assessing the “room for more funding” for any type of scientific research until we get a better grasp on the basics.
  • When we know little enough about a type of philanthropy, there is some appeal to examining the most highly-funded areas, to get a sense of “what a highly-funded area looks like” and how other areas might look different if more funding came into them.
  • We would consider pursuing a meta-research focus, i.e., a focus on advocating for changes in research practice and infrastructure; if we did this, working in a well-funded area would mean more opportunity for leveraged impact.

Two separate questions

We’ve been working in parallel on two questions:

1. Within life sciences, what are the best opportunities for funders?
2. How does the “good accomplished per dollar” of life sciences funding compared to that of other giving options (such as LLIN distribution)?

In theory, we’d want to answer these in the order given, because one needs to know the best available giving opportunity within a category in order to assess that category against other categories we’ve assessed the best available giving opportunities for. However, we expect assessing #1 to be a very long-term project, so in the meantime we’ve done what we could to assess the state of the existing literature on #2. We’re now at a stopping point on both questions: we believe that moving further on #1 requires recruiting scientific advisors, and moving further on #2 would require commissioning (rather than just reviewing) research, which is something we wouldn’t be comfortable doing until after more progress on #1.

Future posts will discuss our progress on both.