Translational science and the “valley of death”

As we’ve looked for potential gaps in the world of scientific research funding – focusing for now on life sciences – we’ve come across many suggestions to look at the “valley of death” that sits between traditional academic research and industry research. Speaking very broadly, the basic idea is that:

  • The world of life sciences research has become increasingly complex, with a widening gulf between traditional academic research – which aims at uncovering fundamental insights – and industry work, which is focused on developing drugs and other marketable products.
  • There is a lot of work that could be done on figuring out how to apply insights from academia and therefore close this gap. However, this “translational science” tends not to fit well within the traditional academic ecosystem – perhaps because it focuses on useful applications rather than on intellectual novelty – and so may be under-supported.
  • As a result, the world is becoming increasingly inefficient at translating basic research into concrete applications, and this explains why drug development has seemingly been slowing despite increasing expenditures on biomedical research (though recent data suggests that this trend may be changing).

For examples of this basic argument, see Translational Research: Crossing the Valley of Death (Nature 2008) and Helping New Drugs Out of Research’s ‘Valley of Death’ (New York Times 2011). In particular, the Nature article contains a pair of charts giving a rough illustration of two basic trends that may represent the causes and consequences of the growing “valley of death”: (a) rising government expenditures on research, increasingly supporting pure academics as opposed to medical practitioners, and (b) declining drug development production despite rising pharmaceutical R&D expenditures. (As noted above, more recent data may indicate that these trends are changing.)

We find this theory extremely challenging to assess for several reasons. One is that there doesn’t appear to be any one clear definition of “translational science” or of the “valley of death,” and some “translational science” seems quite well-suited to industry – to the point where it’s not entirely clear why we should think of it as a candidate for philanthropic or government funding at all. Another is that there has been growing interest in the issue over the last decade, including the 2011 debut of NCATS, a new institute at NIH dedicated to translational science; it’s hard to say whether translational science still represents one of the main “gaps” in the existing system.

Finally, there are other strong explanations for the observed decline in pharmaceutical output. The most comprehensive article I’ve seen on the subject names multiple possible explanations for the decline, many having to do with regulatory issues as well as the inherent challenges of improving on already-available drugs. The “valley of death,” as outlined above, doesn’t figure prominently in its account.

I am skeptical of some of the arguments people have made for the importance of translational science. These arguments often do not distinguish between different possible definitions of “translational science,” and often do not make a strong case that nonprofit funding (as opposed to industry funding) is what’s needed. In addition, it seems quite possible to me that the goals of promoting “translational science” might be better served by policy change (on regulatory and intellectual property law, for example) than by scientific research. With that said, I think the idea of translational science is worth keeping in mind, and that certain kinds of research in this category could be under-invested in because they do not fit cleanly into an academic or for-profit framework.

The rest of this post will:

  • List several different definitions of “translational science” that I’ve come across, noting that in some cases it isn’t clear why a proposed sort of research is a fit for the nonprofit as opposed to for-profit world. More
  • Briefly discuss the recent creation of the U.S. government’s National Center for Advancing Translational Sciences (NCATS). More
  • List some other potential reasons for the decline in pharmaceutical output, which may point to solutions outside of “translational science.” More

Five different definitions of “translational science”

The Nature article on translational science states, “Ask ten people what translational research means and you’re likely to get ten different answers.” Here I give five definitions I’ve come across that seem quite distinct from each other – particularly in terms of what they imply about the appropriateness of nonprofit funding.

1. Not-for-profit preclinical research. “Preclinical research” here refers to categories D-E (mostly E) from my previous post on different phases of scientific research. A possible new medical treatment is often first tested “in vitro” – in a simplified environment, where researchers can isolate how it works. (For example, seeing whether a chemical can kill isolated parasites in a dish.) But ultimately, a treatment’s value depends on how it interacts with the complex biology of the human body, and whether its benefits outweigh its side effects. Since testing with human subjects is extremely expensive and time-consuming, it can be valuable to first test and refine possible treatments in other ways, including animal testing.

The idea of carrying out this kind of work outside of industry – both in vitro screening to identify potential new medical technologies, and other tests to improve estimates of their promise – appears to be one of the most common definitions of translational research.

 

  • The Nature article states “For basic researchers clutching a new prospective drug, it might involve medicinal chemistry along with the animal tests and reams of paperwork required to enter a first clinical trial … In some sense much translational research is just rebranding — clinical R&D by a different name.”
  • The NYT article states, “For a discovery to reach the threshold where a pharmaceutical company will move it forward what’s needed is called “translational” research — research that validates targets and reduces the risk. This involves things like replicating and standardizing studies, testing chemicals (potentially millions) against targets, and if something produces a desired reaction, modifying compounds or varying concentration levels to balance efficacy and safety (usually in rats). It is repetitive, time consuming work — often described as “grunt work.” It’s vital for developing cures, but it’s not the kind of research that will advance the career of a young scientist in a university setting.”
  • The examples of translational research listed by the Science: Translational Medicine journal seem to fit this basic framework, as does much of the activity described in the most recent annual report for NCATS (the recently created government institute focused on translational science).

I don’t feel that there’s a clear case for supporting this kind of work with nonprofit (government or philanthropic) funds. Unlike much basic research, this sort of work seems generally to have a very specific medical application in mind, and I believe that companies are often able to monetize the value created by new technologies they develop (especially drugs). Therefore, when looking at this kind of “translational science,” I think it is fair to ask: “If this research is generating more expected value than it costs, why isn’t industry investing in it? Why the need for nonprofit funds?”

There are a few possible answers. One is that that this kind of research may have positive expected value, but it is too risky for any one investor to take on – even the large industries that consider investing in it. This may be true, but I’ve rarely seen it spelled out by comparing the level of risk in particular kinds of research to the level of risk that various industry players are likely to bear. In addition, if risk is the key issue, this doesn’t necessarily call for a nonprofit solution. An economics-and-financed-focused group at MIT has proposed that a large enough for-profit fund – perhaps made possible via financial engineering – could result in much more investment in this type of research. This group appears to be working on a collaboration with NCATS. I am unsure about whether (and if so, for what diseases) financial engineering could ever turn a set of biomedical research investments (which I believe will generally have fairly correlated odds of success) into a high-grade-bond-quality investment, but I think it is an interesting approach.

There are other possible answers to the question. Perhaps industry can’t fully monetize the benefits its products bring, for reasons including the fact that (a) there may be many beneficiaries who can’t afford to pay (and don’t have insurance for paying) full price; (b) patents on medical products eventually expire. Taking existing health care and intellectual property law as a given, this could serve as some defense of investing nonprofit funds in “industry-style” research. I haven’t explicitly seen this argument made anywhere, except in cases where a disease has a clearly disproportionate impact on very low-income people.

In my limited readings on translational science, I’ve felt that this basic issue – the question of why we ought to support research with nonprofit funds when it appears to be a fairly good fit for industry – is rarely addressed.

2. Research on public goods – such as new tools and techniques – for preclinical research. The 2012-2013 annual report from NCATS cites several projects aimed at developing generally-useful tools and insights, that might be taken up by industry for a broad variety of purposes. For example, improving general methods for predicting how toxic a drug will end up being (page 7). In cases where such research aims to release public insights that others can build on, the case for a nonprofit model seems stronger than with the above category (targeted preclinical work with more specific aims).

3. Improving communication between clinical and academic professionals, via multidisciplinary groups as well as multidisciplinary career tracks. The idea here is that academics might do more useful research if they had more observations about how medical care works in practice – not only in terms of understanding the greatest needs, but also in terms of potentially drawing scientific inspiration from observing the effects of treatments on patients. It could be argued that there were more medical breakthroughs in the past, before academic biology and clinical medicine became as separated as they are today. A related idea is that it might be productive to provide academics with more support in understanding market demand for the kinds of technologies they’re working toward, via market research, competition analysis, etc.

The Nature article states,

Back in the 1950s and 60s, basic and clinical research were fairly tightly linked in agencies such as the NIH. Medical research was largely done by physician–scientists who also treated patients. That changed with the explosion of molecular biology in the 1970s. Clinical and basic research started to separate, and biomedical research emerged as a discipline in its own right, with its own training … Science and innovation have become too complex for any nostalgic return to the physician–scientist on their own as the motor of health research. Reinventing that culture is therefore the focus of the CTSCs [CTSCs are centers supported by NCATS] in the form of larger, multidisciplinary groups, including both basic scientists and clinicians, but also bioinformaticians, statisticians, engineers and industry experts. Zerhouni says he expects them to be breeding grounds for a new corps of researchers who will effectively stand on the bridge and help others across.

This issue was a major focus of a 2000 roundtable on clinical research as well.

4. Conducting academic research in the “style” of industry research. The NYT article highlights research-focused nonprofits that are “intensely goal-directed and collaborative; they see the creation of new cures as a process that needs to be managed; and they bring a sense of urgency to the task.” The Nature article mentions that CTSCs (the same NCATS-supported centers discussed above) will evaluate scientists “with business techniques, such as milestones and the ability to work in multidisciplinary groups, rather than by their publications alone.” The focus on collaboration and setting specific goals seems conceptually distinct from a focus on the preclinical phases of research, though I’ve generally seen the two side by side in discussions of translational science.

5. Supporting and improving clinical trials. Clinical trials (category F from my previous post on different phases of scientific research) are generally the most expensive part of developing new medical technologies, and they are traditionally paid for mostly by industry. NCATS reports (page 10) working to improve their cost-effectiveness and usefulness in a variety of ways, including improving data sharing and recruitment of participants: “investigators work together on data sharing, multisite trial regulatory hurdles, patient recruitment, communication and other functional areas of research to enhance the efficiency and quality of clinical and translational research … the University of California Research eXchange (UC ReX) Data Explorer is a secure,online system that enables cross-institution queries of clinical aggregate data from 12 million de-identified patient records derived from patient care activities.”

The recent creation of NCATS

The National Center for Advancing Translational Sciences (NCATS) was established in December 2011, making it “the newest of 27 Institutes and Centers (ICs) at the National Institutes of Health (NIH).” Its annual budget is in the range of $600 million (page 4). Going over its 2012-2013 annual report, I note quite a broad variety of activities, seemingly including all five of the categories described above (note that it spends over $400 million per year (page 5) on clinical research centers, which I believe are the same as the centers referred to under #3 and #4 from the previous section). NCATS also appears to engage in attempting to improve policy (e.g., regulation and intellectual property law – see page 22). It appears to pay special attention to rare diseases (pages 13-16), though the reasons for this are not obvious to me.

It appears to me that the creation of NCATS was met with some negative reaction from the scientific community, as evidenced by three posts (1, 2, 3) by chemist Derek Lowe. The negative reaction appears to be based partly on a perceived vagueness of mission and partly on fears of diverting funding from other science.

Most discussion I’ve seen of the “valley of death” and need for translational science pre-dates the creation of NCATS. It is unclear to what extent the creation of NCATS has addressed the relevant gaps.

I should also note that there are longer-running NIH mechanisms for supporting translational science, such as SBIR and STTR grants for “domestic small businesses [that] engage in R&D that has a strong potential for technology commercialization.”

Why has pharmaceutical productivity been declining in recent years?

Advocates of translational science often point to the seeming paradox of declining pharmaceutical productivity despite an ever-growing world of academic research (example). It appears that the decline in productivity has been real, and concerning (though there is also preliminary data that the situation may be changing). However, the decline has multiple possible explanations. The most useful-seeming paper I’ve seen on this topic is Scannell et al. 2012, and I highly recommend it to those interested in the subject. A brief summary:

  • Over the past 60 years, “R&D efficiency, measured simply in terms of the number of new drugs brought to market by the global bio- technology and pharmaceutical industries per billion US dollars of R&D spending, has declined fairly steadily.” The authors call this “Eroom’s law” (Moore’s Law reversed).
  • The decline has occurred despite major improvements in efficiency on many fronts, from better understanding of biology to more efficient methods for screening large numbers of potential drugs. The authors are skeptical that there is any easy fix, noting that many potential fixes have been explored. They believe the magnitude and consistency of the decline in productivity “indicates that powerful forces have outweighed scientific, technical and managerial improvements over the past 60 years, and/or that some of the improvements have been less ‘improving’ than commonly thought.”
  • One of the major explanations the authors offer is the “better-than-the-Beatles problem”: each potential new drug has to compete with the best drugs developed to date in order to justify its development. It has to compete in clinical trials (making the trials challenging and expensive), and it has to compete for patients (making it hard to recoup revenue). The authors list some classes of drugs that “could have been blockbusters” 15 years ago, but today are not worth the costs and risks of development because there are existing drugs that are probably nearly as good.
  • The authors also hypothesize that drug development has transitioned to a fundamentally different new kind of approach, and that this approach – while superficially seeming clearly superior – may actually be inferior. In the past, drug development consisted largely of testing a relatively small number of potential drugs in animals (and humans), and observing results via trial-and-error. Today, there are more attempts to logically segment the process: for example, it is common to first identify a biological “target” via academic research, then look for compounds that do an outstanding job binding to the target in a lab environment, and only then to move on to animal/human trials. The authors believe that the old process may in fact have been more efficient (their arguments are somewhat complex and I do not summarize them here). It’s worth noting that if true, this hypothesis calls for a different approach to drug development, but does not necessarily call for “translational science” as defined above.
  • Many of the other explanations offered by the authors have to do with increasingly cautious regulation, which is likely responsible for longer, more expensive, more challenging clinical trials. From my limited readings on the history of biomedical research, it seems to me that getting drugs tested and approved used to be much easier than it is today, and that many key experiments were highly speculative and dangerous; such experiments would have been much more difficult to carry out with today’s regulation and social norms.

If the authors were right, it wouldn’t necessarily mean translational science isn’t valuable. It does seem true that academic biology has gotten far more complex, and translational science may be crucial in taking advantage of improved basic science and thereby improving pharmaceutical productivity. But I believe it is far from clear that translational challenges are the source of the productivity decline we’ve seen.

The Most Good You Can Do

The Most Good You Can Do is a new book by Peter Singer. It is an introduction to effective altruism, which we’ve previously defined as “trying to do as much good as possible with each dollar and each hour that we have.”

It emphasizes the importance of giving both generously and effectively – highlighting the role of GiveWell’s charity recommendations – as well as discussing how to do as much good as possible with one’s career, and discussing some of the causes that effective altruists tend to be particularly interested in. It is a broad, well-written introduction to the set of ideas associated with effective altruism, and we highly recommend it to those interested in the topic.

Top charities’ room for more funding

In December, we published targets for how much money we hoped to move to each of our top four charities, with the expectation of revisiting these targets mid-year:

In past years, we’ve worked on an annual cycle, refreshing our recommendations each December. This year, because we anticipate closing (or nearly closing) the funding gaps of some of our top charities during giving season and moving a significant amount of money (~$5 million) after giving season before our next scheduled refresh, we plan to update our recommendations based solely on room for more funding in the middle of next year. We’re tentatively planning to do this on April 1st, the earliest we will realistically be able to post an update on charities’ ongoing funding needs that accounts for the funds they will receive over the next few months.

These targets were based on a guess that GiveWell-influenced donors would give $7.5 million to our top four charities in December 2014 to March 2015 (excluding Good Ventures and a $1 million gift to SCI from an individual that we knew about prior to setting the targets). Our actual money moved for this period was about $8.7 million to the top four charities, plus $0.4 million that we can allocate at our discretion and have not yet allocated.

Over the past couple of months, we have spoken with each of our top charities to get updates on how much funding they have received from GiveWell-influenced and other donors and their current room for more funding. In sum, the amounts that our top charities raised as a result of our recommendations were broadly consistent with what we expected and there have not been any significant updates to the charities’ room for more funding. Therefore, we are not revising our recommended allocation (for every $7.5 given, $5 to AMF, $1 to GiveDirectly, $1 to SCI, and $0.5 to Deworm the World) at this time.

Summary for December 2014 to March 2015 (all figures in USD millions):

Charity Target from individuals (Dec 2014) Max from individuals (Dec 2014) Actual from individuals Summary
Against Malaria Foundation 5 5 4.5 Close to target
Schistosomiasis Control Initiative 1 1 1.1 On target
Deworm the World Initiative 0.5 1 0.7 Reached target but did not exceed max
GiveDirectly 1 25 2.4 Reached target but did not exceed max

 

Against Malaria Foundation (AMF)

Donations to AMF from GiveWell-influenced donors were short of our target by about $0.5 million. AMF is currently in discussions about funding several large-scale bednet distributions. It is our understanding that the amount of funding AMF has available is a limiting factor on both how many nets it can provide to each distribution it is considering and on how many discussions it can pursue at one time.

We have written before about AMF’s lack of track record at signing agreements for and successfully completing large-scale distributions with partners other than Concern Universal in Malawi. In 2014, AMF signed its first agreement to fund a large-scale distribution with another partner in a different country: IMA World Health in the province of Kasaï Occidental in the Democratic Republic of the Congo (more). The Kasaï Occidental distribution was scheduled to be completed in late 2014. We have not yet seen results from this distribution, and AMF’s track record of completing and reporting on successful large-scale distributions remains limited. AMF expects to be able to share information from this distribution in the next few weeks.

We plan to continue recommending funds to AMF for now and to reassess AMF’s progress later in the year.

GiveDirectly

In December, we noted that GiveDirectly could likely absorb up to $25 million in funding from GiveWell-influenced individuals. We tracked $2.4 million to GiveDirectly from these individuals and it is possible that GiveWell influenced several million dollars more – between February 2014 and January 2015, GiveDirectly received several million dollars from individuals who did not provide information on how they learned about the organization. We continue to believe that GiveDirectly has substantial room for more funding.

Schistosomiasis Control Initiative (SCI)

In December we set a target of SCI receiving $1 million from GiveWell-influenced individual donors and set the max we aimed for SCI to receive from this group at the same amount. We estimate that SCI received about $1.1 million based on GiveWell’s recommendation.

We have fairly limited information on SCI’s room for more funding because (a) SCI recently began working with a new financial director and is in the process of reorganizing its financial system, and so has not yet been able to provide us with a comprehensive financial update; and (b) SCI held a meeting on March 24 to allocate unrestricted funds and sent us a report from that meeting recently, which we have not yet had time to review. We will be following up with SCI to learn more about its plans and funding needs.

We plan to continue recommending funds to SCI because (a) our room for more funding estimates for SCI are rough and we believe there is a reasonable chance that SCI has room for more funding; (b) we expect to learn more about SCI’s room for more funding in the next few months; and (c) we do not expect SCI to receive a large amount of funding due to our recommendation over the next few months (since most donors give in December).

Deworm the World Initiative, which is led by Evidence Action

In December we set a target of $0.5 million from GiveWell-influenced individual donors to Deworm the World and set the max we aimed for Deworm the World to receive from this group at $1 million. We estimate that Deworm the World received about $0.66 million based on GiveWell’s recommendation.

It’s our understanding that Deworm the World may have opportunities over the next few years to support up to three deworming programs which could each cost several million dollars. We are in the process of following up with Deworm the World to learn more about how likely these programs are to require unrestricted funding from Deworm the World and when funding might become a bottleneck to moving forward with these programs.

We plan to continue recommending funds to Deworm the World.

Investigating neglected goals in scientific research

A major goal of the Open Philanthropy Project is to explore the topic of scientific research funding, starting with life sciences. This post discusses the process we’ve used so far, including some of the challenges we’ve faced and changes we’ve made in our investigation methods:

  • We first discuss some of the general challenges of finding good giving opportunities in this space.
  • We then introduce the concept of scientific research “gaps” – areas that the existing system doesn’t put enough investment into, leaving potential philanthropic opportunities. One type of gap, which we call a “neglected goal,” has been the focus of many of our efforts so far.
  • We discuss our process so far for investigating neglected goals, and our plans for the future. Future posts will discuss other types of potential gaps that we think could be very important, but would find more difficult to investigate: gaps in high-risk early-stage research and gaps in “translational” research that sits between academic and industry work.

Some challenges of investigating scientific research funding
There are several reasons that we’re finding this space particularly challenging.

One is the sheer level of expertise required. As we’ve written previously, we often don’t feel positioned even to understand the meaning – much less the plausibility – of many key claims. We’ve sought generalist scientific advisors to help us with this issue. My early intuition is that even with strong scientific advisors, it would take far more time to do a shallow investigation for a cause in this space than for a cause in another space, which may mean we have to take more shortcuts in order to arrive at priorities.

Another challenge – discussed more in later sections of this post – is that it seems more difficult to get engagement from relevant experts. When investigating U.S. policy and global catastrophic risks, we’ve found that many of the people we want to talk to see educating the public and/or influencing funders as an essential part of their role. This is often much less true of scientists.

Another challenge is that there is already a lot of funding going toward science (particularly life sciences). Tens of billions of dollars per year come from U.S. government and industry sources (see data we’ve pulled from a government survey on R&D funding at universities (details of the query are in the spreadsheet) (XLS)). In addition, philanthropic sources are significant: Howard Hughes Medical Institute alone spends in the range of $700 million per year. It’s reasonable to ask how much value a new funder – even a relatively large one – can add in this context.

Gaps and neglected goals
In our early conversations, we’ve therefore tried to focus on what we call “gaps” (or sometimes “broad market inefficiencies“) in the current system of scientists, funders and supporting institutions. We’ve sought out institutional constraints, suboptimal incentives, and other factors that might make the current system “miss” particularly strong opportunities to do good. Our hope is that a new funder might be able to do an outsized amount of good by supporting what the existing system can’t or won’t.

One type of gap is what I’ll call a “neglected goal”: a case where the existing system doesn’t put enough investment into solving a particular social problem or developing a particular kind of technology. For example, improving malaria control and elimination efforts is arguably a neglected goal. New drugs, vaccines, and other methods for controlling malaria could result in helping an enormous number of people, mostly low-income people in the developing world. Because relatively few wealthy people are affected by malaria, both government and industry may not put enough funding into the scientific research and development necessary to develop such things. Similar logic could be extended to tuberculosis, HIV/AIDS, and other diseases/conditions that primarily affect the global poor. The Gates Foundation has often publicly expressed this line of reasoning.

More broadly, I’m familiar with arguments for many other neglected goals, such as developing in vitro meat (to reduce animal suffering); improving treatment of chronic pain; treating and slowing symptoms of aging; developing methods of food production that are primarily useful in worst-case scenarios; developing biological interventions that enhance people’s abilities rather than simply counteract diseases; and more.

Neglected goals are not the only kinds of gaps we’ve considered. However, they are probably the easiest for non-scientists to understand and engage with. They stem from insufficient societal attention to a particular social problem, rather than issues with the details of how science is pursued. They are the focus of this post; future posts will discuss other types of potential gaps.

Our progress so far on evaluating neglected goals
When exploring an unfamiliar area, we often take the approach of:

  • Starting with a question that seems both (a) relatively tangible and tractable, and (b) reasonably representative of other, future investigations we might do in the area.
  • Being ready to take many wrong turns and rethink our process multiple times, until we come to a satisfying answer to the question.
  • Try to repeat our process for other similar, questions, making it more efficient and systematic over time.

In this case, we decided – about a year ago – to tackle the question: “How much good could we accomplish by funding research and development targeting the needs of the global poor?” This is probably the most straightforward example I know of a potential neglected goal. We recruited a small group of scientific advisors and set up periodic calls and meetings to work on this investigation, though it was a low priority for the year (our main goal was progress on U.S. policy and global catastrophic risks).

This question, while more contained than “What sort of scientific research should we fund?”, is extremely broad. Early in our investigation, we determined that it included the following sub-questions:

  1. Would it be most productive to focus on malaria, tuberculosis, HIV/AIDS, water/sanitation-related applications, or something else?
  2. What sort of research is likely to be most valuable? Presumably, there is more work being done on diseases like cancer than on diseases like malaria – but where is the gap greatest and most important? For example, would it be best to focus on earlier-stage research or later-stage research?
  3. How should we view the work of the Gates Foundation, which has shown a strong interest in the idea that this sort of research is underfunded? Should we believe that the Gates Foundation is taking the most effective possible approach (in which case the best research in this area would be research the Gates Foundation is funding) or that there are important opportunities it is missing?
  4. How valuable is the most valuable work in this area, compared to other possible uses of money?

After an initial period of better defining these questions, we started scheduling conversations with people who could bring a broad perspective to these questions, help us cut through many possibilities and identify the most promising types of research to support. We sought to speak with people who work on developed-world diseases such as cancer, in order to get perspective on #2 above. We sought to speak with people who have broad, cross-cutting roles working on developing-world diseases, in order to get perspective on #1 and #3.

Unfortunately, we ultimately weren’t able to make the sort of progress we had hoped for using this method. The people we hoped to speak with were not always interested in talking, and when they were it was often off the record. In most cases, we didn’t get much feedback on which types of research were most promising.

Trying a different approach, I spoke with two of our junior scientific advisors about the fields they knew best. I spoke about HIV/AIDS, tuberculosis, and other areas with Anna Bershteyn of the Institute for Disease Modeling (conversation notes here), and I spoke about malaria (primarily focused on drug development and risks of drug resistance) with Micah Manary, a graduate student specializing in malaria (conversation notes here). For simplicity, I focus here on the second conversation, though similar dynamics applied to both.

Rather than focusing on broad questions about the most promising paths, I aimed to gain a basic understanding of how malaria works, what the shortcomings are in current treatment/control methods, and how new drugs/diagnostics/vaccines/other technologies might help. I learned that one of the most important goals of drug development is simply to stay ahead in the race against drug resistance, which evolves rapidly and presents a major problem. I learned that due to the complexity of the parasite, there’s fairly little basis for predicting what sort of compound might be effective in killing it, so much of drug development comes down to essentially testing random compounds against malaria samples in a lab. Learning these things gave me a basic framework for thinking about how to quantify the humanitarian benefits of malaria-focused research and development: for any given research path (from funding drug development directly to funding higher-risk research aimed at making drug development more efficient), one could estimate the effect on the number of compounds tested per year, and from there estimate the impact of testing additional compounds on the threat of drug resistance among other things. Having this framework in mind made it easier to see how a variety of different research paths could be relevant, and what their benefits might look like.

Details are available via our public notes on the conversation.

We’re still far from having a view on the best research paths for malaria drug development, and we’re still far from an estimate of how much good (per dollar) funding such things would accomplish. However, the process for getting there seems more clearly defined and tractable than it did a year ago. I now see such investigations as ideally consisting of:

  • A scientist who can familiarize himself/herself with how a disease works, what methods we have for controlling it, what potential future technologies might be useful, and what kind of science might help speed these along – while also being available for intensive conversations with GiveWell staff.
  • A GiveWell staffer who can interview the scientist about potential research paths, think about the likely “good accomplished per dollar” of such paths, and work with the scientist to learn more about the most promising ones.

We’re currently working on two such investigations, each involving a different scientific advisor. Due to the complexity of the subjects and the limited time our advisors have available, we expect these investigations to be challenging and to take a few months. As we go, we’ll be thinking about how to make the process more systematic and efficient, how to increase our scientific advisory capacity, and how else we might narrow the field of possible neglected goals to focus on.

General progress and plans for GiveWell as an organization in 2015

This is the fifth post (of six) we’re planning to make focused on our self-evaluation and future plans.

Previous posts have discussed our 2014 progress on, and 2015 plans for,

This post outlines our plans and thoughts on issues that cut across these two projects, and pertain to GiveWell the organization as a whole.

Our staff has expanded significantly, and we expect to expand further.

  • At the beginning of 2014, we had 11 full-time staff and 1 Conversation Notes Writer; as of today, we have 18 full-time staff and 8 Conversation Notes Writers.
  • Of our current full-time staff, five work primarily on the Open Philanthropy Project, while the other thirteen do a mix of top charities work and cross-cutting work (including managing Conversation Notes Writers, vetting content from both projects, and administrative work). Currently, our payroll expenses are roughly evenly allocated between the two projects.
  • For most of 2013, Holden and Alexander were the only staff putting much time into the Open Philanthropy Project. In 2014, we shifted more of their time to the Open Philanthropy Project, as well as ~all of Howie Lempel’s time. These additions made it possible for the Open Philanthropy Project to make substantial progress, covered previously. Recently, we’ve added two more full-time staff to the Open Philanthropy Project team.
  • Through 2013, Elie was intimately involved in all charity reviews. He participated in most calls with charity representatives and carefully reviewed the details of all charity-review-related written work. This changed substantially in 2014 with newer staff taking on more research and management responsibility. These additions made it possible for GiveWell to produce much more in 2014 than in 2013, as covered previously. In 2014, we added 5 new staff members who primarily work on top-charities-related work. Natalie Crispin is now the direct supervisor for seven staff members.
  • We are hoping to involve more Research Analysts in the Open Philanthropy Project, particularly for helping with writeups of cause investigations and grants. Because of this, and our desire to build still more capacity for evaluating potential top charities, we are hoping to add 4-8 additional Research Analysts over the next 12 months. There are two future Research Analysts (previously Summer Research Analysts) who have accepted offers and are starting mid-year.
  • In addition, we are starting to seek cause-specific hires for the Open Philanthropy Project (discussed previously), and we have started to advertise for an Outreach Associate position to help us continue to maintain relationships with a growing number of people who give significantly to our top charities.

Improving management was a major focus of 2014. With our larger staff, we have put a lot more attention into recruiting-, training- and retention-related matters. We’ve implemented regular one-on-one meetings, employee satisfaction checkins, and a variety of practices to help staff engage with GiveWell and with each other (regular roundtables on topics of interest, social events, etc.) We’ve also been continually experimenting with moving more responsibilities – including management itself – to employees other than Elie, and as of early 2015 Natalie has taken over management of seven employees. We believe that running an organization of this size has some qualitative differences with running a ~5-person organization, and we’ve put deliberate attention into adapting. We may be writing more about this in the future.

We are planning to launch new websites for both GiveWell and the Open Philanthropy Project this year. We are working with the brand experience firm Cibo on both websites, and are aiming to launch both in April. This is a major time investment for us.

We have several reasons for launching new websites. First, the current GiveWell website was created in 2009 on a low budget, and we think it has major room for improvement. Second, creating separate websites for GiveWell and the Open Philanthropy Project is another step in the direction of creating clear separation between the two (more below). Finally, it is possible we will receive an unusual amount of publicity in 2015: we know of three different books on the subject of effective altruism, and one more that will feature it prominently, all slated to be published in 2015.

We are continuing to work toward separating GiveWell and the Open Philanthropy Project as brands, websites, and – eventually – organizations. In 2014, we struggled with confusion between the two. People we contacted as part of our Open Philanthropy Project work often Googled us, looked at GiveWell, and came away with mistaken impressions about what sorts of giving opportunities we seek for the Open Philanthropy Project. We have been taking small steps to ameliorate this (such as more consistent use of openphilanthropy.org email addresses and signatures), and in 2015 we plan to further work toward separation. In addition to launching a new website for the Open Philanthropy Project – to which we will be moving relevant content currently on GiveWell.org – we will begin conversations about what it would look like to form two separate organizations.

At the same time, we still see substantial overlap between the skills needed for the two projects. If GiveWell gets to the point of having spare capacity, this capacity could be very valuable to the Open Philanthropy Project. Such capacity could include Elie Hassenfeld’s time, if GiveWell becomes less dependent on his involvement; it could also include more junior staff, who could help with creating public writeups on our work among other things. Being able to flexibly allocate employees between the two projects is still valuable for us at this time.

Fundraising remains a priority. We are currently fundraising for unrestricted support, supporting a team that is allocated flexibly between Open Philanthropy Project and our more traditional work. Details are at our December 2014 update. For people donating to GiveWell by webform for regranting to top charities, we have added an option to allocate 10% to GiveWell unrestricted.

Outreach is still not a top priority for us. Our top priority for the Open Philanthropy Project is research (finding outstanding giving opportunities); we feel we have a long way to go on this front before it will make sense to put much effort into outreach. Our top priority for GiveWell is strengthening capacity and making the operation sustainable without needing as much involvement from the co-founders, which could result in a stronger Open Philanthropy Project (see above) as well as a more robust GiveWell.

In light of GiveWell’s maturing research process, and some early signs that growth in money moved from smaller donors is slowing (more in our upcoming metrics report), there is a growing argument for putting more effort into outreach. Still, we see strengthening capacity as the more important goal for the coming year, especially in light of (a) the potential benefits for the Open Philanthropy Project; (b) the fact that there are now substantial efforts outside of GiveWell aiming to drive more donations to our top chariites. The latter include several organizations promoting generous, effective giving as well as the books mentioned above.

We are putting substantial time into new websites this year, which we feel is probably the highest-value activity for outreach. In the future, if GiveWell becomes less dependent on co-founder involvement and higher-capacity, we may further increase our attention to outreach.

We don’t see other major issues (in the “cross-cutting between Open Philanthropy Project and traditional work” category) that need addressing in 2015.

Open Philanthropy Project: Progress in 2014 and Plans for 2015

This is the fourth post (of six) we’re planning to make focused on our self-evaluation and future plans.

We’re in the midst of finalizing detailed updates on Open Philanthropy Project progress and plans. This post gives a high-level summary, comparing our progress and stage of development with what we hoped for as of a year ago. In brief:

  • We made substantial progress on our main priorities: U.S. policy and global catastrophic risks. The precise nature of our goal (“commitments” to causes) shifted, but we have completed a substantial number of high-level cause investigations and decided on our working cause priorities. We are now shifting our focus from cause investigations to aiming for major grants and/or hires. Within the next six months, we hope to make at least one major grant and/or specialized hire in each of (a) U.S. policy and (b) global catastrophic risks; we consider this a “stretch goal” (substantial probability we will fail to hit it), and we plan another high-level check-in around 6 months from now.
  • We made less progress than hoped on other cause categories: scientific research funding and global health and development. We put substantial time into scientific research funding, but have found it to be probably the most complex and challenging of the broad categories. Our main goal for 2015 is to form clear priorities within scientific research funding, comparable to where we currently stand on U.S. policy and global catastrophic risks. This is a stretch goal. We don’t plan to work on global health and development this year for the Open Philanthropy Project.
  • We made some progress in separating the Open Philanthropy Project brand from the GiveWell brand, including a renaming (the Open Philanthropy Project was known as GiveWell Labs until last August). In the coming year, we plan to launch a more substantial website for the Open Philanthropy Project, continuing the process of separation.

U.S. policy and global catastrophic risks
We covered recent progress in these areas, and compared it to goals set last year, in previous posts (U.S. policy, global catastrophic risks). This section gives a summary of that progress; those who have read the previous two posts may wish to skip it.

Last year, we wrote:

…our top goal for 2014 is a stretch goal (substantial probability we will fail to hit it): making substantial commitments to causes within these two categories. We aren’t sure yet how many causes this will involve; it will depend partly on our ability to find suitable hires. We also haven’t fully formalized the notion of a “substantial commitment to cause X,” but it will likely involve having at least one staff member spending a substantial part of their time on cause X, planning to do so for multiple years, and being ready to commit $5-30 million per year in funding. Given this level of commitment, it is likely that we will not be able to commit to more than 1-3 causes for each broad category (“global catastrophic risks” and “US policy issues” are instances of “broad categories”) in the coming year. Sub-goals of this goal are:

* Completing enough shallow- and medium-depth investigations to feel that we’ve looked near-comprehensively at potential focus causes in these two categories, and writing up our reasons for narrowing the field to a smaller set of “contender causes.”
* Deeply investigating “contender causes” – possibly including some amount of preliminary grantmaking – and prioritizing these “contender causes” relative to each other (and discussing our reasons for such prioritization).
* Recruiting people to focus primarily or exclusively on finding giving opportunities within the causes we select.

We see this as an extremely challenging goal for the coming year, given our current status in these areas. There is no precision to estimating that one year is roughly sufficient, and the project of prioritizing causes in these categories could easily stretch into 2015. With that said, this prioritization is our top priority for 2014, and we think we have a chance to accomplish it. If we do so, we believe that GiveWell Labs [now Open Philanthropy Project] will become a much easier product to understand, discuss and critique, and we will reach the sort of crucial juncture for GiveWell Labs [now Open Philanthropy Project] that we reached for our traditional work around the end of 2009: having concrete recommendations that we can promote and defend, leading to much better engagement with and appeal to donors.

We have not delivered on this goal as initially envisioned. This is mostly because our thinking on how, and how much, to “commit” to causes has evolved. Rather than commit major time and funding up front to a small number of causes, we are going with a longer list of prioritized causes, and we’re looking for a good combination of “high-priority cause” with “strong specific giving and/or hiring opportunity.” The evolution of our thinking on this front is documented in several previous posts: Expert philanthropy vs. broad philanthropy, Thoughts on the Sandler Foundation, and our most recent update on our priorities for U.S. policy causes (similar reasoning applies to our work on global catastrophic risks).

With that said, we’ve done a large number of shallow- and medium-depth cause investigations, and as of the end of January 2015 (a little less than a month behind the schedule implied above), we were transferring the bulk of our energy from these sorts of investigations to seeking out hires and grants in the causes we’ve prioritized.

We haven’t yet made specialized hires, and we feel that the progress we had hoped for on making the Open Philanthropy Project “more concrete” has been partially but not fully realized.

Within the next six months, we hope to make at least one major grant and/or specialized hire in each of (a) U.S. policy and (b) global catastrophic risks; we consider this a “stretch goal” (substantial probability we will fail to hit it). We plan another high-level check-in around 6 months from now.

Scientific research funding and global health and development
Last year, we wrote:

We feel that we are at an earlier stage with two other broad categories of philanthropic causes: scientific research funding and foreign aid. In the case of scientific research funding, we have determined that scientific advisors are crucial, and we have recently recruited several such advisors and started working with them on a trial basis. In the case of foreign aid, despite our history of recommending charities that aid the developing world, we have not developed a strong understanding of how to evaluate a broad cause such as “malaria control” or infrastructure in sub-Saharan Africa from the perspective of flexible, large-scale philanthropy (as opposed to focusing in specifically on delivery of evidence-backed interventions).We hope that at the beginning of 2015, we will be able to say about these two areas what we currently say about global catastrophic risks and US policy: that we have a general sense of the landscape of causes and of how to investigate and evaluate causes, and can aim to make serious commitments to causes in these categories within a year. This is also an ambitious goal, especially in light of its being a secondary priority to the above goal.

We did not deliver on the above goal.

I put substantial time into working with a team of six junior scientific advisors on a couple of fronts:

  • Having general conversations about philanthropic opportunities around scientific research.
  • Investigating the hypothesis that diseases such as malaria and tuberculosis (which disproportionately affect the global poor) are underfunded, and therefore present strong philanthropic opportunities.

I found this work highly challenging. In many cases, I found that the scientists we most wanted to talk to were reluctant to speak at all and/or reluctant to be candid about strong funding opportunities. This contrasts with our experience investigating U.S. policy areas, where many of the people who are best positioned to answer our questions see educating the public as part of their job. In addition, we struggled to find the right way to approach the second investigation listed above: we weren’t able to find people who could give broad, cross-cutting summaries of what philanthropic opportunities might look like, and struggled to think of how to divide the question into smaller questions for optimal efficiency.

Looking at where we stand today, it seems to me that:

  • We are still struggling to identify the right division of labor between full-time employees and scientific advisors when investigating a question of interest. We recently transitioned from working with a team of six advisors on fairly broad questions to working more intensively with individual advisors on investigating opportunities around specific diseases. We expect our thinking on this point to evolve further.
  • I am well behind where I’d like to be in terms of having basic background knowledge about scientific research. Obviously, I will never be an expert in any particular scientific field, so I need to be thoughtful about what a realistic level of background knowledge is, but I would like to improve significantly more before stepping up our efforts to investigate specific causes within scientific research.
  • Between the above two points, we still aren’t yet at the point where we can be working on large numbers of shallow- and medium-depth investigations of scientific research causes.
  • All of the above pertains only to biomedical sciences. We haven’t even begun exploring other scientific fields, including social sciences, energy, and computer science and machine learning.

In light of this situation, we are planning to free up some capacity by postponing Open Philanthropy Project work on global health and development until we’ve made significantly more progress on exploring scientific research. We feel that GiveWell’s top charities represent outstanding giving opportunities for people whose main concern is global health and development – in fact, we are not highly confident that we will find better ones for this particular goal through Open Philanthropy Project. We see improving our understanding of scientific research as the more daunting and pressing goal.

Our main goal for 2015 is to form clear priorities within scientific research funding, comparable to where we currently stand on U.S. policy and global catastrophic risks. This is a stretch goal.

We’ll be writing more about the specifics of our findings and views on biomedical sciences in the coming weeks.

Other progress and plans for Open Philanthropy Project

  • We have recently been prioritizing investigation over public writeups, and our public content is running well behind our private investigations. We are experimenting with different processes for writing up completed investigations – in particular, trying to assign more of the work to more junior staff. If we could do this, it would make a major difference to our capacity, since senior staff already have a substantial challenge keeping up with all of our priority causes. By the end of 2015, we hope that our public content will be no further behind our private investigations than it is at the moment.
  • In 2014, we created the “Open Philanthropy Project” name – replacing GiveWell Labs – and created a preliminary website. In the coming year, we plan to launch a more substantial website for the Open Philanthropy Project, and to look further into the idea of creating a separate organization (which we see as highly likely to happen in the long run).
  • We continued work on our History of Philanthropy project; we put out the first public case study, completed two more that are not yet public, and funded a conference on History of Philanthropy at the Rockefeller Archive Center. We plan to continue work on case studies with our existing consultants, and take new opportunities that come up to make grants and/or bring on new consultants. However, we have no specific goals for the year on this front.
  • We are continuing to network with people at major foundations and people who may become major funders in the future. We do not have specific goals on this front for 2015.
  • We are in the process of considering two grants to major U.S. think tanks trying to build their presence in India and/or China. We feel that we will eventually find it important to be able to work in other countries, particularly countries as important to the future global economy as India and China. However, we expect doing so to be extremely challenging, due largely to the difficulty of forming good relationships and having background context. We see these grants as a small step that will better position us to learn, over time, about the challenges of working overseas. This is not a major priority for 2015.