The GiveWell Blog

Science policy and infrastructure

We’ve tried to approach scientific research funding – focusing initially on life sciences – by looking for gaps and deficiencies in the current system for supporting scientific research. We’ve identified several possibilities, including a set of systematic issues that make it difficult to support attempts at breakthrough fundamental science.

One way to respond to a gap in the system would be to fill it ourselves: support the kind of science that has trouble getting support from existing funding agencies, universities, etc. We believe this is the approach taken by organizations such as Howard Hughes Medical Institute. But another way to respond would be to try to improve the system directly, by funding the development of – and advocacy for – proposals for structural changes. Structural changes could include changes in how government agencies allocate funding, in how universities determine professorships, or in other practices that we believe are important in influencing what scientists are able to do. (We broadly refer to universities, journals, and other institutions that play an important role in scientists’ incentives and support as “infrastructure.”)

We find the latter idea intriguing. It appears to us that the strongest scientific funders have little interest in policy analysis and advocacy, while the strongest funders of policy analysis and advocacy tend not to take interest in the scientific research issues discussed in this post. We’re interested in the idea of combining – in a dedicated organization – great scientists and great policy analysts, in order to put in the substantial amount of work needed to develop and promote the best possible proposals for improving science policy and infrastructure. It would be a high-risk, potentially very high-return project to attempt. We aren’t aware of any attempts to do something along these lines at the moment, and we think it could be a risk worth taking.

The rest of this post outlines:

  • Examples of science policy and infrastructure issues we’d like to see more work on. More
  • A brief sketch of how an organization dedicated to these issues might operate. More
  • What we know about existing attempts to improve science policy and infrastructure, and why we believe a new organization (or a dedicated team within an existing organization) could be a significant addition. More
  • Why we believe that supporting such an organization would be worthwhile. More

Examples of science policy and infrastructure issues

We previously wrote about claims that the current life sciences system has trouble supporting attempts at breakthrough fundamental science, and we featured a PNAS paper on the subject. This paper gives multiple concrete suggestions for how changes in U.S. policy might reduce competitiveness between scientists, improving prospects for early-career scientists, and supporting higher-risk, higher-reward research:

  • Making the government budget for scientific funding more “predictable and stable,” in order to facilitate long-term planning and avoid the sorts of supply-demand imbalances described previously.
  • Making changes in what sorts of grants can be used for what sorts of expenses (in particular, putting restrictions on the ability to support graduate students and postdocs using research grant funds), in order to allow more deliberate control of the number of graduate students and postdocs who will end up competing for professorships.
  • Aiming to broaden the possible career paths for young scientists, including increasing the use of “staff scientists” rather than trainees to support lab research. These changes could further diminish the intensity of competition for professorships as well as improving the overall productivity of labs.
  • Increasing the size of grant programs such as the NIH Director’s New Innovator award, which may be more conducive to supporting attempts at breakthrough fundamental science.
  • Improving the quality of grant application evaluation by revising criteria and scoring methodologies, and making more effort to include top scientists in evaluation.

These ideas are, by and large, fairly concrete and (to my eyes) practical-seeming suggestions for policy change. I haven’t been able to find information on the extent to which they are being implemented or actively discussed (other than that the number of Pioneer Award recipients seems to have shrunk rather than grown from last year to this year). To my knowledge, none have been substantially implemented.

In addition to these sorts of ideas, I think the following could also be highly worthwhile:

Thinking through how universities could experiment with new models for determining professorships, as well as how journals could experiment with new processes for highlighting noteworthy science. Both processes are extremely important factors in what kind of work is supported and incentivized in academia. Universities and journals tend to follow certain common cultural norms today, but given the degree of apparent agreement about room for improvement in the current system, it’s plausible to me that a dedicated effort at developing and promoting new approaches could spur experimentation and change.

Examining existing regulations – regulations on research, regulations regarding sharing of data, etc. – from the perspective of optimizing the ability to gain new knowledge and reap the benefits of innovation. Both the paper linked above and the paper I previously discussed on declining pharmaceutical productivity have identified increasing regulatory burdens as a major issue. In addition, from my limited readings on the history of biomedical research, it seems to me that getting new medical technologies tested and approved used to be much easier than it is today, and that many key experiments were highly speculative and dangerous. Such experiments would have been much more difficult to carry out with today’s regulation and social norms. Work in this category could include the following (these ideas are fairly speculative and may overlap to some degree with work being done at existing institutions):

  • Improving the balance between patients’ privacy and scientists’ ability to access large amounts of data for research purposes.
  • Improving the FDA process with an emphasis on increasing scientists’ ability to experiment and innovate, especially if and when new tools for data sharing present new possibilities for demonstrating safety and efficacy of medical technologies.
  • Improving the balance between ethical considerations and scientists’ ability to run informative experiments without excessive overhead.
  • Bringing a science- and scientist-focused perspective to debates over intellectual property law.
  • Regulating data sharing practices in clinical trials with an eye to enabling “reverse translation” research.
  • Working on optimal regulation of emerging technologies, in a framework that emphasizes the importance of innovation’s benefits as much as the importance of caution.

An opportunity for impact?

It seems to me that there could be a great deal of value in an organization dedicated to bringing together great scientists and great policy analysts, in order to develop and promote the best possible proposals for improving science policy and infrastructure. I would see such work as primarily aiming to have influence on universities, journals, and government agencies via developing well-thought-through ideas and making the case for them on the merits, rather than via exerting political pressure based on grassroots mobilization, media, etc. This strategy of aiming for impact would be comparable to that of organizations such as Center for Global Development and Center on Budget and Policy Priorities. (Note that claims of impact are available for both CGD and CBPP; we have not vetted either list but find both lists quite plausible.)

I think this kind of activity could be quite influential, and that the difference between a dedicated effort to carry it out and the status quo could be substantial. This would be consistent with my understanding of many past cases of nonprofits influencing policy, as well as with my understanding of how both corporate and nonprofit actors often have influence.

I discussed this idea at some length in my conversation with Neal Lane.

Does such an organization already exist?

My impression is that there are no organizations playing the role described above. The policy issues I’ve laid out have been raised by scientists via op-eds (such as the PNAS paper discussed above) and committee reports (such as a recent piece released by the American Academy of Arts and Sciences), but are not the focus of any dedicated organizations (or teams within organizations).

It’s possible that such an organization or team does exist – I haven’t searched exhaustively – but I have several reasons to believe it does not:

  • Most importantly, I have discussed this topic with many people, including those mentioned in our previous post. I have generally asked explicitly whether the kind of organization I’m envisioning already exists, and sought referrals to others who are knowledgeable on the subject. None of the people I’ve asked have been aware of such an organization – so even if one does exist, it seems unlikely that it has achieved much prominence. Neal Lane seemed particularly interested in these issues, and stated that he is not aware of such an organization (see public conversation notes).
  • I’ve searched the web for groups focused on science policy. At the moment, my impression is that “science policy” (as commonly used) tends to refer to some combination of (a) promoting a high level of funding for science; (b) working on policy around science education and outreach; (c) working on a wide range of policy issues, such as climate change mitigation, in a way that is informed by science (and/or emphasizes the importance of scientific knowledge and evidence in decision-making). I’ve examined Wikipedia’s category page on science advocacy organizations, and these organizations seem to generally be in one of the aforementioned categories. None of them seem to be focused on the sorts of issues I’ve discussed in this post. Note that Wikipedia’s list excludes Research!America, an organization that focuses on making the case for a high level of government support for science.
  • I’ve examined a list of think tanks by category, and none of those listed under “Science and technology” appear to do significant work on the sorts of issues discussed in this post.

Speaking generally from conversations I’ve had with major funders, it appears that the strongest scientific funders have little interest in policy analysis and advocacy, while the strongest funders of policy analysis and advocacy tend not to take interest in the scientific research issues discussed in this post.

Could work along these lines be worthwhile?

In conversations about this idea so far, I’ve encountered a mix of enthusiasm and skepticism. (I’ve also generally heard from science funders that it would be outside of their model, regardless of merits, because of the focus on influencing policy rather than directly supporting research.) Most of the skepticism has been along the lines of, “The current system’s cultural norms and practices are too deeply entrenched; it’s futile to try to change them, and better to support the best research directly.”

This may turn out to be true, but I’m not convinced:

So far, we haven’t been able to find a person or organization who seems both qualified and willing to lead the creation of the sort of organization described in this post. We plan to continue looking for such a person or organization, while continuing to discuss, refine and reflect on these ideas.

Comments

  • Luke Muehlhauser on April 21, 2015 at 3:45 pm said:

    Under “Examples of science policy and infrastructure issues,” you don’t list:

    * Improving incentives for study pre-registration, replication attempts, and publishing negative results
    * Creating incentives for better peer review
    * Experimenting with alternatives to journals, ala GitWikXiv (and many other proposals)

    Do you have any comments on these proposals for infrastructure change?

  • Have you spoken to the Wellcome Trust about this? Might be up their alley (and they’re the #2 foundation after Gates).

  • It’s worth mentioning that the National Academies and AAAS also work on things like the AAA&S committee report you link, with some broad “science policy” committees and some more focused workshops. I think there’s still plenty of room for a dedicated organization of policy analysts and scientists working on these problems.

  • Jesse Spaulding on April 21, 2015 at 7:24 pm said:

    It seems to me a think tank dedicated to improving science policy and infrastructure is a great idea!

    However I agree with Luke (above) — your post seems to be missing many of the big systemic problems in scientific research today. For example:

    * Most research is published with substantial delays and is locked behind paywalls
    * Data, software, and detailed methods are not shared
    * Negative results are not shared
    * The research process is distorted by the need to generate interesting results
    * Time and money are wasted on needless repetition of experiments
    * Scientists work in silos and rarely help each other outside of formal collaborations
    * Critical research design flaws are often not caught
    * Statistical mistakes are widespread
    * Peer review is slow, inefficient, and ineffective
    * Much of research is not reproducible

    If you can help fix these problems you won’t just be making the NIH 1% more efficient you’ll be making it 10X more efficient! I’ve written more about this here: http://thinklab.com/blog/10-consequences-of-a-broken-scientific-reward-system/36

    The bottom line is the advent of the Internet has created the possibility for a highly collaborative, open, online model of research that would dramatically accelerate scientific progress. For more on this I highly recommend Michael Nielsens book: “Reinventing Discovery” http://www.amazon.com/Reinventing-Discovery-The-Networked-Science/dp/0691160198

    Why aren’t things evolving to a better model? Well, right now science is stuck with a reward system from the 17th-century. Science needs incentives updated for the Internet era in which we live! And I think you’re right, most science funders aren’t interested in dealing with this issue.

    So in my opinion you should create a think tank dedicated to helping science funders understand how they can distribute their money in a way that creates better incentives. Of course — I’m biased — I’ve created a startup that is also dedicated to this. Not really advising — but actually helping them create those better incentives. http://thinklab.com/

  • Tom Stocker on April 23, 2015 at 9:25 am said:

    Couple of initiatives in the UK might be worth liaising with, although they’re not massive.
    http://www.betterdata.org.uk/
    http://www.alltrials.net/
    http://www.senseaboutscience.org/pages/our-work.html

    And Ian Chalmers, Muir Grey, Ben Goldacre, Gird Gigerenzer would all be keen and have good ideas if on health related issues.

  • Holden on April 23, 2015 at 12:54 pm said:

    Thanks for the comments, all!

    I’d say the issues highlighted by Luke and Jesse relate to two related topics we’ve written about before: reproducibility and open science. Past posts on these topics can be found here and here.

    Comments on these as they pertain to this post:

    • Reproducibility hasn’t been as salient (relative to the issues discussed above) in my conversations about life sciences specifically. As I argued at the very end of this post, I see it as an issue primarily for clinical trials, within biomedical research (I also think it is a major issue for social sciences). For other kinds of biomedical research, I think the costs of non-reproducibility are likely less severe, and I think common solutions such as preregistration and data sharing are likely to work very differently (or not to be key) because of the differences between lab work and human trials. I also believe that there is substantially more dedicated effort going on for these issues than for the others I named in the post (though I’m less confident regarding some of the regulation-related ideas). (The Laura and John Arnold Foundation funds work on reproducibility, including the Meta-Research Innovation Center at Stanford.) With all of that said, I should have included this issue in the post, and I would want to see this issue on the possible agenda for the kind of organization outlined here, though it isn’t one of the core issues motivating the vision for such an organization.
    • Open science is clearly relevant here – a major part of the issues I’ve laid out might be attributable to the journal system, and alternative systems could hold promise. We’ve previously laid out a landscape of the open science community. I perceive open science as having more full-time, funded proponents than the issues I’ve discussed in this post, and I don’t feel it would be a good fit for the type of think tank I’m envisioning. I see many of the challenges of open science as being related to designing alternatives that people would want to use, which doesn’t seem like the sort of issue that would ideally be handled by top scientists and policy analysts. I think it makes sense to tackle the challenge of improving the journal system from both sides: decentralized efforts to build new tools and platforms that scientists might take up, and advocacy-focused efforts to rethink how the existing institutions work.
  • Jesse Spaulding on April 23, 2015 at 2:07 pm said:

    Hi Holden,

    In my opinion the primary challenge with open science is that the incentives in the system do not support it. The incentive is to work in silos, hoard knowledge, and only share something when you can publish a paper. And when you do publish the incentive is to go for a journal that is as high-impact as possible — which tend to be the journals that hide your research behind paywalls.

    The thing is, there are really simple policy solutions that would have a huge positive impact here. For example, the US government could just decide that all publicly funded research must be published open access immediately upon completion. This is something the Gates Foundation is doing: http://www.gatesfoundation.org/how-we-work/general-information/open-access-policy

    Of course, it’s hard for the government to implement changes when doing so would seriously damage a $10 billion dollar journal industry. But this is what needs to be done.

    Maybe your proposed organization can help push for this!

    Jesse

  • Francis on April 23, 2015 at 4:55 pm said:

    In the UK, the Royal Society https://royalsociety.org does ‘policy for science’ (e.g. open data, diversity in science, science education) as well as ‘science for policy’ (climate change, fracking) with the backing of scientists.

    You might also want to reach out to the Open Science Working Group of the Open Knowledge Foundation http://science.okfn.org Peter Murray-Rust knows everyone and everything.

  • Holden on April 29, 2015 at 10:13 pm said:

    Jesse, it might be helpful to distinguish between “open access” and “open science.” The former refers specifically to the issue with paywalls. This is an issue where I agree that the right policy could essentially solve the problem. I perceive it as a relatively minor problem. By contrast, “open science” refers more broadly to finding alternatives to the journal system that are superior in many respects, not just regarding whether access is free. I see open science as a more important and more challenging issue, and my previous comment about decentralized vs. advocacy-oriented solutions was referring to open science.

Comments are closed.