The GiveWell Blog

Meta-research update

As mentioned previously, we are currently conducting an in-depth investigation of meta-research, with the hopes of producing our first “medium-depth” report on the giving opportunities in a cause.

Our investigation isn’t yet complete, but it has taken several turns that we’ve found educational, and our vision of what it means to investigate a “cause” has evolved. This post gives an update on how we’ve gone from “investigating meta-research in general, starting with development economics” to “specifically investigating the issue of reproducibility in medical research” to “investigating alternatives to the traditional journal system.”

The big-picture takeaway is that if one defines a “cause” the way we did previously – as “a particular set of problems, or opportunities, such that the people and organizations working on them are likely to interact with each other, and such that evaluating many of these people and organizations requires knowledge of overlapping subjects” – then it can be difficult to predict exactly what will turn out to be a “cause” and what won’t. We started by articulating a broad topic – a seeming disconnect between the incentives academics face and the incentives that would be in line with producing work of maximal benefit to society – and looking for people and organizations who do work related to this topic, but found that this topic breaks down into many sub-topics that are a better match for the concept of a “cause.”

Simply identifying which sub-topics can be approached as “causes” is non-trivial. We believe it is important to do so, if one wishes to deliberately focus in on the most promising causes that can be understood in a reasonable time frame, rather than spreading one’s investigative resources exploring several causes at once.

From development economics to medicine

In a previous meta-research update, we focused on the field of development economics. Following that update, we collaborated for several months with an institutional funder that supports a significant amount of development economics work and has expressed similar “meta-research” interests; we also explored some other fields, as discussed in a recent post. We ultimately came to the working conclusion that

  • Meta-research in medicine-related fields is “further along” than in social sciences, in the sense that there are more established organizations and infrastructure around meta-research (for example, Cochrane Collaboration and EQUATOR network) and there has been more research on related issues (particularly the work of John Ioannidis).
  • With that said, meta-research in medicine-related fields still has a long enough way to go – and little enough in the way of existing funders working on it – to make it a potentially promising area.
  • In social sciences, studies are often so expensive and lengthy to conduct (the deworming study we’ve discussed before took over a decade to produce what we consider its most relevant results) that the prospects for robustly establishing conclusions to inform policy generally seem distant. By contrast, we believe that improving the reliability of medical research would likely have fairly direct and quick impacts on medical practice.
  • The institutional funder we have collaborated with continues to work in social sciences (specifically development economics), and we believe its approach and attitude is similar enough to ours that our value-added in this area would be limited.

With these points in mind, we decided to shift our focus and deeply investigate meta-research in medicine-related fields rather than meta-research in development economics. This was a provisional decision; we remain interested in the latter.

Exploring meta-research in medicine

Alexander Berger led an investigation of meta-research in medicine, beginning in February. His basic approach was to start with the leads we had – contacts at Cochrane as well as individuals suggested by John Ioannidis – and get referrals from them to other people he should be speaking with.

In early May, we paused the investigation to take stock of where we were. It occurred to us that the people and organizations we had come across were divided into a few categories, which didn’t necessarily overlap:

1. The “efficiency and integrity of medical research” community. This community focuses on improving the efficiency with which medical research funding is translated into reliable, actionable evidence, by promoting practices such as (a) systematic reviews, which synthesize many studies to provide overall conclusions that can inform medical practitioners; (b) data sharing, especially of clinical trial data; (c) preregistration; and (d) replications of existing studies to check their reliability. This community includes the Cochrane Collaboration.

People in this community that we spoke to include:

2. The “open science” community. This community focuses on new tools for producing, sharing, reviewing, and evaluating research, many of them focusing on the idea of a transition from traditional paper journals to more powerful and flexible online applications. Some such tools (such as Open Science Framework) are produced by nonprofits, while others (such as ResearchGate and JournalLab) are produced by for-profits.

People in this community that we spoke to include:

Widespread adoption of tools such as those listed above could eventually make it much easier for researchers to share their data, check the reliability of each others’ work, and synthesize all existing research on a given question – in other words, such adoption could eventually lead to resolution of many of the same issues that the “efficiency of medical research” community deals with. Not surprisingly, many of the people in the “open science” community emphasize the same problems with today’s research world that people in the “efficiency of medical research” community emphasize – so it’s not surprising that, when we expressed interest in these issues, we were pointed to people in both categories.

That said, there is little overlap between communities #1 and #2, and we believe that this is largely for good reason. Community #1 focuses on medical research; community #2 is generally working across many fields at once. Community #1 focuses on actions that could directly and quickly improve the usability of medical research; community #2 is largely working on a longer time horizon, and hopes to see dramatic improvements when widespread adoption of its tools takes place. (Despite this, when there are organizations that have a disciplinary bent, we’ve continued to focus on the more bio-medically relevant ones, as opposed to those focused on, e.g. astronomy or geosciences.)

3. Other communities. Some other communities that could fall under the heading of “meta-research relevant to medical practice” include:

  • The evidence-based medicine community, which seeks to improve the usefulness of evidence for medical practice by increasing the extent to which available high-quality evidence is used in medical practice. (We see this community as distinct from the “efficiency and integrity of biomedical research” community because it focuses on the use, as opposed to the production, of evidence, though many of the practitioners overlap.)
  • People seeking to improve the practice of epidemiology (whose methods and issues are quite distinct from those of the sort of research that Cochrane Collaboration synthesizes). One such group is the Observational Medical Outcomes Partnership (OMOP), which we spoke with David Madigan about.
  • John Ioannidis, whose work seems largely unique as far as we can tell. Prof. Ioannidis has studied a wide variety of “meta-research” issues in a wide variety of fields, including reproducibility of clinical research, bias, reliability of genome-wide association studies, and conformity vs. creativity in biology research.
  • Vannevar, a group started by Dario Amodei (who is a GiveWell fan and personal friend), which aims to improve the infrastructure around fields such as basic biology (which is distinct from both epidemiology and the sort of medical research that the Cochrane Collaboration addresses) and machine learning. Unlike most of the groups discussed above, Vannevar is focused on improving the ability of academia to produce high-risk, revolutionary work, rather than on improving its ability to efficiently produce immediately actionable recommendations for medical practitioners and policymakers.

Many of the individuals working in these communities may have cross-cutting interests and play some role in multiple communities, but we see the communities as having discrete identities. The characterization above is not meant to be exhaustive or to eliminate the possibility of other groupings, but rather to convey our understanding of the relationships between various problems, interventions, and individuals.

The path from here

At this point, the community we feel we have covered the most thoroughly is #2, the “open science” community. This hasn’t been an entirely deliberate decision: we’ve spoken to the people we’ve been pointed to and the people they’ve pointed us to, and only after many conversations have we noticed the patterns and distinct communities discussed above.

Because it is important to us to complete a medium-depth writeup, we’re currently aiming to complete such a writeup on open science. We will add the other communities discussed above to our list of potential shallow investigations.

In this process, we’ve learned that it can take a fair amount of work and reflection just to determine what counts as a “cause” in the relevant way. We think such work and reflection is worthwhile. Rather than speaking to everyone who is somehow connected to a problem of interest, we seek to identify different causes, deliberately pick the ones we want to focus in on, and cover those thoroughly.


  • Samuel on June 6, 2013 at 3:21 pm said:

    Many of the links to the discussions don’t work.

  • Alexander on June 6, 2013 at 3:27 pm said:

    Samuel – thanks for catching that. Should be fixed now.

  • Jonah Sinick on June 7, 2013 at 1:58 pm said:

    I agree with your bottom line that medicine-related meta-research looks more promising than development economics meta-research, but this

    >In social sciences, studies are often so expensive and lengthy to conduct (the deworming study we’ve discussed before took over a decade to produce what we consider its most relevant results) that the prospects for robustly establishing conclusions to inform policy generally seem distant. By contrast, we believe that improving the reliability of medical research would likely have fairly direct and quick impacts on medical practice.

    is not at all clear to me, because of the (temporally) long path from basic research to the development of new medical interventions. If your preference for medicine-related meta-research hinges on views about different temporal lags, it would be nice to see your reasoning spelled out.

    Note: I formerly worked as a research analyst at GiveWell.

  • Holden on June 7, 2013 at 5:44 pm said:

    Jonah, it’s important to distinguish between basic biology research (which can take a very long time to become applied) and the sort of medical research (e.g., randomized controlled trials) reviewed by Cochrane, whose path to potential impact is much shorter. We had the latter in mind at the juncture discussed in the post, though we’ve since become interested in meta-research for more basic work as well.

  • Jonah Sinick on June 7, 2013 at 11:10 pm said:

    Ok, this makes sense — I had in mind basic research, because of GiveWell’s recent conversations about basic research.

  • Uri Katz on June 9, 2013 at 7:09 am said:

    I am sorry if I am entering this conversation in the middle (I did not find the answer to the following questions in other posts, but that could be my own fault), but the causal link between, for instance, digitizing academic journals and saving lives in a Africa requires justification in my opinion. Justification both theoretically and with empirical evidence to back it up.

    Specifically I wonder whether we really do not know how to help the poor, or is it that we don’t know how best to implement the solutions we already have. I tend to think the latter is true, otherwise the claim that there are millions living in poverty conditions that are preventable is not true at the moment. It might be preventable, once we figure out how to prevent it. This is specifically true of meta-research in medicine, we have the vaccines, the mosquito nets, and so forth. What we lack are the resources and effective methods of implementing this solutions.

    If your chosen end goal was reducing climate change I can see the benefit of meta-research, but not if your end goal is eliminating poverty.

    Where is my error?

  • Holden on June 10, 2013 at 2:26 pm said:

    Uri, our goal is to do the most good possible, broadly speaking. We believe that meta-research could lead to better technologies and information that could directly help the poor, and could also contribute to a faster pace of global development, which we believe would be broadly beneficial as well.

    It can simultaneously be true that (a) we have many technologies available today that could help the poor if they were better delivered; (b) coming up with still better technologies could benefit the poor more than focusing purely on delivery of today’s technologies.

  • Ian Turner on June 12, 2013 at 9:31 pm said:

    Uri, with respect to poverty, it’s worth noting that we actually don’t really know how to end it. There are certain health related interventions (such as bednets or deworming) which are known to have an effect, but the overall question of how to make development happen remains an unknown.

    Part of the problem is that, as Holden once noted, development is correlated with most everything good (such as education, health, employment, infrastructure, investment, trade, etc.), so it’s hard to tell which good things are the ones causing development and which ones are just along for the ride. Speaking for myself, I find the Tolstoy approach plausible — essentially, that successful economies have much in common that but failed ones often have their own individual reasons for not thriving (such as bad neighbors, too many / too few resources, war, illiteracy, bad governance, climate factors, communicable disease, etc.).

  • Uri Katz on June 13, 2013 at 1:02 am said:

    Holden – Having read the post on flow-through effects I must reiterate my initial point about justification. The post you referred me to includes a very vague, and very general, argument about the history of the last 200 years in the Western world, and its possible reasons. I question whether effective interventions can result from such loose argumentation. What is needed to justify meta-research is:

    1) Empirical evidence of the specific link between the research, or meta-research, and the effects, including some of the flow-through effects, it will have.

    However, since I understand and appreciate the fact that you may want to be pioneers when it comes to meta-research, rather than use tried and tested methods, this can suffice:

    2) A highly plausible theoretical link, drawn out in as much detail as possible, between the meta-research and its effects, including potential flow-through effects.

    I do not think that is an impossible requirement to meet in general, but in some cases it might be. In those cases what is at the very least needed is:

    3) A detailed list of the possible obstacles in the link between the meta-research and its effects, including flow-through effects.

    Ian – I wonder if what you say is true with regard to medicine. After all, most, if not all, the epidemics that plague the poorest countries on earth are mostly eradicated from the richest countries. I agree that economic development might be a tougher nut to crack, but this post is about medicine.
    There are, of course, wider health issues such as nutrition and sanitation, that are closely connected to economic development.

  • Ian Turner on June 13, 2013 at 1:39 am said:

    Uri — I agree with respect to medicine; I was writing in response to your remark that “Specifically I wonder whether we really do not know how to help the poor, or is it that we don’t know how best to implement the solutions we already have. I tend to think the latter is true, otherwise the claim that there are millions living in poverty conditions that are preventable is not true at the moment.”, which seemed to be speaking to development generally.

  • Holden on June 13, 2013 at 2:09 pm said:

    Uri – thanks for your thoughts on what you would find compelling as an argument for meta-research as a promising cause. We could provide 2) and 3) and will do so if and when we issue an official recommendation in the field. But we don’t feel it is the most efficient use of our resources to provide more elaboration at this time, given that what we’re primarily doing is exploring the field (largely as a way to get practice with the general idea of exploring a field). More feedback such as yours could change our priorities on this point.

  • Chris on August 7, 2013 at 3:52 am said:

    How do you define “medicine”? I would have counted deworming as medical.

  • Holden on August 8, 2013 at 7:28 pm said:

    Chris, we generally think of “medical research” as being in a lab or clinical setting rather than field research. Field research, including the deworming study, would generally be thought of as “epidemiology” or “public health” though it can also (as with the deworming study) be associated with development economics.

  • Chris on August 8, 2013 at 9:10 pm said:

    Holden, how would you classify the Province of Ontario’s flu-shot program? On the one hand, it’s intended to prevent a clearly-defined disease and control its spread; on the other hand, the Province’s ad campaigns lead me to believe most of the public benefit is in reducing missed work days.

  • Holden on August 8, 2013 at 9:16 pm said:

    Chris, that sounds like a public health program, not research. A study on the program’s effect on missing work days would probably best fit under epidemiology or social science.

Comments are closed.