Via the grantmaking process described previously, Good Ventures has decided – with GiveWell’s input – to make a grant to the Center for Global Development (CGD) for general operating support. The grant will be $300,000 paid evenly over the next three years. This post lays out the thinking behind this grant. As mentioned previously, this grant is distinct from our charity recommendations in terms of the primary justification.

The observation that led to this grant – and underlies much of the reasoning behind it – is that we (GiveWell and Good Ventures) are relying on CGD substantially for help with our learning agenda. This has implications for all of the principles we previously laid out for making grants:

• A grant to CGD is likely to have “learning value” via increasing our access to CGD.
• Because we’re directly engaging with CGD’s “product,” we feel relatively well positioned to evaluate the quality of that “product” and by extension the quality of CGD. Though our view of CGD is far from exhaustive, what we have seen of the organization is quite positive, which implies to us that general operating support is a good giving opportunity.
• Because CGD’s work is important and valuable to us and because CGD relies on (and seeks) philanthropic funding, we believe that supporting CGD falls under the heading of “good citizenship” discussed previously.

In general, we’re planning to frequently consider grants to organizations whose work is highly valuable to our research, because such situations tend to be associated with the above points: they tend to be situations in which we value access, in which we are reasonably positioned to have a favorable view of at least a part of the organization’s work, and in which “good citizenship” principles call for providing support.

Our experience as “customers” of CGDWe have benefited from CGD’s work for several years, dating back to when we were focused exclusively on direct aid.

• Millions Saved has been a valuable resource for us in identifying interventions that have worked at scale. It’s the only work we’ve seen that has collected success stories like these into one place, and we’ve vetted the report and found it to be of reasonably good quality. We are currently engaged in an update of the report and have found this project to be one of the most promising “shovel-ready” opportunities within the category of history of philanthropy.
• We have been extremely impressed with – and helped by – David Roodman, one of CGD’s resident scholars. We have made use of his critiques of nonexperimental studies of microfinance, review of the problems with literature on the macro effects of aid, and review of higher-quality evidence regarding microfinance. He’s the person we know of who has made the biggest contributions to examining the validity and reliability of research relevant to foreign aid (something we have looked for a great deal). In addition, I found his book Due Diligence to be the best discussion of microfinance I’ve seen, and to be generally a model of analysis that is simultaneously thoughtful and careful, holistic (looking at many different angles of a problem), and transparent (in the sense that it is always clear what his claims are based on).
• More recently, we have had conversations with Todd Moss and Michael Clemens (both CGD staff) as part of our shallow investigations of new causes. We spoke to Dr. Clemens because we had been repeatedly pointed to him as a leading scholar and advocate on the topic of international migration (and his work in reviewing the literature was a major contributor to sparking our interest in this cause as a potentially high-impact one in the first place). We spoke to Dr. Moss because we had run across his work in our review of discussions of developing-world infrastructure (writeup forthcoming). In both cases, we found the conversations helpful and informative, though we have not independently investigated the accuracy of the statements made in these conversations.
• As we continue to go down our list of potential shallow investigations, there are many more for which we anticipate that speaking to a CGD staffer will be a good starting point. This is because of CGD’s relatively unique standing as an organization that examines both the intellectual and practical/political aspects of designing policy to help the global poor.
• As a more minor point, we found our conversation with Lant Pritchett (also a CGD staffer) to be one of the more interesting open-ended conversations we’ve had on how a funder can accomplish as much good as possible. Dr. Pritchett had concrete ideas for areas that could plausibly be high-impact and do not seem to be already “crowded” with funders.

More on CGD as an organizationCGD describes itself as conducting “research and analysis on a wide range of topics related to how rich country policies impact people in the developing world.” Looking across CGD’s topics, initiatives, and experts, the consistent picture is of an organization doing analysis on practical policy ideas aimed at improving conditions for the world’s poorest. We know of few other organizations with similar missions.

We have not done an exhaustive review of CGD’s activities and the case for each, and in particular we know little about CGD’s influence. (We have also not done room for more funding analysis, though we do know that CGD is soliciting general operating support.) But the fact that the people and work we’ve seen so far are generally high-quality – and that CGD’s activities are both broad and consistently aimed at a population that we think is particularly appropriate as a target of philanthropic efforts – point to CGD as an organization that stands out on both mission and staff, and therefore can potentially do substantial good with general operating support.

One more point influencing our overall impression of CGD is its data disclosure policy, announced in 2011. We have previously written about our interest in data/code sharing, and we know of no other research organization (focused on social sciences) with a similar policy, even two years later.

Grant size and structureWe haven’t done an in-depth investigation of CGD, and since much of our goal is to fulfill the goals of “access” and “good citizenship” discussed previously, we have tried to settle on a grant large enough to show seriousness of support and to ensure that CGD will consider interacting with us to be worth its time. Via informal conversations with other funders and organizations, we have arrived at the figure of $100,000 per year as a reasonable figure to accomplish this for a fairly large (~$10 million per year) and established organization such as CGD.

The grant is a three-year grant ($100,000 each year for the next three years). As we will be discussing in a future post, we believe that providing multiple-year commitments is helpful for other organizations’ planning, and that a three-year grant is therefore substantially better for the grantee better than a one-year grant renewed twice.

[Added August 27, 2014: GiveWell Labs is now known as the Open Philanthropy Project.]

As stated previously, we expect that it will take quite a long time for us to reach the point of issuing major recommendations based on our GiveWell Labs work. That said, there have been – and will be – situations in which making a grant is appropriate and helpful. Since we are working closely with Good Ventures on Labs, our default approach has been – and will be – to jointly assess situations in which a grant may be called for, with the final call (and any grant) being made by Good Ventures. (If we encounter a point of disagreement, in which we feel it is important to make a grant and Good Ventures does not, we may approach other donors.) This post lays out the basic principles by which we (GiveWell and Good Ventures) decide when to make a grant.

Note that these grants are importantly different from our official recommendations. There is much less emphasis on thorough investigation and maximizing good accomplished per dollar (though the latter is a consideration), and much more weight placed on practical value to our agenda (particularly learning opportunities).

1. Giving to learn

We’ve written before about the concept of “giving to learn,” stating that “gaining information from an organization … is much easier to obtain as a ‘supporter’ (someone who has helped get funding to an organization in the past) than simply as an evaluator (someone who might help get funding to an organization in the future).”

To elaborate a bit on this idea, there are multiple forms that “giving to learn” can take:

• A grant can improve our access to an organization that we want to learn more about, or an organization whose personnel are good sources of information. The work we’ve done on co-funding generally goes in this category.
• A grant may directly pay for work that generates useful information, or may help us influence the direction that such work takes. Potential examples include any grants from our history of philanthropy project, including the recent $50,000 grant to the Millions Saved project.
• In some cases a grant can be viewed as an “experiment” – a way to test a theory that a particular project will have a particular result, or will more generally be a worthwhile investment.In general, we believe that “betting on one’s beliefs, and seeing what happens” is a good way to learn about the world, though we also think that this approach has major and unusual limitations when it comes to philanthropy. In our experience, understanding the outcomes/results of a given philanthropic project is usually a major undertaking, and it’s easy to learn nothing from a grant if one does not commit to such an undertaking. Therefore, we try to pick “learning grants” of this type carefully. The giving that fits best into this category so far is the money we’ve moved to our top charities, which we believe to be excellent giving opportunities that we can follow and adjust our views of over time.

2. Strong giving opportunities

Because we believe that good accomplished compounds over time, we want to take advantage of unusually strong giving opportunities when we come across them. Doing so will sometimes have the added benefit of providing further “experiments” to learn from in line with the previous section.

We believe that it is usually difficult to assess the quality of a giving opportunity without having strong cause-level knowledge. As such, we expect to make fairly few grants in this category in the near future, though as we expand the set of causes we understand well, we expect to make more over time.

3. Good citizenship

We are just getting started in exploring many relevant areas; our reputation and relationships are important. Therefore, we think it is important to generally behave as “good citizens” when it comes to grantmaking. The idea of being a “good citizen” is a vague one that we’re still fleshing out, but it includes things like

• Being direct and open with potential funding partners and grantees, and not withholding information for the sake of saving money.
• Not behaving in ways that “reward” potential funding partners/grantees for being less than direct and open with us, or “punish” potential funding partners/grantees for being direct and open with us.

Imagine that both we and another funder are considering making the same grant, and we have the feeling that the other funder might make the grant if we did not. In such a case, we could hold back and disguise our interest for the purpose of saving money, but we feel such an action would fail the “good citizenship” test. Rather, we intend to err on the side of making grants that we would have been willing to make under slightly different circumstances (concerning funding partners’ and potential grantees’ plans and preferences). If we value an organization’s help enough that we would be willing to make a “learning grant” to gain better access to it, we will err on the side of making such a grant even if we happen to believe that we could gain such access without a grant. If we are interested enough in a project that we would be willing to fund it if a potential partner weren’t, we will err on the side of contributing to funding even if we feel that the potential partner doesn’t need our help.

Weighing factors and making decisions

We plan to make grants when some combination of the above factors calls for doing so.

For any given grant, we will need to determine the appropriate level of investigation, as well as the appropriate level of followup and public discussion. In all cases, we will announce grants and give at least a basic characterization of the thinking behind them. But we also will be trying to make the level of investigation, followup and public discussion conceptually “proportional” to the size of the grant. The $50,000 grant to Millions Saved is simply too small – in the scope of the amount of funding we hope eventually to direct – to justify the sort of intensive investigation and followup we’ve done of our top charities. On the flip side, if we were contemplating a very large grant (in the millions of dollars), we would generally plan on serious investigation, and accordingly we would have a much higher bar that the grant would have to clear regarding the above criteria. We wouldn’t undertake a major investigation and major grant unless we felt an opportunity was highly outstanding (and/or in line with our learning agenda).

Over the coming months, GiveWell and Good Ventures expect to announce a reasonable number of grants. Such grants will not always be accompanied by exhaustive research or explicit cost-effectiveness analysis, but they will be carefully selected to fulfill the above criteria and further our mission of finding and funding the most outstanding giving opportunities possible.

As mentioned previously, we are currently conducting an in-depth investigation of meta-research, with the hopes of producing our first “medium-depth” report on the giving opportunities in a cause.

Our investigation isn’t yet complete, but it has taken several turns that we’ve found educational, and our vision of what it means to investigate a “cause” has evolved. This post gives an update on how we’ve gone from “investigating meta-research in general, starting with development economics” to “specifically investigating the issue of reproducibility in medical research” to “investigating alternatives to the traditional journal system.”

The big-picture takeaway is that if one defines a “cause” the way we did previously – as “a particular set of problems, or opportunities, such that the people and organizations working on them are likely to interact with each other, and such that evaluating many of these people and organizations requires knowledge of overlapping subjects” – then it can be difficult to predict exactly what will turn out to be a “cause” and what won’t. We started by articulating a broad topic – a seeming disconnect between the incentives academics face and the incentives that would be in line with producing work of maximal benefit to society – and looking for people and organizations who do work related to this topic, but found that this topic breaks down into many sub-topics that are a better match for the concept of a “cause.”

Simply identifying which sub-topics can be approached as “causes” is non-trivial. We believe it is important to do so, if one wishes to deliberately focus in on the most promising causes that can be understood in a reasonable time frame, rather than spreading one’s investigative resources exploring several causes at once.

From development economics to medicine

In a previous meta-research update, we focused on the field of development economics. Following that update, we collaborated for several months with an institutional funder that supports a significant amount of development economics work and has expressed similar “meta-research” interests; we also explored some other fields, as discussed in a recent post. We ultimately came to the working conclusion that

• Meta-research in medicine-related fields is “further along” than in social sciences, in the sense that there are more established organizations and infrastructure around meta-research (for example, Cochrane Collaboration and EQUATOR network) and there has been more research on related issues (particularly the work of John Ioannidis).
• With that said, meta-research in medicine-related fields still has a long enough way to go – and little enough in the way of existing funders working on it – to make it a potentially promising area.
• In social sciences, studies are often so expensive and lengthy to conduct (the deworming study we’ve discussed before took over a decade to produce what we consider its most relevant results) that the prospects for robustly establishing conclusions to inform policy generally seem distant. By contrast, we believe that improving the reliability of medical research would likely have fairly direct and quick impacts on medical practice.
• The institutional funder we have collaborated with continues to work in social sciences (specifically development economics), and we believe its approach and attitude is similar enough to ours that our value-added in this area would be limited.

With these points in mind, we decided to shift our focus and deeply investigate meta-research in medicine-related fields rather than meta-research in development economics. This was a provisional decision; we remain interested in the latter.

Exploring meta-research in medicine

Alexander Berger led an investigation of meta-research in medicine, beginning in February. His basic approach was to start with the leads we had – contacts at Cochrane as well as individuals suggested by John Ioannidis – and get referrals from them to other people he should be speaking with.

In early May, we paused the investigation to take stock of where we were. It occurred to us that the people and organizations we had come across were divided into a few categories, which didn’t necessarily overlap:

1. The “efficiency and integrity of medical research” community. This community focuses on improving the efficiency with which medical research funding is translated into reliable, actionable evidence, by promoting practices such as (a) systematic reviews, which synthesize many studies to provide overall conclusions that can inform medical practitioners; (b) data sharing, especially of clinical trial data; (c) preregistration; and (d) replications of existing studies to check their reliability. This community includes the Cochrane Collaboration.

People in this community that we spoke to include:

• Many individuals affiliated with the Cochrane Collaboration (see here).
• Steven Goodman, Associate Dean of Clinical and Translational Medicine at Stanford University.
• John Ioannidis, Professor of Medicine at Stanford University.
• Doug Altman, chair, and David Moher, member, of the Steering Group of the EQUATOR Network, which promotes transparent and accurate reporting of medical research studies.
• Tom Kenny, Director of External Relations, the UK’s National Institute for Health Research Evaluation, Trials and Studies Coordinating Centre.
• Ivan Oransky, co-founder of Retraction Watch.
• Elizabeth Iorns, founder of Science Exchange and the Reproducibility Initiative.
• Ken Witwer, Assistant Professor at Johns Hopkins School of Medicine.
• Ferric Fang, Professor of Laboratory Medicine and Microbiology at the University of Washington.
• Stanley Young, Assistant Director of Bioinformatics, National Institute of
  Statistical Science.

2. The “open science” community. This community focuses on new tools for producing, sharing, reviewing, and evaluating research, many of them focusing on the idea of a transition from traditional paper journals to more powerful and flexible online applications. Some such tools (such as Open Science Framework) are produced by nonprofits, while others (such as ResearchGate and JournalLab) are produced by for-profits.

People in this community that we spoke to include:

• Joshua Greenberg, program officer for the Sloan Foundation.
• Jason Priem, co-founder of alternative metrics provider ImpactStory.
• David Jay, co-founder of JournalLab.
• Jane Hunter, Managing Director of Facutly of 1,000, and Rebecca Lawrence, Publisher of F1000Research.
• John Wilbanks, Chief Commons Officer, Sage Bionetworks.
• Cameron Neylon, Advocacy Director, Public Library of Science (PLoS).
• William Gunn, Head of Academic Outreach, Mendeley.
• Kaitlin Thaney, Manager, External Relationships, Digital Science.
• Heather Joseph, Executive Director of SPARC

Widespread adoption of tools such as those listed above could eventually make it much easier for researchers to share their data, check the reliability of each others’ work, and synthesize all existing research on a given question – in other words, such adoption could eventually lead to resolution of many of the same issues that the “efficiency of medical research” community deals with. Not surprisingly, many of the people in the “open science” community emphasize the same problems with today’s research world that people in the “efficiency of medical research” community emphasize – so it’s not surprising that, when we expressed interest in these issues, we were pointed to people in both categories.

That said, there is little overlap between communities #1 and #2, and we believe that this is largely for good reason. Community #1 focuses on medical research; community #2 is generally working across many fields at once. Community #1 focuses on actions that could directly and quickly improve the usability of medical research; community #2 is largely working on a longer time horizon, and hopes to see dramatic improvements when widespread adoption of its tools takes place. (Despite this, when there are organizations that have a disciplinary bent, we’ve continued to focus on the more bio-medically relevant ones, as opposed to those focused on, e.g. astronomy or geosciences.)

3. Other communities. Some other communities that could fall under the heading of “meta-research relevant to medical practice” include:

• The evidence-based medicine community, which seeks to improve the usefulness of evidence for medical practice by increasing the extent to which available high-quality evidence is used in medical practice. (We see this community as distinct from the “efficiency and integrity of biomedical research” community because it focuses on the use, as opposed to the production, of evidence, though many of the practitioners overlap.)
• People seeking to improve the practice of epidemiology (whose methods and issues are quite distinct from those of the sort of research that Cochrane Collaboration synthesizes). One such group is the Observational Medical Outcomes Partnership (OMOP), which we spoke with David Madigan about.
• John Ioannidis, whose work seems largely unique as far as we can tell. Prof. Ioannidis has studied a wide variety of “meta-research” issues in a wide variety of fields, including reproducibility of clinical research, bias, reliability of genome-wide association studies, and conformity vs. creativity in biology research.
• Vannevar, a group started by Dario Amodei (who is a GiveWell fan and personal friend), which aims to improve the infrastructure around fields such as basic biology (which is distinct from both epidemiology and the sort of medical research that the Cochrane Collaboration addresses) and machine learning. Unlike most of the groups discussed above, Vannevar is focused on improving the ability of academia to produce high-risk, revolutionary work, rather than on improving its ability to efficiently produce immediately actionable recommendations for medical practitioners and policymakers.

Many of the individuals working in these communities may have cross-cutting interests and play some role in multiple communities, but we see the communities as having discrete identities. The characterization above is not meant to be exhaustive or to eliminate the possibility of other groupings, but rather to convey our understanding of the relationships between various problems, interventions, and individuals.

The path from here

At this point, the community we feel we have covered the most thoroughly is #2, the “open science” community. This hasn’t been an entirely deliberate decision: we’ve spoken to the people we’ve been pointed to and the people they’ve pointed us to, and only after many conversations have we noticed the patterns and distinct communities discussed above.

Because it is important to us to complete a medium-depth writeup, we’re currently aiming to complete such a writeup on open science. We will add the other communities discussed above to our list of potential shallow investigations.

In this process, we’ve learned that it can take a fair amount of work and reflection just to determine what counts as a “cause” in the relevant way. We think such work and reflection is worthwhile. Rather than speaking to everyone who is somehow connected to a problem of interest, we seek to identify different causes, deliberately pick the ones we want to focus in on, and cover those thoroughly.