The GiveWell Blog

Giving to GiveWell’s recommended charities helps GiveWell

(updated on: ) | by Holden

GiveWell does not solicit donations from the general public. We cover our operating expenses mostly by privately soliciting donations from people (and institutions) who are big fans of our work. This allows us to operate – when dealing with those outside our “inner circle” – as a pure advisor, without having to “compete with our own top charities.”

Occasionally this policy creates some confusion for donors interested in giving to promote “effective giving” in general, not just interested in directly funding proven interventions. Such donors sometimes express interest in donating directly to GiveWell, and when we state that we aren’t seeking donations from them, they conclude that their giving can’t help us. But it can, significantly, if they give to our recommended charities (in such a way that we can track the donation).

The main way this takes place is through our money moved metric, which tracks the donations given to our recommended charities on the basis of our recommendations. This figure has several important implications for our mission:

We advertise our “money moved” figure for charities in order to make the case that our investigations are worth their time. Since our founding 2007, we’ve seen continuously improving access to the people whom we need to speak with in order to do our research well: charity representatives, academics, etc. We believe our growing “money moved” has been a key factor. Because of this dynamic, higher “money moved” directly improves the quality and efficiency of our research process.

More generally, our “money moved” figure serves as a general indicator for interest in effective giving. We hope that in the future, the “effective giving” world will be much larger, with more organizations devoted to helping to find the best giving opportunities. People who would consider starting such organizations can learn about the size of the potential “market” from our figures. Charities that are outside our current scope may still wish to start planning for a future in which our sort of analysis plays a larger role, and our figures may serve as an indication of how likely this future is.

Our “money moved” figure is a major tool for our own fundraising, and a major input into the size of our operating budget. The higher the figure, the stronger the case for a growing operating budget, and the easier time we have raising the funds we need for it.

Our growth in money moved has been strong so far. If this weren’t the case, we’d have had far more trouble raising money, doing research, and making the case for our influence.

If you’re a donor looking to help GiveWell in our mission, please consider

  • Donating to our top charities.
  • Making sure that the donation is attributable to GiveWell and thus countable in our “money moved” metric. You can do this by giving using GiveWell’s website (note that in the case of our #1 charity, AMF, our website points to AMF’s website and AMF provides the tracking function) or by using our donation report form.
  • Telling your friends about us using Facebook, Twitter, email, or in-person conversations. Note that we track the number of donations we influence as well as the number of dollars, and we pay attention to (and advertise) the robustness of our “money moved” figure as well as the level.

Evaluating people

(updated on: ) | by Holden

A crucial consideration, when evaluating a giving opportunity, is: “Who are the people involved?”

I believe that any given project is likely to run into many factors that are unexpected (or can’t be captured in the evaluation process), and indeed that such unexpected factors often dominate the expected factors. Therefore, no matter how promising a project looks on paper, I expect it to fail if the people behind it aren’t capable enough; conversely, if the people behind a project are sufficiently capable, even a project that sounds terrible to me could turn out to be excellent (or be adjusted as it progresses to be excellent) based on factors that I fail to understand at the time of evaluation. Bottom line – I expect funding great people to lead at least to good (if not great) things, and funding incompetent people to lead to failure, in many cases regardless of most other factors. (Two important counterpoints: the choice of sector is important, as great people often work on issues they’re passionate about and the good they can accomplish can be limited by their choice of issue, and room for more funding is always crucial: even the most capable people generally have limits to how much funding they can productively use).

I’ve believed this for a long time, but GiveWell hasn’t used “evaluation of people” as a significant explicit factor in our evaluations (though we have taken the position that much of what we look for implicitly selects for good people). Lately, we’ve found ourselves putting more explicit weight on our impressions of an organization’s people. This post discusses why we’ve previously been hesitant to explicitly weigh and discuss this factor, what has changed, and how we evaluate people.

Challenges of explicitly evaluating people

It’s worth noting that GiveWell’s process to date has contained a great deal of implicit evaluation of the people involved in the charities we’ve examined.

  • One of our main goals is evaluating a charity’s track record. We’ve argued in the past that evaluating an organization’s track record can be a good way to evaluate its people. In the nonprofit world, there’s an additional wrinkle in assessing track records, which is that most nonprofits don’t have the data to make such assessment possible; we think those that do are likely to have important values in common with us, as discussed below.
  • More generally, we tend to ask many critical questions of a given charity, and to publish its responses along with our evaluation of those responses. We feel that asking critical questions about a person’s work and evaluating how much the answers seem to make sense is another good method for evaluating people.
  • Our emphasis on transparency has created an interesting way of distinguishing between charities: some groups take a near-“open book” attitude, others aren’t willing to put any of their materials or communications with us on the record, and others fall somewhere in between. We believe that this distinction may (imperfectly) capture important values, as discussed below.

That said, there are observations we make and opinions we form of people that don’t necessarily show up on their organizations’ GiveWell reviews. We notice not just how good an organization’s track record is, but how intelligently (from our perspective) its staff discuss the relevant issues; we notice not just how good its final answers to our critical questions are, but how efficient are the exchanges leading up to these answers. We form impressions that can’t always be pinned down to specific (or public) occurrences.

So far, we’ve been hesitant to put much weight on, or publicly discuss, these sorts of factors. A major hesitation is that we’re wary of putting too much weight on informal, intangible impressions that are likely to end up being “tests of how similar charity representatives are to us” rather than “tests of how well-suited charity representatives are to do their work.” It seems easy for evaluations of people to end up being driven by charisma, by similarity to the evaluator, and/or by characteristics that don’t translate well to nonprofit work. (Regarding the latter: many philanthropists have been successful in other domains, and are used to being able to pick out people who can succeed in the domains they know well; we fear that they aren’t able to build up the same experience and reliability of intuition in the nonprofit world, where it’s hard to know who has and hasn’t succeeded, and that the qualities they’re looking for may not translate well.)

Reasons our stance is shifting

Despite the concerns above, we’re now starting to factor our views on people more heavily into our prioritizations and recommendations, and we’re starting to feel that it’s important to write explicitly about our views on this front (not just about charities’ performance on our standard criteria). This is because

  • We’ve had enough experience to start feeling that we recognize patterns in what sorts of people/organizations eventually do well vs. poorly in our process, and as our process has improved we’ve become more confident in its conclusions.
  • Improvement in our process has also led us to have less confidence in any given empirical case (example). We’ve come to feel that we need to supplement empirical analysis with whatever other sources of views are available, even if they’re somewhat intangible.
  • We’ve been thinking more about how to evaluate charities without relying on our “proven cost-effective” framework; when doing so, explicit evaluation of people is more important.
  • We’ve been learning how rare it is that one can see how a charity has performed over time; because of this, it’s become more important to us to support groups that could contribute to changing this, i.e., groups that share some of our core values such as valuing self-evaluation and transparency.

That said, the concerns listed in the previous section still apply, and thus we are still cautious about placing heavy weight on our reads of people.

How we evaluate people

We look for the following qualities:

  • Clear, direct communication. We view clear, direct communication as a sign that someone welcomes – rather than prefers to avoid – (a) substantive discussion of their work; (b) honest criticism leading to improvement. It’s also generally easier for us to communicate with people who have this quality, which makes us more confident in our evaluation of them and their organization.
  • A sense that the person is following their own agenda rather than catering to ours. We try to avoid people whose primary goal seems to be telling us what we want to hear. If someone seems averse to contradicting or criticizing us – or seems more interested in the question of “what plans GiveWell would like the organization to carry out” than “what plans the organization would like to carry out” – we don’t consider this a good sign. We prefer organizations that set their own agenda and directly engage our questions.

    Clear, direct communication is a positive sign on this front, particularly when it includes criticizing, contradicting, and educating us.

    (We discussed this factor in a previous discussion of our stay in India.)

  • Thoughtful, plausible answers to our critical questions. In our view, one of the marks of an exceptionally capable person is that they have thought extremely deeply and thoroughly about the work they specialize in, to the point where they have thought of nearly any critical question that we could raise, and have thought about it more deeply than we have. This doesn’t necessarily mean they will be able to give an answer we find fully satisfying on any given critical question, but it means they will usually give an answer that shows they have thought hard about the issues and have an answer that we could plausibly find satisfying if we knew more.
  • Personal and organizational history. We look for generally impressive accomplishments and for experience relevant to the work being done. How we weigh these factors is highly context-dependent, and particularly depends on the organization’s mission. There are some types of work for which we’re comfortable with people who have little directly relevant experience, and others for which we find the personal and organizational resume to be more important.
  • Self-skepticism and valuing self-evaluation. This is an area where we see a significant disanalogy between for-profit and nonprofit work. When speaking to for-profit investors, we’ve heard the sentiment expressed that it’s natural and healthy for an entrepreneur to be overconfident; we can imagine that in the for-profit world, feedback loops and course correction often play out despite overconfidence, since people know they need to make adjustments when their deliverables (traffic, revenue, profit, etc.) are below expectations. But in the nonprofit world, there’s often (or always) no way to learn about whether your work is helping people unless you make a substantial, explicit effort to learn. People with high confidence and charisma and low self-skepticism worry us: we fear that they will continually succeed in raising money from donors but will never do the investigation and adjustment necessary to optimize their work for helping people.

    From what we’ve seen, it’s extremely rare to find people in the nonprofit world who (a) concede many possible ways in which their programs could be failing and (b) have put substantial thought and effort into investigating the relevant issues. People who do have these qualities tend to produce unusually meaningful and informative self-evaluation. We value such self-evaluation greatly not just because it’s informative but because of what it reflects about the people involved.

  • Valuing transparency. While there are sometimes good reasons to keep particular information confidential, a general preference to share more information publicly is likely, in our view, to go hand-in-hand with (a) self-skepticism and interest in being critiqued; (b) an interest in creating public goods (i.e., helping others to learn from one’s own experience) and helping the world as a whole rather than in focusing on the success of one’s organization. We see significant, tangible variation in attitudes toward transparency: some organizations want everything kept off the record, others are willing to share some information but wish to withhold whatever does not reflect well on them; others take more of an “open book” attitude.

Perhaps not surprisingly, these qualities mirror the qualities we believe ourselves/GiveWell to have. This makes sense for a couple of reasons. First, we’d guess that people are usually most impressed by people who are similar to them in important ways – sharing basic values, sharing communication styles, etc.; while we’d like to be able to identify people who are outstanding at what they do despite being poor at communicating with us, this is simply practically difficult. Second, we perceive ourselves as bringing important and undervalued values and qualities to the nonprofit sector, and part of what we find ourselves looking for in charities is others who will spread similar values and qualities.

We recognize that we may be overrating the importance of the qualities discussed above, but we feel that we maximize our own value-added by pursuing and promoting them, and part of the goal of this post is to ensure that our audience knows these are the values we’re seeking to pursue and promote.
We believe that other funders often look for substantially different qualities – in particular, they often seem to place more weight on charisma and fundraising ability, perhaps because they are hoping their investment will be leveraged into donations from many others. There is sound logic behind this preference. However, our feeling is that the current nonprofit sector overemphasizes storytelling ability and fundraising success, relative to thoughtfulness and transparency – with the result that we see a lot of the former and less than we’d like to see of the latter – so the qualities we look for are different..

Over the coming weeks, we will be adding sections to the pages on our top charities that discuss our views on each organization as a whole (including its people).

Revisiting the case for insecticide-treated nets (ITNs)

(updated on: ) | by Holden

GiveWell seeks to constantly revisit and rethink our content, to make sure that it continues to represent a reasonable interpretation of the currently available facts. To this end, we assigned Jonah Sinick, a Research Analyst, to perform a thorough review of the content most relevant to our #1 charity (the Against Malaria Foundation): the AMF review page and the page on distribution of insecticide-treated nets. We asked Jonah to check our footnotes, think critically about our reasoning, and list as many potential problems with our content as he could find; we committed to publishing a list of all such potential problems along with our responses.

The full output of this exercise is here; a summary of the major issues raised and our responses is here. This post focuses on one particular issue Jonah raised – a question about the extent to which small-scale studies of ITN efficacy are representative of real-world conditions (in particular, real-world malaria burdens) – and describes the steps we took to investigate the matter. In brief:

  • Jonah found some initial reason to suspect that our “cost per life saved” for ITN distribution was unrealistically low. He noted that based on available country-level estimates for malaria mortality, ITNs could not save lives with the frequency we were claiming, even if they prevented all malaria deaths.
  • In response, I did some initial estimates that implied the correct “cost per life saved” figure might be 5-10x higher than our current figure, and accordingly marked the issue as a high priority.
  • After further investigation, we concluded that, in fact, no major revision to our “cost per life saved” figure is called for. The deaths averted by ITNs appear to include deaths that are not strictly classified as “malaria deaths,” and available data does not give us reason to believe that the ITN efficacy studies are unrepresentative.
  • However, a smaller issue – the decline in all-cause child mortality between the time that ITN efficacy studies took place and today – does potentially call for a relatively small adjustment, and also adds a new source of uncertainty regarding the cost-effectiveness of ITN distributions.
  • There are multiple issues that we continue not to account for in our cost-effectiveness estimate, including other positive impacts of nets: prevention of non-lethal malaria cases, reduction of malaria deaths among those over the age of 5, and contribution to the prevention of other diseases transmitted by mosquitoes such as lymphatic filariasis.

Our bottom line after going through this exercise remains that insecticide-treated net distribution remains one of the most cost-effective programs we know of and we continue to rank the Against Malaria Foundation as our #1-ranked charity.

The rest of this post describes our investigation in greater detail than we usually do; since we are publishing the results of the vetting, we thought we’d take this opportunity to give interested parties a more-in-depth-than-usual look at the kind of back-and-forth and analysis we are constantly engaging in to get a better understanding of our top charities.

Overall, we take this episode as another example of how difficult it is to have high confidence in a “cost per life saved” type figure. Our estimate represents the figure that we come to when we use the best evidence we can to estimate each parameter as fairly as we can, but it does not include a Bayesian adjustment to account for the substantial uncertainty, both on account of issues we’ve thought of and issues we haven’t thought of; as such, it should not be taken literally.

That said, we also think the issues discussed in this post illustrate a good reason to do cost-effectiveness estimation: it is a way of disciplining ourselves to make sure we’ve addressed every input and question that matters on the causal chain between interventions (e.g., nets) and morally relevant outcomes (e.g., lives saved). In this case, Jonah’s examination of our cost-effectiveness estimate led to many new questions and observations, and we feel that our overall understanding of the strengths and weaknesses of the case for ITN distribution has improved. This benefit of cost-effectiveness estimation is only realized when estimates are continually revisited and rechecked, which in turn requires that they be fully transparent to those checking them.

Note that a future post will discuss another concern we have investigated this year: the question of whether mosquitoes develop resistance and/or adapt their behavior to ITNs over time.

Sharing notes from conversations: Case study in pursuing transparency in philanthropy

(updated on: ) | by Holden

Early in our history, we got most of our information in the form of documents: grant applications and other documents sent to us by charities, academic literature, etc. Being transparent about why we believe what we believe was thus relatively straightforward: we sought to publish whatever documents we could.

Over time, phone and in-person conversations have become a much larger part of our process. The biggest reason for this is that our “access” – our ability to get people to talk to us – has improved (see our annual review for 2011). Conversations add a lot of value to our process, above and beyond what we could get from documents, but being transparent about this source of information – i.e., publicly sharing what we’ve learned in conversations – has been a challenge.

Below, we outline the trial-and-error by which we arrived at our current process for sharing notes from conversations; lay out what that process is at the moment; and discuss this case as an illustration of a general dynamic that we think applies fairly broadly when it comes to transparency in philanthropy:

  • It is possible to arrive at reasonable processes for conducting philanthropic investigations transparently.
  • But developing these processes can take time and can have significant short-term costs.
  • A funder that holds transparency as a high value can often find ways to fulfill the value. But a funder that has traditionally operated without transparency as a high value will not necessarily have an easy time making the switch.
  • We see ourselves as “trailblazing for transparency”: while being transparent isn’t easy for most philanthropic funders today (and it presents challenges for us), we are developing and writing publicly about our experiences and processes, so that future philanthropists will find it easier to emphasize transparency from day one.

Good Ventures has had a similar goal of sharing notes from conversations, and it has been an active participant in the evolution we describe below.

Early on, when we had a particularly informative conversation, we would often write up a summary (or rough transcript) of it and ask the person we had spoken with what they would think of our publishing it publicly. In all cases, we offered the person the opportunity to make edits to the document beforehand – our intent was not to capture the conversation word for word, but to share the insights and knowledge of the person in question.

There were cases in which this worked out (for example, a 2009 conversation with VisionSpring), but there were also a couple of cases in which the person we spoke with was taken aback at our request. In the latter cases, the problem seemed to stem from the fact that we had asked about publishing notes after the conversation rather than before it (we had been inconsistent on this point): we received complaints along the lines of “I spoke with you frankly because I didn’t know you would be seeking to publish anything publicly; now I feel ambushed.” In response to these cases, we determined that we needed to standardize our process for conducting conversations to ensure that we always raised the possibility of publishing notes before the conversation began. We also sought to clean up our notes before sending them for review, to reduce the editing burden on the person we sent them to and increase the odds of their being approved.

As our volume of conversations increased, taking, cleaning and sending notes for review became a major burden on our capacity, particularly because it was hard to specialize the note-taking role (since notes generally had to be put together by someone who had been present for the conversation).

We discussed the idea of simply recording conversations (with permission) and posting the recordings, but in the few cases where we had raised this as a possibility, it had generally seemed to make people uncomfortable (though there were exceptions, such as a conversation with William Easterly). Instead, we started asking to record the conversations for internal purposes only, and we dedicated one of our staff members to transcribing these conversations after the fact. We settled on relatively standardized language for making our requests (language that Good Ventures helped craft and also uses in its communications), and adopted a procedure of putting this language in pre-conversation emails (usually when finalizing the time/date/venue for a conversation):

A goal of ours is to share as much as possible about our research publicly, so that others can learn from our work. So if you’re up for it, we’d like to take notes on this call for the purpose of posting them on our website later, pending your review and approval.

After the call, we’d run the notes by you, and if there were anything you wanted to keep confidential, or any changes you wanted to make, we’d be happy to do so. If you decided you’d rather we not publish the notes at all, we’d be happy to oblige, because we never want to create a disincentive for people to speak frankly with us. What do you think?

Along the same lines, would it be OK to record the call? The recording would be for our internal use only, so that we can focus on listening rather than on taking notes.

Good Ventures also currently uses (and helped to develop) the language and procedure described above.

We found that implementing this process solved a lot of our former problems: we no longer had cases where people were taken aback at our requests (people who weren’t comfortable with notes or recording could tell us so in advance of the call), and we had a dedicated staff member producing transcripts and saving the rest of staff’s time. The dedicated staff member has, over time, become better at taking notes that concisely hit the main points of the call and are highly likely to get approved with few changes (at one point we experimented with simply producing full transcripts, but these weren’t well received because they required the people we had spoken with to do a lot of reading and editing).

So we now have a process that we feel works fairly well, with the result that the vast majority of our conversations lead to public conversation notes capturing the highlights, and the costs in terms of relationships and our capacity seem quite manageable. But as detailed above, it took a good amount of time, trial and error to get to this point.

Notes from our conversations are available at our conversations page and via our new content feed.

Update on GiveWell’s web traffic / money moved: Q3 2012

(updated on: ) | by Natalie Crispin

In addition to evaluations of other charities, GiveWell publishes substantial evaluation on itself, from the quality of its research to its impact on donations. We publish quarterly updates regarding two key metrics: (a) donations to top charities and (b) web traffic.

The charts below present basic information about our growth in money moved and web traffic thus far in 2012. Website traffic tends to peak in December of each year (circled in the chart below). Growth in web traffic has generally remained strong in 2012.

So far in 2012, there have been 342,551 monthly unique visitors (calculated as the sum of unique visitors in each month) to the website, compared with 238,172 at this time in 2011, or 44% annual growth. A significant contributor to this growth was an increase in our Google Grants budget (which provides us free advertising via Google AdWords). Excluding this, we have had 23% annual growth in web traffic.

Growth in money moved has remained strong as well. The majority of the funds GiveWell moves come from a relatively small number donors giving larger gifts. These larger donors tend to give in December, and we have found that growth in donations from smaller donors throughout the year tends to provide a reasonable estimate of the growth from the larger donors by the end of the year.

Below, we show two charts illustrating growth among smaller donors.

Thus far in 2012, GiveWell has directed $604,327 to our top charities from donors giving less than $10,000. This is approximately 2.4x the amount we had directed at this point last year.

Most donors give less than $1,000; the chart below shows the growth in the number of smaller donors giving to our top charities.

Overall, 1531 donors have given to GiveWell’s top charities this year (compared to 624 donors at this point last year).

In total, GiveWell donors have directed $1,394,071 to our top charities this year, compared with $766,561 at this point in 2011. For the reason described above, we don’t find this number to be particularly meaningful at this time of year.

US Cochrane Center (USCC) gets our first “quick grant” recommendation

(updated on: ) | by Holden

Recently, we did something that may strike many GiveWell followers as out of character. We recommended a $100,000 grant to the US Cochrane Center, despite the fact that we have done relatively little investigation of it so far (compared with our investigations of current top charities)—and have many unanswered questions. Good Ventures, which helped with our investigation and therefore followed it closely, was a part of the conversation in which we came to the conclusion that this grant represented a good giving opportunity, and it committed the funds shortly afterward (before we had finalized our writeup; we considered this appropriate since, as we discuss below, speed was desirable in this situation.*)

This post covers two topics:

  • Why we believe it is important to be able to make quick grants (i.e., grants with far less than our usual level of investigation) when warranted, and we are working on principles for doing so.
  • Why we believe that the grant discussed in this post meets our working criteria for a quick grant.

In brief:

  • We believe that in certain cases, speed is valuable in grantmaking: sometimes because circumstances demand action by a certain date (for example, some projects involve close coordination between multiple entities, including governments, and may need to happen on timelines that work for these entities), and sometimes because speed in giving can help organizations make better planning decisions.
  • We have developed working principles for when a “quick grant” is called for, which are discussed in this post. We intend to experiment with “quick grants” while ensuring that they represent only a small portion of our money moved in the near future.
  • We see the USCC as an ideal recipient for our first “quick grant” partly because we are continuing to investigate the USCC as a potential top charity (in which case we would recommend it to individual donors). The grant discussed in this post meets our working criteria for “quick grants,” and in addition, we are confident that we will learn more over time about the extent to which this “quick grant” was warranted.

Note that the bulk of this post was written in July, following the grant recommendation and grant. We held publication of the post, after drafting it, for a variety of reasons including (a) the desire to get and respond to feedback from all parties discussed in this post (as we generally do); (b) the desire to publish more public communications about our evolution as an organization in order to give more context on how this recommendation fits in.

The importance of making “quick grant” recommendations

To date, our usual approach to making recommendations has been:

  • Survey an entire field, looking for the organizations that perform best according to various heuristics.
  • Deeply investigate the top-contender organizations, making a major effort to answer all major questions to the degree that we reasonably can.
  • Set a deadline (usually giving season) by which we must make or refresh our recommendations; this ensures that we avoid perfectionism and make the best recommendations we can with the information we have.

We’ve put an increasing amount of effort into investigating contenders for our #1 ranking; we’ve found it very important that we be able to fully stand behind any such recommendation, with reasonable answers to any question that one might raise.

There are a lot of positive things about this approach, and we intend to retain it for the recommendations that comprise the bulk of our money moved. We believe that being systematic, deliberate and thorough is likely to lead to much better giving, in general, than relying on happenstance and intuition alone; at the same time, having regular deadlines leads to regular giving (which we favor).

That said, we also believe that in certain cases, speed is valuable. Sometimes speed is valuable because circumstances demand action by a certain date (for example, some projects involve close coordination between multiple entities, including governments, and may need to happen on timelines that work for these entities). Sometimes the argument for speed is more subtle – for example, giving quickly can help organizations make better decisions about how to plan their budgets as well as how much time to invest in fundraising from different sources.

We’ve long been intrigued by the ideas of people like Bill Somerville, and wondered whether the approach of taking calculated risks after a shorter period of research than has been our norm has something to recommend it at times. Since connecting with Good Ventures, this issue has become more salient to us for a couple of reasons:

  • We were the recipient of one such “quick grant.”After our first interaction with Cari and Dustin (a 90-minute meeting between them and myself in February 2011), they expressed an intention to contribute $100,000 over two years in operating support to GiveWell. At that time, we were facing a substantial projected deficit and were investing more time than usual in fundraising. This “quick grant” was extremely helpful for our planning and efficiency, in a way that it couldn’t have been if it had required the kind of investigation that GiveWell normally puts in. Noticing this further highlighted the advantages of “quick grants.”
  • In addition, our relationship with Good Ventures means that we are able to make “quick grant” recommendations that can be taken up quickly; while we have some other connections who provide similar opportunities, the bulk of our other money moved tends to be concentrated in the month of December.

On May 8, two GiveWell representatives (Stephanie Wykstra and I) visited the US Cochrane Center (the USCC) (note that we have posted notes from the visit). For reasons discussed below, we came away with the impression that (a) the USCC plays an important role in meta-research; (b) the USCC faced a drastic shortage of operating funding, even to the point where it might not be able to continue uninterrupted, minimal staff support. Over the next six weeks (as discussed below), we tried to find persuasive counterarguments to this viewpoint and failed to do so. While I still had many unanswered questions, it struck me that a grant to the USCC, made sooner rather than later, could potentially do an enormous amount of good in terms of helping the USCC plan intelligently; the mere fact that I wasn’t confident in the value of such a grant didn’t seem to change the fact that the expected value of making such a grant quickly seemed high.

While principles such as thoroughness and transparency are important, the principle of doing as much good as possible with one’s giving is more core to GiveWell than any other. We strongly believe that we should never find ourselves passing up what we see as an opportunity to do maximal good, just because the opportunity is an awkward fit with our existing habits and processes. We are constantly watching the actions of other funders and asking whether they are finding great opportunities to do good, in a way that our existing approach would fail to (and if so, what we can do about it).

Thus, once we saw the potential value of a “quick grant” to the USCC, we started actively considering the idea of modifying our approach to facilitate such a grant (rather than passing on the grant because it didn’t fit well with our existing approach). At the same time, we wanted to make sure that we were adopting and discussing a principled modification to our general approach, not simply recommending a “quick grant” on a whim. As such, we worked to develop the best set of general principles we could for when a “quick grant” is called for. These principles were inspired by the case of the USCC, but we took our best shot at making them reasonable for general application.

Principles and process for making “quick grant” recommendations

This section discusses (a) the key questions we feel are appropriate for a potential “quick grant”; (b) the process we intend to follow for answering these questions quickly and efficiently and making “quick grants.”

Key questions for a potential “quick grant”

  1. Is there a reason that speed, in and of itself, is valuable for this grant? Some possible reasons that speed may be valuable:
    • There may be a specific reason that funds are needed by a particular date in order to go forward with a particular project.
    • There is always a potential argument that (as outlined above) an early commitment can help an organization plan better and use its time more efficiently. This argument will generally be stronger in the case when (a) a relatively small grant can make a big difference to an organization’s planning (usually because the organization has little in the way of unrestricted support), and (b) we have a positive view of the organization and its people overall (as opposed to a small subset of the work it does).
    • An early grant can also be helpful for learning purposes. If an organization is highly promising from our perspective, but hesitant to engage in our process because of how it perceives the costs and benefits of doing so, an early grant may be valuable as a way to improve our access and ability to learn.
  2. Is the organization/project in question focused on work that seems valuable, reasonably cost-effective, suited to philanthropy (as opposed to other approaches) and thus “worth doing” overall? For the purposes of a “quick grant,” the approach to this question will be highly intuitive and perhaps unsatisfying compared to the cost-effectiveness analysis we often do. Explicit cost-effectiveness analysis is very difficult to do with reasonable speed, without sacrificing reasonable robustness, and when comparing radically different approaches to doing good it can be nearly impossible to say with much confidence how they compare in terms of “bang for the buck.” This question acts only as a basic screen: is the organization/project in question filling a valuable role in an important ecosystem, leveraging the work of others, and overall taking on an approach to helping people that is at least defensible from a cost-effectiveness standpoint?
  3. Do we see a convincing reason that this organization would not be able to raise the funding it needs in the relevant time frame, even if it made a good case for such funding? As discussed below, one of the things that excites us about the USCC as a funding opportunity is the sense that the USCC is “structurally underfunded”: it struggles to raise funds not because major funders have thoughtfully considered and rejected its case, but because major funders largely have program areas and issue focuses that don’t leave room for the kind of work the USCC does.

    This is not often the case. We often find ourselves saying, “It seems that if this work were as valuable as the organization claims, funder X would support it.” When such a statement can be convincingly refuted, the case for a “quick grant” (as for a recommendation in general) becomes much stronger.

  4. Are the people involved impressive, competent, and capable of making good on-the-fly decisions, such that we’re comfortable with grants that we have fairly little visibility into the specific intended use of? We’ll be writing more in the future about how we evaluate people. This question is important for all recommendations but is particularly important for “quick grants,” since it generally takes us a long time to feel confident about the specifics of how funds will be used; we are much more likely to recommend “quick grants” when these specifics aren’t necessary in order to feel the funds will be used well.
  5. How much will we learn in the future about the organization/project and the extent to which the “quick grant” was a good idea? Since “quick grants” are likely to be smaller than the money we moved to our top charities, it isn’t a given that we will engage in the same sort of followup on them that we do for our top charities. In addition, we’ve found that it’s much easier to learn about an organization when we invest up front in defining goals, metrics, etc., which is more difficult to do in the case of a “quick grant.” So by default, there is a risk that we won’t learn much from or about a “quick grant”; we need to be attentive to this and have a strong preference for grants with more learning potential.

    We expect that there will often be cases in which we consider “quick grants” to organizations that we also plan to evaluate more thoroughly (with the possibility of moving significantly more money to them). In these situations, the situation looks much better in terms of learning about whether our grant was a good one and what ultimately came of it. The USCC is one of these cases.

Our process for making “quick grants”

We’ve provisionally agreed to the following process for making quick grants:

  • The process starts when we have some unusually strong signs that the answers to the above key questions are positive. We do not need a definitive case; the case may be largely intuitive and suggestive, but it should be unusually strong in the scope of opportunities we come across.
  • We then pick any “low-hanging fruit” in terms of further investigating the answers to our questions – any investigative work that can be done quickly and is likely to lead to a substantially better understanding of the situation.

    While all of the above key questions are important, we are likely to have a fairly quick and intuitive read on questions #2, #4, and #5, and the questions that we are most likely to focus on investigating are #1 (is there a reason that speed is likely to be helpful?) and particularly #3 (are there other funders who are a logical fit to fund this organization/project or is it underfunded for structural reasons?)

  • We actively seek out counterarguments to our views on key questions, as effectively and efficiently as we can. The main approach we used for the USCC – an approach we are likely to use for future “quick grants” as well – is to seek out conversations with funders who seem like the closest logical fit for the funding opportunity, and try to understand whether they are (a) planning to fund the project/organization; (b) planning not to fund the project/organization, for reasons we find compelling; (c) planning not to fund the project/organization, for reasons we don’t find compelling. (A “quick grant” should be made when (c) holds, not when (a) or (b) holds.)
  • Before any “quick grant,” we hold a meeting or conference call that combines (a) all staff who have been highly involved with the investigation; (b) some staff who haven’t; (c) funders who are particularly likely to follow the recommendation for a “quick grant.” The staff recommending the “quick grant” summarize the answers to key questions as well as what investigations have been done to learn more about these questions and to identify counterarguments. Others on the call focus on (a) evaluating the strength of the arguments (answers to key questions) given the information already available; (b) determining whether there is other information that would be likely to quickly and substantially shift answers to the key questions (i.e., whether there is “low-hanging fruit” in investigative terms).
  • If the basic case for a “quick grant” is accepted, the next step is to determine the size of the grant recommendation (i.e., the dollar amount past which we would stop recommending a “quick grant”). When the “quick grant” is to meet a specific need or fund a specific project, we should understand the nature of the time sensitivity and the size of the need, and the current funding status, before recommending the award. When the “quick grant” is more along the lines of general support for purposes of helping the organization plan and/or improving our access to the organization, we should pay more attention to the size and variance of the organization’s budget (as well as any available room for more funding analysis) and try to aim for something that provides substantial benefit (in terms of planning and/or access) but doesn’t come close to meeting all the organization’s needs.
  • When we do recommend a “quick grant,” we publicly write up the recommended grant amount, recipient, and a summary of our process and reasoning in making the recommendation.

How we decided to recommend a “quick grant” to the USCC

We’ve long been familiar with the work of the Cochrane Collaboration, having used it in our research. We’ve noted before that

We have found that its reports generally review a large number of studies and are very clear about the findings, strengths and weaknesses of these studies. For health programs, when there are often many high-quality studies available, we therefore use Cochrane as our main source of information on “micro” evidence when possible.

In April of this year, I attended a meeting on preregistration in development economics and encountered Kay Dickersin of the USCC, who stated to me that (a) the USCC is struggling to attract unrestricted support; (b) if the USCC had sufficient funds, it would provide general support to US-based Cochrane entities, including direct financial grants in cases where these entities appeared underfunded. We scheduled a full-day visit to the USCC in Baltimore to learn more, because (a) we have long respected the Cochrane Collaboration’s work and were surprised to hear that the USCC was struggling to attract unrestricted funding; (b) this was the first concrete giving opportunity we’d encountered in the area of meta-research, which is a new high-priority focus area for us and which we’re seeking to learn more about; (c) the USCC showed a high level of interest in engaging with us and offered to put together a full-day meeting with multiple representatives, which both raised our expectations about what how much we could learn and served as an additional signal that the USCC was struggling to attract sufficient operating funding.

Our notes from the full-day meeting are published online (DOC). We came away from the meeting feeling there was a strong preliminary case for the USCC based on the five key questions above (though we had not yet formalized these questions as the key ones for “quick grants”):

  • Is there a reason that speed, in and of itself, is valuable for this grant? The USCC appeared to have a concrete and time-sensitive need for unrestricted funding, including the immediate need for funds to continue uninterrupted, minimal, staff support. In our view, this situation is notable not just because of the specific consequences that a grant might have (allowing the USCC to retain core staff), but also because it more broadly illustrates that the USCC does not have a stable situation in terms of unrestricted funds, and thus that support could help it to plan and set priorities more effectively.
  • Is the organization/project in question focused on work that seems valuable, reasonably cost-effective, suited to philanthropy and thus “worth doing” overall? We are positive on the quality of the Cochrane Collaboration’s work, as discussed above; at the meeting we also came away with preliminary reasons to believe the work is influential as well (though we plan to investigate this more). As for the USCC’s role in the Cochrane Collaboration, we saw fairly strong arguments on this point. The Cochrane Collaboration relies on training and supporting volunteers, many of whom are academics. The U.S. has many potential volunteers, including those based within the country’s large university system. But in the U.S. there is far less funding for Cochrane infrastructure (i.e., to train and support volunteers) than in other English-speaking countries such as the UK, Canada and Australia. Jeremy Grimshaw, co-chair of the Cochrane Collaboration’s International Steering Group and Director of Canada’s Cochrane Center, was present by phone at the meeting and supported the message being sent that the USCC is a point of particularly high leverage and importance for the Cochrane Collaboration as a whole. For the reasons discussed above, we don’t feel that formal cost-effectiveness analysis is likely to be helpful in this case.
  • Do we see a convincing reason that this organization would not be able to raise the funding it needs in the relevant time frame, even if it made a good case for such funding? We questioned the USCC about many potential sources of funding and were told that the lack of funding was largely for structural, not substantive reasons. That is, the funding needed would support the infrastructure required for the Cochrane Collaboration’s work, such as staff for training, and methodological support for reviews, and not hypothesis-testing research. As such, the Collaboration’s needs do not fit into the pre-defined categories and issue areas of major funders. Thus, potential funders have considered and declined Cochrane requests and applications for general operating support based on “not a good fit” rather than the quality or importance of the work, overall. This was the point we felt we most needed to examine further after the meeting, and we did so, as discussed below.
  • Are the people involved impressive, competent, and capable of making good on-the-fly decisions, such that we’re comfortable with grants that we have fairly little visibility into the specific intended use of? Multiple representatives were present, and overall we felt they answered our questions reasonably clearly and well; the USCC also appears comfortable with transparency, having signed off quickly and permissively on our notes from the meeting. Our general positive impression of the Cochrane Collaboration’s work is also relevant here. We currently have moderate confidence on this point; we anticipate learning more as we investigate Cochrane further.
  • How much will we learn in the future about the organization/project and the extent to which the “quick grant” was a good idea? We are currently performing an in-depth investigation of the USCC, considering recommending it for more funding than the initial “quick grant,” so we believe that we will learn a great deal about the extent to which this “quick grant” was warranted.

After the meeting, we agreed that the USCC was a promising organization, and our top priority became looking efficiently for counterarguments to the case for funding it. With Good Ventures’s help, we sought out conversations with the major funders that seemed to us like potential fits for the USCC, based both on our prior knowledge and from conversations with the USCC, hoping that we would gain more context on (a) whether it’s true that the USCC doesn’t fit into the issue areas of existing major funders; (b) whether there were general counterarguments to our preliminary views on the USCC’s value and need for more funds.

Feedback was solicited from:

  • Representatives of the Gates and Hewlett Foundations (we had no reason to believe they were a fit for the USCC, but thought they might know who would be, and see them as relatively impact-oriented funders in general; we are not cleared to share notes about these interactions).
  • A representative from the Wellcome Trust, a large medical research funder (notes available as DOC)
  • Representatives from U.S. government agencies: the National Institute of Child Health and Human Development (which contracts with one of the U.S.-based Cochrane review groups but does not provide unrestricted support to the USCC), the NIH Office of Medical Applications of Research in the Office of the Director (which we were pointed to in order to explore whether the USCC might be a fit for funding from the Office of the Director), and the Agency for Healthcare Research and Quality (which has funded the USCC in the past and, like the Cochrane Collaboration, commissions systematic reviews). We are not cleared to share our notes from these interactions.
  • A representative from the HIV/AIDS department of the World Health Organization. We are not cleared to share our notes from this interaction.

In these interactions, we asked for general impressions of the Cochrane Collaboration, thoughts on what sorts of funders might be structurally able to support the USCC, thoughts on what other groups do the sort of work that the Cochrane Collaboration does, and (when relevant) reasoning behind an entity’s support (or lack thereof) for USCC. We came away with the impression that the Cochrane Collaboration’s work is widely respected and seen as high-quality and important, that we can’t easily identify any major funders that are a structural fit for the USCC, and that the main other group focused on systematic reviews is AHRQ, which we plan to investigate further. (AHRQ’s role relative to Cochrane’s is discussed in the notes from our visit to the USCC; our takeaways from other conversations were broadly consistent with these notes.)

We also spoke to

  • Jeremy Grimshaw, co-chair of the Cochrane Collaboration’s International Steering Group and Director of Canada’s Cochrane Center. We sought to press the question of whether the USCC, specifically, is the best entity to fund in order to supporting the overall mission of the Cochrane Collaboration. Dr. Grimshaw conferred with other international Cochrane Collaboration representatives, including the other Steering Group co-chair, the interim Executive Director and the Editor-in-Chief of the Cochrane Library, and informed us that they endorsed supporting the USCC and would seek the formal endorsement of The Cochrane Collaboration Steering Group. Since then (following the grant), we have further pressed the issue of whether there might be other opportunities to support the Cochrane Collaboration that are higher-leverage than the USCC, and we are continuing to speak with Dr. Grimshaw about how best to work with international representatives to investigate this question. We therefore intend to investigate other Cochrane entities as well, and believe that doing so will take a significant amount of time, so given that the USCC had been endorsed as a strong opportunity in terms of potential leverage for a donation at the time of the grant recommendation (though no single opportunity within the Cochrane Collaboration was put forth as the “best”), we feel it was the right decision to move forward.
  • John Ioannidis, whom we see as a leading figure in the field of meta-research generally (and meta-research for medicine in particular). We have published extensive notes from this conversation in transcript form (DOC); Dr. Ioannidis has been involved in the Cochrane Collaboration in the past and believes the USCC to be a strong funding opportunity.
  • Professor Steven Goodman of the Stanford School of Medicine, a referral from one of the funders we spoke with (summary forthcoming).

Finally, we obtained detailed room for more funding analysis from the USCC; this analysis is now available online (DOCX).

Having done the above investigations, we felt that

  • We had strong – though far from conclusive – reasons to believe that the USCC has strong answers to our key questions. In particular, we believed that it had an urgent need for more unrestricted funding to assist with its planning; that its struggle to attract unrestricted funding could largely be attributed to structural issues (the fact that major funders often focus on particular diseases and do not prioritize meta-research) rather than to substantive objections to the USCC’s work; and that it was a strong candidate for “best leverage point for supporting the overall mission of the Cochrane Collaboration.”
  • We had many remaining questions about the USCC, and planned a thorough investigation to answer them. However, we didn’t see any “low-hanging fruit” remaining on the investigative end, and believed it would take a lot of work to obtain more satisfying answers to our key questions.

We discussed a $100,000 grant – enough to replace a specific source of support the USCC expects to lose, and enough to ensure that it can continue uninterrupted, minimal staff support. We came to the conclusion that $100,000 was enough to make a significant difference to the USCC without coming anywhere near meeting its funding needs (as expressed in the “room for more funding” analysis, linked above), and thus that such a grant could be justified not only based on its specific effect (allowing the USCC to retain uninterrupted, minimal staff support) but also on the more general principle (discussed above) of helping a highly underfunded organization with its ability to plan and prioritize. We checked in one more time with the USCC to make sure its funding situation had not changed materially, and recommended the grant.

*There is usually a substantial lag between our coming to a conclusion about a giving opportunity and our writing up & publishing our reasoning. In this case, for reasons discussed below, we did not want to accept the lag of writing up & publishing our reasoning before driving donations, so we made our recommendation via a discussion with Good Ventures and are publishing our reasoning now. In the future, we will generally publish our reasoning publicly before making a recommendation to any particular donor in cases where the funding gap is large and we are seeking to drive donations from many donors, and/or where a lag between the recommendation and the funding commitment is acceptable, but we may act as we have in this case when these conditions do not hold.