Some history behind our shifting approach to research

The approach that GiveWell took from 2007-2011 had two crucial qualities:

• We have been passive. That is, we have focused on finding the best existing organizations and supporting them with no-strings-attached donations, rather than a more “active” approach of designing our own strategy, treating charities as partners in carrying out this strategy, and restricting donations accordingly.
• We have sought proven cost-effective giving opportunities. That is, we have looked for situations where a donor can be reasonably confident – based on empirical evidence – that his/her donation will result in lives being changed for the better, at a high rate of “expected good accomplished per dollar spent.”

This year, we have been experimenting with giving opportunities that lack one or both of these qualities. We previously defended our shift in this direction; this post gives more context on the history that has led us to this point and discusses why we don’t think we can retain both of the qualities above and continue to find great giving opportunities at an acceptable rate. A future post will go into some of the questions we are addressing as we begin to shift our approach.

The history of GiveWell’s approach to finding outstanding giving opportunities
In our first year, we took an approach that was highly passive and highly focused on proven cost-effective interventions. We invited grant applications from a wide variety of nonprofits, and we didn’t attempt to focus on any particular strategies for helping people; we emphasized only that we wished to see a convincing case for proven, cost-effective, scalable impact, and we picked recommended charities accordingly. (See our first-round application linked from our overview of applications we received for our 2007-2008 research process.)

At that point we weren’t sure what to expect; what we found was that much of the most convincing evidence for effectiveness was at the “intervention level” rather than the “charity level” (i.e., there are programs, such as distribution of insecticide-treated nets, that have strong publicly available evidence bases, and few charities whose in-house evidence seems to add much to the case). Accordingly, for our 2008-2009 process, we did substantial independent review of research on aid interventions, and published a list of “priority programs” with strong evidence bases. This was a step in the direction of being less “passive”: doing our own independent analysis to determine what sorts of charities were most promising, rather than simply asking charities to make their own case.

In 2011, we announced an intensifying focus on deeply examining the best charities we could find (rather than evaluating charities by the standards of their issue areas). The research process that followed (leading up to our 2011 recommendations) was broad and open to many different types of groups, but it was also “active” in the sense that we often deprioritized a charity after the initial phone call, based on our judgment of how likely it seemed to ultimately merit a confident recommendation. In addition to examining charities focused on what we considered to be proven interventions, we also flagged charities for having seeming “high upside” in various ways, and considered these groups for recommendations; we tried to be open to groups we could recommend even though they didn’t work on what we considered priority interventions.

Ultimately, we didn’t find any such groups promising enough, and our top charities ended up being groups focused on interventions with strong evidence bases.

At this point, we feel that

• There is a distinct set of interventions – concentrated in the area of global health and nutrition – for which there is strong and generalizable empirical evidence of cost-effective impact on saving and improving lives.
• We have made intensive efforts over the past 3+ years to find all charities that focus on these interventions. Since these interventions are frequently delivered by/through developing-world governments, it is rare to see many charities taking different approaches to a given such intervention; it is more common that for each such intervention, there are a small number of fairly large charities that work with governments to provide funding, technical assistance, etc. in delivering these interventions.
• Our current two top charities are the groups we’ve identified that both focus on such interventions (e.g., we can confidently predict that additional donations to these groups will result in more of these interventions) and demonstrate the necessary transparency such that we can perform thorough evaluations and updates.
• Therefore, we do not expect to find any more “top charities” (in the sense we’ve previously used – “charities that will use additional dollars to carry out cost-effective, proven, scalable activities with high transparency and accountability”) in the near future.
• By focusing our efforts at the project level rather than the organization level, we may be able to generate more options for donors to deliver such interventions (e.g., considering a single nutritional program implemented by UNICEF. By being open to recommending the funding of particular projects – rather than just the writing of unrestricted checks – we would be further shifting in the direction of “active funding.”

Why aren’t there more organizations focused on our priority interventions?
It may seem puzzling that there are relatively few charities focused on what we consider the most proven interventions. Our basic picture of the reasons for this:

• As mentioned above, there are only a small number of interventions – concentrated in the area of global health and nutrition – for which there is strong and generalizable empirical evidence of cost-effective impact on saving and improving lives. It seems that global health and nutrition are particularly amenable to meaningful data and analysis; we feel that other sectors are very far from having meaningful data on how to improve lives, perhaps due to the inherent difficulty of measurement rather than due to a failure of effort. (We’ll be writing more about this idea in the future.)
• In many cases, these interventions are generally thought to be best delivered in partnership with the government (and often many other organizations) and at large scale. Large funders (for example, government funders and major foundations), when they seek to roll out these interventions, often work directly with governments; they may pull in nonprofits for specific sorts of support (example: the rollout of ART in Botswana).This dynamic may limit the opportunities for “entrepreneurial” charities working on these interventions (charities that start small and earn prominence through the work they do). Many of the organizations that do focus on these interventions were essentially founded with very large grants from large funders (examples: Schistosomiasis Control Initiative, GAVI).
• It is rare for a charity to be exclusively focused on one of these interventions; in fact, it is somewhat rare for a charity to be exclusively focused on any particular intervention. (For illustration, see our list of charities considered in 2009 and note how many have “highly diverse activities.”) A common theme in our conversations with charities has been that we often ask, “What would you do with more unrestricted funding?” and have trouble getting a definite answer; charities often come back with multiple possibilities, asking us which we prefer and offering to submit proposals tailored to our interests.Our impression – both from looking at the grants of major funders and from these conversations with charities – is that charities’ agendas are often partly or fully set by major funders, and thus often reflect the diversity of the different major funders the charities work with, with smaller unrestricted donations serving as support for these diverse agendas. This dynamic makes it difficult to find groups that focus exclusively on a particular proven intervention.

Earlier in our development, we expected the nonprofit sector to look something like the for-profit sector in terms of how organizational strategies and agendas are set. That is, we expected to find that nonprofits usually set their own agendas and seek funders who will support these agendas with relatively limited stipulations and modifications. Instead, we found charities constantly asking what agenda we wanted to fund.

Perhaps this reflects one of the fundamental differences between the two sectors. For-profit investment is what we might call “accountable to profits”: the success of a for-profit investment ultimately depends on whether the company can ultimately turn a profit, and thus it depends on things like the company’s ability to win over customers. By contrast, nonprofit investment is ultimately not accountable to anyone or anything: if a funder sets a poor agenda or fails to support a good one, there are no consequences except what the funder chooses to impose on itself. Nonprofit funders thus have fewer reasons (other than self-imposed ones) to defer to others on agenda-setting and strategizing.

Bottom line: it seems to us that agendas are often set by funders, not charities; looking for charities that have their own predefined agendas limits our options; looking for charities that focus on proven interventions further limits our options; so we have few options unless we expand our scope beyond “charities focused on proven interventions.”

What would our options be if we remained committed to both “passive funding” and “proven cost-effective” interventions?
We could “hold out” for more giving opportunities that meet our original criteria, continuing to recommend the best charities we’ve found while waiting for other charities meeting our criteria to emerge organically (and hoping that our money moved to top charities served as an incentive for this to happen). This would be less work per year than what we’ve done so far (which has involved a lot of exploration, getting up to speed on academic literature, examining many different charities, etc.), so if we went this route we would probably either shrink GiveWell (to perform the same role with minimal resources) or further deepen our due diligence on existing top charities (e.g., perform more site visits).

We’ve also considered looking into areas such as clean water provision and surgery, where we might find giving opportunities that still fit the rubric of “passive” and “proven cost-effective,” but with less strong evidence and/or likely inferior cost-effectiveness. Though we doubt we would find giving opportunities here as strong as our top charities, there could be benefits down the line simply to having more absorptive capacity (i.e., if our money moved continues to grow, we may need more options for donors than what we currently offer.)

In our view, sticking to either of the above approaches would be leaving a lot of potential for impact on the table. We believe that we can broaden our criteria while continuing to bring a level of transparency and public critical reflection that is absent, but badly needed, in today’s nonprofit sector. We believe that this approach may lead to better giving opportunities than those we’ve found so far (even if not by the original criteria), as well as a broader influence on donors and nonprofits. In the future, we’ll be writing more about how we plan to accomplish this.

• Eric Friedman on July 22, 2012 at 12:05 pm said:

I’ve heard nonprofit executives state similar themes that charities have difficulty raising funds without letting donors influence their agendas via restricted gifts, which makes it difficult for the charities to pursue their own agendas. Your post alludes to GiveWell becoming another donor providing restricted funding for its own agenda—potentially a better agenda—for the most cost-effective interventions. But I wonder if this is necessary. There may be some organizations that want to focus on the most cost-effective, proven interventions for the same reasons that GiveWell does, but they have more diverse programs because it is more financially viable.

There is evidence for this because some organizations with highly diverse programs use their unrestricted funding very differently than a proportional allocation across programs. For example, UNICEF discussed how they use unrestricted donations (they use the term “Regular Resources”) in its Report on Regular Resources (http://www.unicef.org/publications/index_59759.html):

“Regular Resources are particularly valuable because they are provided by donors who not only believe in UNICEF’s mandate but who have also chosen to trust that the organization will allocate their funds where they are needed most. This type of funding ensures UNICEF’s independence, neutrality, and role as a trusted partner to national governments. Regular Resources are not driven by any individual donor priorities or agendas. Rather, they are allocated based on a carefully structured formula that takes into account key development indicators, including under-five child mortality, child population, and per capita gross national income. Because such funds are not tied to any country or sectoral programme, they can be used to support those areas that will have the greatest impact on the largest number of children.” (pages 3-4)

Do you think thing it is likely that there are charities with diverse programs that spend their unrestricted resources in a way that does a disproportionate amount of good relative to their restricted resources, potentially rivaling that of GiveWell’s top-rated charities like AMF and SCI that do not have diverse programs? If yes, how can those organizations be identified?

One way might be simply to ask them “What would you do with more unrestricted funding?” Your post mentions that you asked this question to many charities, and most were unable to adequately respond. I’ve had similar experiences, including with UNICEF, even though their report indicates they put a lot of thought into this type of question. They may be unprepared for such a question (and confused by it) because donors rarely ask it, even though they do have a budgeting process in place to send unrestricted donations where they are needed most.

What are your thoughts about GiveWell helping carefully selected charities become more financially independent of donor agendas, rather than providing restricted funds and imposing another donor agenda on them?

• Ian Turner on July 23, 2012 at 6:48 pm said:

First, thanks, Holden, for the retrospective. It’s nice to have a recap on how Givewell arrived where it is today.

In this post, you’ve painted a picture whereby the intervention agenda is set by a few large donors (rather than the charity itself). But I’m not sure how to reconcile this with the financial reality. According to Giving USA, individuals (in 2010) gave $212 billion to charity, accounting for 73% of non-governmental spending. In the same year, non-security foreign aid only totaled to just$29 billion. How does your story match up with these numbers?

Regarding the idea of restricted grants, you (and I) have in the past held the opinion that such a donation constitutes a financial fiction, in the sense that funds are fungible, so your restricted grant to one program may just offset unrestricted funds which can then be allocated to some other program. Has something changed (e.g., new information) to change your mind on this question?

Finally, there seems to be something of a dichotomy here, in the sense that you propose either admitting restricted grants as a candidate for recommendation; or else focusing only on charities with a narrowly-defined program focus. Why does Givewell feel that it is not acceptable to recommend unrestricted grants to charities with diverse programs?

Thanks, as always, for your continuing efforts.

–Ian

• Holden on July 26, 2012 at 11:48 am said:

Eric: thanks for the thoughts. In response to your question, we’d be happy to provide unrestricted funding to charities that are looking to be freed from donor agendas – as long as we believe that the charities will make good use of those funds. We’ve been convinced that this is the case for our top charities, but in general, it seems that the large charities dominated by donor agendas become optimized to fundraising and executing on donor agendas – it’s not clear that they have the capacity (including personnel) for effectively strategizing about how to use unrestricted funding. Our door remains open to cases that fit the vision you described, but we’re feeling that we need to be open to other approaches as well.

Ian:

• Regarding funds from individual vs. major donors:
• Not all individual charity was given to foreign aid. I believe the ratio between government and individual aid is something closer to 1:1, though I haven’t looked up the figures for this comment response.
• There are some organizations that run primarily on individual donations, some that run primarily on support from major donors, and some that operate on a mix. The ones that operate on a mix sometimes seem to allow major donors to set the agenda, while using individual donations to supplement their budgets (in a sense the individual donors are “subsidizing” costs paid by major donors). The ones that run on individual donations don’t generally seem focused on what we consider the most proven interventions.
• A fair amount of our reasoning behind why few organizations seem focused on priority interventions is speculation. What we know is that large organizations seem to be in the mode of “asking us what agenda we’d like” rather than “proposing their own,” and that few organizations are focused on what we consider proven cost-effective interventions. What we’re speculating about is why this is.
• Regarding restricted donations as a “fiction”: I think that if we were talking about a large enough restricted grant, we could have ways of knowing whether our grant was fungible with others’ or not. We could investigate other potential funders and see whether others were willing to come forward; we could fund some proposals and not others and see whether the others got carried out anyway. There’s no way to do this sort of investigation when earmarking a single \$1000 donation (for example). It is still an issue with a larger donation, but the issue seems more tractable. (And to some degree it is an issue even with unrestricted donations: your gift may close a funding gap, thus preventing another funder from filling the gap.)
• We don’t have a problem recommending unrestricted grants to charities with diverse programs, but we need to understand what the result of the marginal dollars will be (which has usually turned out not to be doable in these cases, though there are exceptions such as VillageReach). We would also recommend an unrestricted grant to a charity with diverse programs if we felt that all its work was excellent/that the organization as a whole set priorities better than we could.