# The track record of policy-oriented philanthropy

[Added August 27, 2014: GiveWell Labs is now known as the Open Philanthropy Project.]

As noted previously, I’ve been working on improving our broad understanding of the role that philanthropy can play in influencing and informing policy. One of the questions I’ve been focused on is “What is the track record of policy-oriented philanthropy?” Specifically:

• Are there compelling cases in which a major policy change can be partially or fully credited to philanthropic efforts?
• How often have philanthropic efforts succeeded in bringing about change? Often enough to imply a good “return on investment?”

I’ve concluded that answering these questions reasonably well could require an enormous, long-term effort. This is true both because the questions are inherently difficult to answer – many of philanthropy’s claimed impacts on politics are highly long-term and diffuse, such that it is difficult to confidently isolate impact – and because there has been little academic work on the subject.

Below, I first list salient examples I’ve seen in which philanthropy is believed (by some) to have had an important impact on public policy. I think the nature of these examples illustrates some of the challenges with isolating the impact of philanthropy. I then discuss our understanding of the current state of the literature on this topic, and what we would do to make more progress. For the purposes of this post only, “success” is defined as causally impacting public policy, not as having positive social impact, since the goal is to determine how and when philanthropy has been effective in changing policy (the question of how policy should be changed is a different question).

Potential cases of philanthropic impact on public policy

Many of the claimed success stories of policy-oriented philanthropy involve long-term (sometimes 10+ years), diffuse impacts. Pages 4-8 of The Elusive Craft of Evaluating Advocacy, by Steven Teles and Mark Schmitt (both of whom we’ve spoken with – see conversations notes for Teles and Schmitt) give a good feel for this kind of story, and cover some of the cases we’ve frequently heard reference to: the role of Atlantic Philanthropies and others in laying the groundwork for the Affordable Care Act, the role of (presumably philanthropy-backed) church and student groups in “claw[ing] back the application of the death penalty,” the repeal of the “Don’t Ask, Don’t Tell” policy, and impact of the conservative Federalist Society, which is claimed to have magnified the impact of other organizations on law. We excerpt the part about health care reform, as an example:

The effort that culminated in 2010 was the work of decades, including a previous, high-profile failure in the early 1990s, multiple waves of state-based reform and numerous incremental efforts at the national level. Advocates invested hundreds of millions of dollars on initiatives ranging from media initiatives such as encouraging television producers to include stories of the uninsured, multiple coalition-building projects, university- and think tank-based research and well-funded grassroots initiatives.2 The basic outlines of reform policies were worked out well in advance, in advocacy groups and think tanks, who delivered a workable plan to presidential candidates. Key interest groups who could block reform, such as small business, had been part of foundation-supported roundtables seeking common ground for years. Technical problems had been worked out. And tens of millions of dollars had been set aside as long ago as 2007 for politically savvy grassroots advocacy initiatives targeted at key legislators. After a very long slog, the outcome was the Affordable Care Act.

Another very long-term, diffuse impact that has been cited to us is the case of the Robert Wood Johnson Foundation’s efforts to reduce tobacco use.

• Activities included funding research on the impact of policy on tobacco use (page 13), funding early-career researchers in the hopes of developing a stronger set of professionals focused on the issue (page 17), creating the Campaign for Tobacco-Free Kids (page 17), and indirect support for state-level lobbying (page 20).
• The program started in 1991 and had reached ~$20 million per year by 1993 (page 12), but its goals – changing policy to be more discouraging of tobacco use and ultimately reducing smoking rates, particularly among minors – don’t seem to have shown much progress until after 1996 (see pages 36-38 for policy change indicators, page 26 for smoking rate data). • Given the number of different initiatives RWJF created and invested in, the number of states it worked in, and the long-term nature of some of its investments (building organizations, funding research and researchers), evaluating the extent to which its funding can be “credited” for reductions in tobacco use could be an extremely challenging project. • The Foundation spent ~$700 million in this area over time (page 12); if one assumes it reduced the number of smokers by 10 million (its evaluation estimates 5.3-14.2 million, page 28), then the total cost of the program was ~$70 per “person stopped from smoking,” which would compare favorably to our estimate of ~$80 per life-year gained for our top charity if smoking causes one to lose multiple years of life. The point of this calculation isn’t to praise the program – we haven’t vetted these figures and realize there are a substantial number of questions here – but to illustrate that the sort of claims made about policy impact imply potential competitiveness with our top charities.

We have also seen claims of shorter-term, more tangible impact of funders and nonprofits on public policy, though usually at smaller scale. Change Philanthropy goes through several such cases, including a movement in which lawsuits were brought against states alleging that they needed to invest more in education to comply with their own constitutions. The Center for Lobbying in the Public Interest also lists smaller-scale success stories. I’ve heard multiple informal claims that online advocacy groups had tangible, extremely rapid impact on halting (a) passage of the Stop Online Piracy Act (b) construction of the Keystone XL pipeline.

Assessing the ROI of policy-oriented philanthropy

While we’ve seen many claims of impact, none appear easy to assess. Moreover, the most prominent and largest-scale claimed impacts generally appear the hardest to evaluate.

We’ve asked multiple people (including Frank Baumgartner, Mark Schmitt and Steven Teles) whether there is a literature that might help us in this endeavor. It appears that we aren’t going to find any work systematically looking across a large number of philanthropic efforts to change policy, such that we could get an idea of aggregate return-on-investment. We may be able to find case studies on specific policy changes.

We don’t see a clear or easy path to assessing the question, “Can policy-oriented philanthropy have a high enough probability of success to make the cost-effectiveness competitive with our top charities?” We expect our History of Philanthropy project to generate better information on particular, high-profile claimed successes (perhaps via reviewing existing case studies, perhaps via generating new ones focused on the role of philanthropy). If it does, we may consider a more systematic project aiming to catalogue and investigate the policy-oriented work of a representative set of foundations. But it could be a long time before we have useful estimates of the “good accomplished per dollar” of policy-oriented philanthropy, and in the meantime, we will be moving forward on exploring good opportunities within policy-oriented philanthropy.

Reason to be optimistic about ROI

One rough, heuristic-based reason that I’m optimistic about policy-oriented philanthropy (in terms of “impact per dollar”) is that it generally seems like a given amount of money “means more” in the context of policy-oriented work than in other contexts.

There are many reasons that this comparison could be unfair or irrelevant, but it syncs with my general impression that the amount of money it takes to be a “significant player” is smaller in policy-oriented work than in direct aid or scientific research.

• Kris Zyp on November 6, 2013 at 1:32 pm said:

I have really appreciate this exploration. My research has led me to make policy-oriented philanthropy a signficant part of my giving, and I have enjoyed your analysis and perspective. A couple more reasons that policy-oriented philanthropy seems compelling to me:
* Lobbying seems to have an excellent ROI in other industries. This is evidenced both through market behavior, as corporations consistently set aside money for lobbying, and by direct study, for example: http://papers.ssrn.com/sol3/papers.cfm?abstract_id=1375082. While there is likely to be variation between industries, it seems the high degree of success in other industries strongly suggest it should have a good ROI in the aid industry (or other policy areas).
* I know you have mentioned that other foundations often do great research, but lack transparency. Their giving is indicative though, and from what I can tell, just about every organization focused on aid advocacy, has a high percentager of funding from foundations, with the Gates foundation usually at or near the top.

• Joe D. on November 6, 2013 at 2:05 pm said:

The author is correct to suggest that ROI may not be comparable between policy-oriented philanthropy and GiveWell’s top charities, both because the investment is not comparable and because the return is not comparable.

Investment:
In the context of GiveWell’s top charities, the initial investment is measured in terms of the donor’s dollars. For policy-oriented philanthropy, the measure of the initial investment should include not only the donor’s dollars but also the dollars of taxpayers that are used to fund the policy change, giving extra weight to taxes paid by the poorest taxpayers, as well as adding some equivalent of the moral cost of imposing the donor’s will on the taxpayers or those forced to obey new laws or regulations.

Return:
GiveWell’s top charities generally measure return in terms of lives saved. Policy-oriented philanthropy would likely have numerous competing measures of the success that the author has called positive social impact. For example, how does one measure the positive social impact of banning the death penalty? Should I count each death row inmate whose sentence is commuted to life-in-prison as a live saved in the same way I count a child saved from death by malaria as a life saved? What kind of life is life in prison? Or how do I measure the social impact of stopping the Keystone pipeline? How many lives did it save? Some other measure of social impact may be needed for each different policy change pursued.

For these reasons, ROI between policy-oriented philanthropy and GiveWell’s top charities are not comparable.

• Kris Zyp on November 6, 2013 at 3:39 pm said:

Joe, I would agree with all your points about investment and return. However, while these factors may be difficult to measure, they are certainly are not impossible to assess and estimate. CDC and others do research on the impact of climate change in terms of human deaths (although it has high levels uncertainty). The negative social impact of taxes can also be estimated reasonably well, the marginal impact of after-tax income changes on health/longevity in America is something that is relatively easy to analyze (even accounting for varying income levels). The investment cost, (based on Givewell’s research on domestic vs international causes) as a deduction from funding something like the global fund, seems likely to reduce efficiency by less than 10%, probably closer to 1%. Estimations can be made, and consequently, concluding that policy-oriented approaches are incomparable is non-sequitur.

These challenges do add to the uncertainties, but they are probably relatively small uncertainties in comparison to the rather large uncertainty of the impact of lobbying/advocacy itself, as Holden noted. The uncertainties in this area are undoubtedly large, but again, that doesn’t preclude rational comparison. As a donor, I can compare opportunities that involve both varying levels of impact as well as varying levels of certainty in the impact. Likewise, as an investor, I can compare my CD against my stocks, as long as I am willing to account for both varying expected yield and risk levels. From what I can tell, policy-oriented approaches might be kind of the venture capital of philanthropy, high-risk, but possibly high-return.

• Holden on November 7, 2013 at 9:47 pm said:

Thanks for the comments, Kris and Joe. A couple of thoughts …

• I agree that both of Kris’s points potentially weigh in favor of policy-oriented philanthropy – my impression is that both corporations and major foundations see policy advocacy as worth the investment, and I think in both cases (though for somewhat different reasons) this carries some weight.
• We don’t expect to make apples-to-apples comparisons between policy-oriented philanthropy and other activities. We do expect to find the best giving opportunities we can (probably under a range of different big-picture assumptions about some of the key uncertainties) and share the reasoning behind our recommendations.
• Aaron Dorfman on November 8, 2013 at 4:00 pm said:

There has already been an enormous, long-term effort to explore this question. NCRP spent three years and approximately $1 million to study the benefits of foundation funding for policy advocacy, community organizing and civic engagement. We documented 400 policy wins across seven sites — red states, blue states, urban, rural, etc. — and found a return on investment of$115 to \$1. The main page for the project is http://www.ncrp.org/gcip and the summary report — all the findings boiled down into ten easy to read pages — is called “Leveraging Limited Dollars.”

• You mention data for annual expenditure in global health, scientific research, and policy advocacy, and then suggest that the relatively low annual expenditure in policy advocacy is some reason to explore this field further.

I seem to remember that in earlier blog posts you posit an efficient market hypothesis for philanthropic foundations, which implies that the marginal return on investment in each of the three fields – global health, scientific research, and policy advocacy – should be the same.

So, the data on the relative annual expenditures in each of these three fields would be irrelevant for making a educated guess of which area has the best marginal return on investment.

What am I missing here?

• Chris Jenkins on November 12, 2013 at 12:37 am said:

Off-topic, but topical:

After the disaster in the Philippines, having seen photographs and read stories about the immense suffering taking place there, there’s an overwhelming psychological urge to divert money (which would otherwise be spent on charities known to be effective) toward organizations working on the relief effort. Effective altruism requires a long-run approach, of course, so it may be important to resist the emotional pull of such events.

It would be nice to hear from Givewell at a time like this– why should one, or shouldn’t one, shift donations toward disaster relief now?

• Ian Turner on November 12, 2013 at 10:17 pm said:

Chris:

You probably want to see here:
http://www.givewell.org/international/disaster-relief
And here:
http://blog.givewell.org/category/disaster-relief/
And especially here:
http://blog.givewell.org/2008/08/29/the-case-against-disaster-relief/

From that last one: “When a natural disaster and humanitarian crisis hits the headlines, many of us (including me) reach straight for our wallets. Emergencies have an easier time getting our attention (and emotional investment) than the chronic health problems that plague the developing world every day. But to hear the Disease Control Priorities report tell it, emergency aid is one of the worst uses of donations, despite being one of the most emotionally compelling.”

• Aaron – thanks! I briefly glanced at this report, and it looks quite useful. I wasn’t able to find information about the details of any of the cases – just a list of cases in which nonprofits are believed to have been effective – so let me know if I missed it. Either way, it’s good to have a list of believed impacts.

David, our post on broad market efficiency did not claim that the broad market is efficient. Rather, it laid out the concept of broad market efficiency, as a topic for exploration. I believe that some underestimate the efficiency of the broad market, but I also believe it is very far from “perfectly efficient”; I certainly wouldn’t assume that the marginal returns to all giving opportunities are equal (if I did believe that I wouldn’t be working on GiveWell).

• Aaron Dorfman on November 25, 2013 at 1:13 pm said:

Holden: Details are found in each of the seven reports that make up the series. Each study is about 65 pages in length, and each is accessible from that home page for the project. Seeing your comment, it would be great if we had also worked backwards and created links from the impacts directory to the section of each study where that win was discussed, but we didn’t do that.

• Great, thanks!

One thing I’d still like to see is more on the case for causal impact. The report I looked at states, “NCRP research staff verified the impacts to ensure that the dollar amounts and number of beneficiaries estimated by groups, as well as the groups’ role in the wins, were accurate. NCRP consulted with public officials, researchers and other experts and examined source materials such as newspaper articles and state budgets.” with a footnote stating, “Detailed verification and quantification methodology is available upon request.” Would it be relatively convenient to share more on this front (particularly regarding groups’ role in the wins)? If so, we might do a review of this report and use it as evidence on the track record of policy-oriented philanthropy (as well as a model for other investigations we might consider). It does appear to be the most systematic approach to this that I’ve seen.