The GiveWell Blog

Update on our work on outreach

GiveWell’s impact is a function of the quality of our research and the amount of money we direct to our recommended charities (our “money moved”). Historically, we’ve focused mostly on research because we felt that the quality of our recommendations was a greater constraint to our impact than our money moved.

This has changed. Outreach is now a major organizational priority. The goal of this work is to increase the amount of money we direct to our top-recommended charities.

In April 2014 I wrote about our work on outreach to explain why we hadn’t prioritized it: in brief, our growth had largely been driven by inbound interest in GiveWell, and proactive outreach efforts (beyond building relationships with existing donors) hadn’t yielded results that were worth the cost.

What changed?

  • We believe that the amount of money we move is now a greater constraint to our impact than additional improvements in the quality of our research. Over the last two years, we’ve added five new top charities (three of which implement programs that weren’t previously represented on our top charities list), and we expect that our top charities, collectively, will have more than $200 million in unfilled funding gaps once they’ve received the funding that we expect to direct to them. (This calculation excludes GiveDirectly, which we believe could absorb and distribute hundreds of millions of dollars.) At the same time, the quality of our research and our capacity for research is higher than it’s ever been, so the returns to adding staff there (in terms of the pace at which we identify significantly better giving opportunities) are now lower.
  • Increased capacity for outreach. In our 2014 post, we wrote that one of our key constraints was that senior staff (which at the time meant primarily GiveWell Co-Founder Holden Karnofsky and me) were necessary for most outreach-related work. This has changed. We now have capacity to take on outreach work as other staff have been hired and trained on this type of work.
  • Better information on the impact of GiveWell’s outreach. We now have better information about the returns to outreach because:
    1. We’ve collected better data (via an improved donations processing system and outreach efforts) about where donors find out about us. Because of our ability to track donors, we know that a single appearance on NPR or major podcasts tends to drive $50,000+ in annual donations.
    2. More time passing has demonstrated that the lifetime value of the donations of a first time donor is higher than we expected. In several cases, we’ve seen major donors (i.e., those giving $10,000-$100,000) increase their annual giving by a factor of 10 or more.

We’re in the early stages of figuring out how we can proactively invest time and money in outreach to significantly increase our money moved. For now, we’ve taken some opportunities that we think will have positive returns; these are the three that we’ve invested the most time and money in to date:

  • Podcast advertising. We’ve been advertising on podcasts that we believe our target audience listens to, based on interviews with current donors and GiveWell staff. In February and March, we ran a small experiment with a few ads on FiveThirtyEight’s Politics podcast and Vox’s The Weeds.1We’ve also been running ads on Julia Galef’s Rationally Speaking podcast since then. Because it’s much smaller and more targeted, we’ve excluded it from this analysis. Measured returns to advertising on Rationally Speaking have been significantly better than the more mainstream podcasts discussed in this post.

    In total, we spent approximately $20,000 on ads for this initial experiment. We ask donors who give via our website to tell us where they learned about GiveWell when they donate. GiveWell received approximately $8,000 in donations between February 1 and November 20 from donors who reported that they had learned about us via these podcasts.

    The donations we received were from first-time donors; to assess the impact of our advertising, we need to estimate the lifetime value of acquiring a new donor. In work we’ve done to assess our retention rate, we’ve seen that (a) approximately 20-25% of the donors who make a first-time donation of less than $1,000 give again in the subsequent year but (b) because many first-time donors increase the size of their donation over time, collectively, the donors who recur give more than 100% of the value of what they give in their first year.

    At higher donation levels ($1,000-$100,000), we measure 40-45% retention among donors, which leads to retention of approximately two-thirds of dollars given.2I say “measure” retention because we’ve learned that many donors give subsequent donations directly to our top charities and don’t report those donations to us. We’ve tried to follow up with lapsed donors and with charities to track these donors down.

    We therefore estimated the net present value of expected future donations (over the next five years) from these podcasts ads as somewhere between approximately $20,000 (assuming two-thirds dollar retention for the first two years and 100% dollar retention subsequently) and $45,000 (assuming 100% dollar retention).3We only projected donations over five years. This is fairly arbitrary because we don’t have long-term enough data to know whether or not this is a reasonable assumption. We capped it to prevent our assessment being driven by speculation about how much money would be donated many years in the future.

    A few additional facts are worth keeping in mind about the above figures:

    • We ran this experiment in February and March; most donors give at the end of the calendar year. We consistently see donors who find out about GiveWell during the course of the year, but donate in December. Other things equal, we expect that our advertising would have had greater measured returns in December than earlier in the year.
    • We are only able to track donors who (a) fill out our donation form telling us where they learned about us and (b) give directly through our website rather than to our top charities. Less than 50% of donors who give via credit card (and a smaller percentage of donors who give via check) tell us where they learned about GiveWell. Also, roughly speaking, approximately 50% of the donors and dollars we influence come through GiveWell rather than going to our top charities.4I took this rough estimate from footnote 26, on page 15, of GiveWell’s 2015 metrics report.
    • It’s certainly possible that donors who learn about us via podcast would be more likely to give through our website than an average donor, more likely to report on how they found us (since their source is clear), or less likely to be retained. My best guess is that donors who learn about us via podcast ads behave similarly to our other donors, but I won’t be surprised if they don’t.

    With all that in mind, I believe that the impact of our podcast advertising is higher than what we directly measured.

    The results we saw from February to November this year were promising enough that we decided to increase the size of our experiment by spending approximately $100,000 on podcast ads. We’re currently running ads on FiveThirtyEight’s Politics podcast and Ezra Klein’s podcast and The Weeds at Vox.

    Added January 25, 2018: Here’s an update on the impact of podcast advertising, incorporating data through Monday, January 22nd:

    As mentioned above, we spent approximately $100,000 on podcast ads during giving season (November and December 2017). Using the same methodology described above, we tracked $89,551 to our recommended charities as a direct result of these ads between November 20th and January 22nd. We estimate these donors as having a five-year, net present value of $250k-$410k.

    (These numbers have all the same caveats described above.)

  • Earned media outreach. Mentions of GiveWell in the media have historically been a strong driver of growth. We aimed to increase mentions of GiveWell in high-quality, high-profile media where we’ve had the most past success as measured by dollars donated (i.e., media like The New York Times, NPR, The Wall Street Journal, and Financial Times). We retained a PR firm that came strongly recommended; we also increased 1-to-1 outreach by GiveWell staff to members of the media who have covered GiveWell in the past. It’s very hard to attribute the impact of the additional effort we’ve invested—overall, our effort has been fairly limited, and it’s hard to easily draw the causal lines between our work and the stories that appear—but my guess is that our increased efforts have led to more coverage of GiveWell and our top charities this giving season than in the recent past.
  • Website improvements. Companies that sell products online invest significant effort into optimizing their websites and checkout pages to maximize their revenues. We retained a marketing consultant, Will Wong of Mission Street, and we’ve been A/B testing different donation pages and plan to test other pages on our website such as our homepage or top charities page to see whether we can increase our conversion rate (i.e., the percentage of visitors to our website who give to one of our top charities). For context, our current conversion rate is 1%. Our understanding is that a standard conversion rate for e-commerce companies is 2%, and that international nonprofits have a similar conversion rate.5See Pg 51 of the study downloadable here. An increase in our conversion rate to the industry average would lead to a significant increase in the amount of money we direct to our top charities.

Notes

Notes
1 We’ve also been running ads on Julia Galef’s Rationally Speaking podcast since then. Because it’s much smaller and more targeted, we’ve excluded it from this analysis. Measured returns to advertising on Rationally Speaking have been significantly better than the more mainstream podcasts discussed in this post.
2 I say “measure” retention because we’ve learned that many donors give subsequent donations directly to our top charities and don’t report those donations to us. We’ve tried to follow up with lapsed donors and with charities to track these donors down.
3 We only projected donations over five years. This is fairly arbitrary because we don’t have long-term enough data to know whether or not this is a reasonable assumption. We capped it to prevent our assessment being driven by speculation about how much money would be donated many years in the future.
4 I took this rough estimate from footnote 26, on page 15, of GiveWell’s 2015 metrics report.
5 See Pg 51 of the study downloadable here.

Comments

  • Caitlin Tulloch on December 19, 2017 at 6:18 pm said:

    Do you have any data on whether new donors who found you through the podcast ads were donating to charity for the first time, versus changing their donation from other (presumably less effective) charities towards those recommended by Givewell? As it currently stands, the implicit cost-benefit being done here either (1) assumes there is no displacement, and this is all donors who wouldn’t have donated to any charity otherwise, or (2) assumes zero “value” to donations to NGOs other than those recommended by Givewell. Given the evidence I would absolutely believe that the ratio of “value” for Givewell vs. non-Givewell charities is greater than 1, but curious to see if there is backing for this assumption. Thanks, as always, for documenting your methods! This is a fantastic amount of rigor going into organizational decisions, and always inspiring and motivating to see.

  • Appreciate the level of openness and analysis! This makes sense to my intuition based on the simple fact that most people have never heard of you, and you’re also something of an outlier as far as giving opportunities. So outreach moves a lot of novelty / information to a lot of people. Whereas further research may have diminishing returns in terms of how likely it is you’ll find new and confident even higher multipliers above GiveDirectly.

    But enough about my intuition. What I wonder is of you’d find it effective to invest further in research and outreach about opportunities you believe are clearly not the highest impact. If they may bring donors in who wouldn’t be here at all otherwise — and may be higher impact than those donors’ other opportunities (or than non-giving). And perhaps “gateway drugs” into more impact-seeking.

    You already do some of this, I know, but it may make sense to align more of it with more outreach.

  • Elie (GiveWell) on December 20, 2017 at 10:49 am said:

    Hi Caitlin,

    Thanks for the question!

    You are right that we are making the simplifying (and incorrect) assumption in this post that non-GiveWell-recommended charities are having no impact; we will keep that fact in mind when thinking about how much to spend on advertising.

    We don’t have data about where donors sourced from podcast ads would have otherwise given, but we survey donors to GiveWell charities who have given more than $2,000 in one year to our recommended charities. Collectively, these donors give more than 90% of our money moved (even excluding donations from Good Ventures, a large foundation with which we work closely).

    On pages 12 and 13 of our 2015 metrics report (our most recent), we summarize responses from these donors about where they said they would have given in GiveWell’s absence. There are a lot of caveats about this data — it’s self reported, we only got responses from ~20% of the donor-dollars we surveyed — but it’s the best we have. In brief, 56% of the money moved came from donors who said they reallocated from other charities to GiveWell charities. Of that 56%, 66% said that they reallocated from other international charities and 26% said they reallocated from a mix of international and domestic organizations.

  • Thank you as always for your openness and commitment to tracking data on all aspects of GW’s work. The information here is very interesting (and I will take a bit of time to ruminate on it) but the overall philosophy of self-evaluation and transparency is so commendable. Thanks, y’all.

  • Elie (GiveWell) on December 20, 2017 at 7:20 pm said:

    Hi eub,

    Yes, it’s definitely something we’ve considered. We did some of this in GiveWell’s early years and didn’t see much demand from our donors for it then (more in the linked post).

    Ultimately, it’s a question of capacity and prioritization. We’re capacity constrained in our core focus area of finding the best opportunities we can and directing as much money as possible to them. At this point, the effort devoted to investigating less-promising-seeming opportunities would trade off directly against more progress on finding and moving money to the opportunities we think are best.

  • Catherine (GiveWell) on January 25, 2018 at 2:24 pm said:

    Note: We published an update in the post above on the impact of podcast advertising, incorporating data through Monday, January 22nd.

Comments are closed.