We’re about to launch another outreach experiment: mailing letters.
This giving season, we’re planning to send a subset of approximately 4,500 GiveWell donors a physical letter encouraging them to renew or increase their support of GiveWell’s top charities. We’ve never done broad outreach to encourage donations through physical mail before.
In the letter, we plan to share our list of top charities, our overall recommendation for donors, and remit materials for individuals who feel compelled to give. We hope the letter helps communicate our research to a group of individuals who have found our work useful in the past to guide their giving. We plan to assess the success of this experiment based on incremental donations that result from the letters.
Why we’re reaching out via mail
Increasing the amount of funding we direct to our recommended charities is an important goal for GiveWell. To help us achieve this, over the past two years, we’ve prioritized expanding our organizational capacity dedicated to communicating about our top charities to donors and potential donors.
However, our retention of donors isn’t as high as we want it to be; 43% of donors who gave in 2017 also gave in 2018. In addition, we plan to increase our focus next year on identifying new donors, and so it may be particularly important to make progress now on the best ways to build an ongoing relationship with donors.
We’re trying new modes of communication, including more traditional nonprofit outreach to donors, because we think they might help us increase our retention rate and the amount of funding our top charities receive. Reaching out to donors via mail is a common strategy nonprofits use to solicit donations.
We expect the donations we receive to exceed the amount we spend on this campaign. We plan to spend $15,000 (including a conservative estimate of staff time) on the letters and we model the potential return to be around $100,000 in new donations. Details are in our model here.
Some of our donors may prefer not to receive mail at all. To avoid reaching them, we won’t send physical letters to donors who have opted out of receiving solicitations from us in previous surveys. (If you’re reading this now and are unsure which preferences you’ve shared with us, we’re happy for you to update your preferences by emailing us at email@example.com.)
Plans for assessing success
We plan to assess success based on the incremental donations made due to the campaign. We’ll look at how much donors who received the letters gave relative to the previous year, compared to a smaller control group of donors who did not receive the letters. If the return on our spending on this campaign is near or better than our projection, we’d consider it a success and would want to weigh that against any serious negative feedback we might receive in a decision on whether or not to do this again in the future.
We would also be open to repeating the direct mail experiment if incremental donations made in 2019 that we attribute to the campaign roughly equal the amount we spent to conduct it. Many GiveWell donors support our top charities over multiple years and we expect some returns from the 2019 campaign in future years. In addition, we anticipate that the GiveWell staff time required to organize a campaign would be lower in subsequent years, as we could build off of the work we did this year to set it up.
Thanks for experimenting with us! We’re excited to try new ways to engage. Please let us know what you think in the comments.
If you send me physical mail, it will probably sit unsorted on the coffee table for like a week, where I’ll likely point it out to my roommate and visitors will see it. So, design for that use case too!
Will all of the letters be identical, or will you test a handful of different letter designs for comparative effectiveness?
Due to the relatively limited size of our test, we’re only planning to use one letter design in 2019. They key variable we’ll be testing is whether donors who receive letters give more than those who do not.
If we send additional mail in the future, we would expect to consider testing the effectiveness of different letter designs.
Any concern about your environmental footprint, no matter how small? Paperless communication is now the norm, to say nothing of the required fossil fuel if opting for delivery. Is there really no other way?
I am not saying that I am the model of environmental consciousness, but shouldn’t you strive to be?
Bill, I imagine they think the good done by the donations these letters generate — above and beyond what emails would accomplish — will be vastly greater than any harm done by using 4,500 pieces of paper and transporting ~50kg worth of letters around on a truck (i.e. 4,500 11-gram letters).
I would be shocked if they weren’t right about that.
I wouldn’t let e.g. ten children die of malaria to avoid printing on 4,500 pieces of paper. That’s in the ballpark of the trade-off they face.
A portion of that drop off will be donors who in 2018 gave to a GiveWell recommended charity, but didn’t let GW know (or let the charity know that they were motivated by GW). Do you know what the statistic is for the subset of donors who gave for distribution at GW’s discretion? (That group should be less vulnerable to this type of measurement error.)
To be clear, I do think this experiment is a great idea. A potential challenge in evaluating it will be that it may lead to a combination of more donations and better attribution (i.e. it will likely cause some people to let GW know about gifts they were going to make anyway). So to disentangle this, I’d suggest looking separately at the subset who gave at GW’s discretion in the prior year (who should be less subject to attribution error).
It’s a good question!
For this experiment, we’re only looking at data from donors who give through GiveWell, either to specific top charities or to “Grants to recommended charities at GiveWell’s discretion.” We are excluding direct-to-charity data because it is a less reliable tracking method. It is possible, and perhaps likely, that our true retention (if we include donors who give directly to charities, or who have switched between giving through GiveWell and giving to charities) is higher than 43%, but we are holding ourselves to a higher bar for this experiment so that we can get a clearer signal.
I would like to know the history and background of Givewell. I agree very much with evidence-backed decisions, and I would like to learn more about the origins of Givewell.org. (I’m already on your email list).
Thanks for asking! You can learn more about our story here. Our Giving Effectively page provides a brief overview of how we work and this page contains answers to commonly asked questions about GiveWell. I hope this helps!
That’s a high bar in terms of levels, but might be a lower bar in terms of changes. The concern is that the letter from GiveWell might prompt some people to donate via GW who otherwise would have given directly to the charity. That would be measured as an improvement in retention, but really would be better described as an improvement in attribution.
You’re absolutely right; that could be an effect of receiving the letter, which would confound the results.
That said, we would caution against viewing this as a negative outcome. In fact, this would be a positive update from our perspective—not from the perspective of accurate data in the experiment, but in terms of communications and maximizing the impact of future donations.
We prefer that donors (who aren’t constrained by tax-deductibility or other factors) give to our top charities through GiveWell. This enables us to keep donors who give based on our research informed of important updates in our analysis and understanding, which may impact our prioritization of individual top charities on our list. We appreciate having the ability to communicate directly with donors and this is the most straightforward mechanism for ensuring that happens. (Of course, donors may give to our top charities and agree to share their information with us, but we would guess that doesn’t happen in all cases.)
It totally makes sense that improved attribution and communication have value (I didn’t mean to suggest otherwise), but I still think that analytically you want to try to disentangle that from increased actual (as opposed to measured) money moved.
It’s worth noting that attribution and communication have value to the charities also. So to the extent it’s possible, I’d think the ideal is to get donor permission to share info (if they gave via GW, to share with the charity and if they gave directly to share with GW). So that’s what you ideally want to achieve.
To the extent we’re talking about a shift from e.g. AMF (and only AMF) getting the donor info to GW (and only GW) getting the info, it’s not obvious to me that’s on balance a positive outcome: that seems like a complicated question with a lot of competing considerations.
I agree! I didn’t mean to imply that we wouldn’t want to disentangle them, just that the outcome of more donors giving through GiveWell is a positive one for us. We’ll absolutely need to take that possibility into consideration when we assess the results of this experiment.
We do ask donors for permission to share their info with charities (and charities do the same for us). So, we fully agree that if donors are willing to share with both us and the charities they support through GiveWell, that is the ideal outcome from a communication and attribution perspective, though of course we (and our top charities) want to respect donors’ wishes around confidentiality when they are expressed.
Comments are closed.