This year, we re-evaluated the cost effectiveness of direct cash transfers as implemented by our friends at GiveDirectly. Our complete writeup is here, and full of fascinating details, but the main headline is: we now estimate that GiveDirectly’s flagship cash program is 3 to 4 times more cost-effective than we’d previously estimated.
It is important to note two things: (1) this won’t alter our Top Charities list or our grantmaking—we believe that the programs we currently direct funding to are at least twice as cost-effective as this new estimate, so we don’t expect to support GiveDirectly’s flagship program in the near term; and (2) this update is the result of re-evaluating the evidence underpinning GiveDirectly’s program, which we hadn’t formally done since 2019—the structure of GiveDirectly’s program has not changed (though they are now carrying it out in more locations since our last evaluation).
We share more information about our research below. You can read our full, detailed report here. You can read GiveDirectly’s blog post on our re-evaluation here.
Refresher: What is GiveDirectly’s flagship program?
GiveDirectly’s Cash for Poverty Relief program sends one-time cash transfers of approximately $1,000 via mobile money platforms to households living in poor regions of Kenya, Malawi, Mozambique, Rwanda, and Uganda. These transfers are unconditional: there are no strings attached, and only very basic eligibility requirements.1The basic eligibility requirements are: (1) recipients must permanently reside in the targeted village, (2) recipients must not have received a GiveDirectly transfer previously, and (3) at least one member of the household must be over 18.
What is driving GiveWell’s updated estimate?
There are four primary drivers contributing to increased cost-effectiveness, the most impactful of which is the increased consumption accruing to non-recipients (“spillovers”). The table below shows (in order of importance) how much each driver impacted our bottom-line estimates across the locations of GiveDirectly’s flagship program:
Kenya | Malawi | Mozambique | Rwanda | Uganda | |
---|---|---|---|---|---|
Previous best guess2We use GiveDirectly’s unconditional cash transfers as a benchmark for comparing the cost-effectiveness of different funding opportunities, and had set the value of our estimate of cash transfers at 1. | 1.0 | ||||
Previous + spillovers update3We calculate spillover benefits as a percentage of recipient consumption benefits. In this table, the figures provided for spillover benefit updates were calculated based on recipient consumption benefits that incorporate our updated estimates of baseline consumption. | 1.8 | 2.0 | 1.9 | 1.9 | 1.8 |
Above + mortality update | 2.1 | 2.4 | 2.5 | 2.2 | 2.2 |
Above + baseline consumption update | 2.2 | 3.1 | 3.0 | 2.6 | 2.4 |
Above + long-run consumption update | 2.4 | 3.5 | 3.4 | 3 | 2.6 |
TOTAL (+ supplementary benefits4Our cost-effectiveness analysis includes a number of additional benefits and downward adjustments, such as reduced morbidity and risk of wastage, that we have opted not to explicitly model; instead, we incorporate them as rough best guesses and add them here. We do not discuss the supplementary benefits in this post; see this section of our full report for more information. ) | 2.6 | 3.8 | 3.7 | 3.3 | 2.8 |
You can access our full model of GiveDirectly’s flagship program here.
Driver 1: Spillovers to individuals who don’t receive the cash transfers
Spillovers are essentially the effects that “spill over” to non-recipients. These could include both positive and negative effects.
Positive spillovers
As a simplified example, imagine a farmer receives a cash transfer. She might now be able to pay a local miller to mill more of her grain, causing economic benefit for the mill owner as well. The mill owner might now be able to buy more eggs from the farmer, causing continued economic benefit for the farmer. This increase in economic activity can benefit households that don’t receive cash, and hence is a positive spillover.
Negative spillovers
Conversely, these kinds of benefits could be counteracted by inflation. Returning to our example, if the mill was already working at full capacity, the owner might simply increase her milling prices. This could offset any consumption gains, such that some households that didn’t receive transfers were made worse-off (a negative spillover).
To assess the evidence on spillovers from the GiveDirectly program, we reassessed the evidence on both GiveDirectly and non-GiveDirectly cash transfer programs and spoke to six external academic experts.5We spoke to the authors of the Egger et al. paper, as well as others who have worked on academic studies of cash transfer programs in low- and middle-income countries: Berk Ozler, Craig McIntosh, Rossa O’Keeffe O’Donovan, Jesse Cunha, and Eeshani Kandpal. Our best guess is that there are positive consumption spillovers from the GiveDirectly program—as nearby non-recipients benefit from the uptick in local economic activity—and that these positive consumption spillovers are about 60% to 70% as large as the direct consumption benefits to recipients.
This update is partly informed by Egger et al. 2022, a new paper that was published since we last evaluated the GiveDirectly program. The study on which the paper is based distributed cash to 10,500 households in Western Kenya between 2014 and 2017. Egger et al. found that every $1 injected into these communities generated $2.50 in economic activity, and that around 70% of this economic activity stemmed from positive consumption spillovers to non-recipients. It also found minimal (0.1%) price inflation.
We think that Egger et al. 2022 is a high-quality study whose findings held up under a reanalysis we commissioned. However, when assessing the impact of a program, we try to take into account all available evidence as well as the opinions of relevant experts. While we place substantial weight on the results of Egger et al. 2022, we continue to place some weight on other studies because:
- The spillovers implied by this paper are much larger than those found in four other studies of the GiveDirectly program, which found no spillovers or mildly negative spillovers. While we think Egger et al. 2022 has important methodological advantages over these other studies, we don’t think it entirely supersedes them, so we retain some weight on these more pessimistic findings.
- The results are relatively imprecise. Compared to the evidence that underpins our Top Charity recommendations, Egger et al.’s multiplier results are relatively statistically imprecise.6Egger et al. 2022 finds an expenditure multiplier of 2.6 and an income multiplier of 2.5. When they do a joint statistical test against a null hypothesis that the economic multiplier was less than or equal to 1 (what we’d expect if there were no spillovers), they reject this hypothesis at the 10% but not 5% or 1% statistical significance level. These multiplier estimates seem relatively less precise than the effect sizes that underpin our Top Charity recommendations, which are typically statistically significant at the 5% or 1% level. We think we should put less weight on more imprecise results.
- The study was conducted in an unusually dense and well-connected setting, and we’d expect offsetting effects like inflation to be more likely in more remote contexts. There’s suggestive evidence of this from two evaluations of non-GiveDirectly cash transfer programs in Mexico and the Philippines, which found more concentrated inflation in more remote markets.
- Academic experts we spoke to thought we shouldn’t use Egger et al.’s findings as our sole data source. While everyone we spoke to agreed the study was high quality, none thought we should update our estimate entirely based on those findings. Some experts think we’re being too conservative in our interpretation of these results, while others think we’re being too generous.
Driver 2: Mortality effects
Preliminary results from a separate paper sent to us by Egger et al. team imply that all-cause mortality for children under five is reduced by 46% as a result of GiveDirectly’s cash transfer program. This is an enormous effect size, the likes of which we rarely see in studies of other health interventions.
After looking at this evidence, we adjusted it downward by 50%, implying that GiveDirectly reduces all-cause mortality for children under five by 23%. The main reasons for this are:
- These are preliminary results that are yet to be scrutinized.
- We don’t have a clear understanding of the mechanism that could drive this effect. The authors hypothesize that it could be the result of behavior change leading to increased use of antenatal and postnatal care.
- This finding seems surprising given basic sense checks. If we take this result at face value, it implies that under-five mortality rates were 33% lower in GiveDirectly households than in average Kenyan households, despite GiveDirectly households still being meaningfully poorer.
Even if we take the 46% mortality reduction at face value, this effect increases the cost-effectiveness by less than 1x. This is because GiveDirectly’s program is relatively expensive. It provides around $1,000 per household and is not targeted only to households with children under the age of five. As you can see in the example below, the implied cost-per-life-saved of GiveDirectly’s program is not competitive with GiveWell’s Top Charities.
- GiveDirectly transfers around $1,000 per household and has around 20% in additional overhead costs. So: $1,000,000 covers approximately 800 households.7With a transfer amount of approximately $1,000, factoring in around 20% in overhead costs, the cost per household is approximately $1,200. $1,000,000 / $1,200 = 833 households.
- Each household has approximately 4.3 members, 13% of whom are children under five. Thus, $1,000,000 targets around 450 children.8833 households x 4.3 household members = 3,582 recipients. 13% of 3,440 = 466 children overall.
- Not all of these children would die in the absence of GiveDirectly. Assuming a 5% all-cause mortality rate for children under five means about 23 would die in the absence of the program.95% of 466 = 23 children.
If we assume the GiveDirectly program leads to a 46% reduction in all-cause mortality among children under five, then $1,000,000 in funding to GiveDirectly would avert around 10 deaths of young children. This corresponds to a cost per life saved of $100,000. This is significantly more than our Top Charities, which save a life for between $3,000 and $5,500.
Driver 3: Baseline consumption update
An important input in our model is baseline consumption before households receive cash. This is because we think there is diminishing marginal utility of consumption. That is, all else equal, we think giving $1,000 to someone making $1,000 per year has a greater impact than giving $1,000 to someone making $2,000 per year.
Our previous model assumed that baseline consumption in Malawi, Mozambique, Rwanda, and Uganda was similar to consumption levels we had calculated in Kenya. Now, we believe that consumption in those newer locations is lower than in Kenya.
We arrived at updated estimates in the table below using:
- Baseline consumption estimates in the control groups of studies of GiveDirectly’s program in Malawi, Rwanda, Uganda, and Kenya.
- Consumption estimates from government surveys, specifically bottom-quintile households, since these most likely reflect typical GiveDirectly recipients.
Kenya | Malawi | Mozambique | Rwanda | Uganda | |
---|---|---|---|---|---|
Best-guess of annual consumption of GiveDirectly recipients (PPP 2017) | $652 | $470 | $450 | $533 | $626 |
Implied consumption per day (PPP 2017) | $1.79 | $1.29 | $1.23 | $1.46 | $1.71 |
Using these inputs, the program looks more cost-effective because the average recipient of GiveDirectly’s program is poorer than in our previous estimate, which only assessed consumption in Kenya.
Driver 4: Long-run consumption benefits to individuals receiving the cash transfers
We also updated our estimate of how much and how long recipients enjoy the consumption benefits of cash transfers. Previously, we assumed a large spike in consumption in year one (likely spent on things like home improvements, livestock, etc.), but then assumed that these consumption gains would taper off quickly after that, with a small additional spike in year ten because we assumed people would at some point sell the assets they had purchased with their cash transfer.
We reviewed existing studies of the long-run effects of unconditional cash transfer programs, alongside new, unpublished evidence from a five to seven year follow-up of the experiment studied in Egger et al. 2022. Our view is that the published evidence suggests that consumption gains gradually dissipate across time, though at a slower rate than we’ve previously assumed. This stands at odds with the unpublished findings we’ve been sent, which suggest almost no fade-out in consumption gains across five to seven years. We don’t put much weight on these findings at the moment because they differ from most existing evidence and are yet to be publicly scrutinized in the peer-review process. Our current view is that there are large consumption gains in the first year followed by a gradual fade-out.
How we could be wrong
We are uncertain about many aspects of this update. Reasonable people could easily disagree about a number of the inputs we settled on. We consulted with other experts, some of whom view our update as too generous and others as too conservative. A summary of our key uncertainties, in rough order of importance, are:
- How big are consumption spillovers to non-recipients? (more)
- How likely are these spillovers to generalize to different contexts? (more)
- How persistent are consumption gains to recipients? (more)
- How much should we value raising consumption vs. saving lives? (more)
- How are consumption gains distributed across households? (more)
- How poor is the average household in villages GiveDirectly targets? (more)
Next steps
This update is not the end of the story. Egger et al. are continuing their research, and we anticipate a follow-up in the next few years about the persistence of the transfer effects in years five to nine; they are also studying the effect of transfers in Malawi, where they will explore whether the non-inflationary effects generalize to another setting. GiveWell will be keeping a close eye on these developments, and (as always) will update our estimates in light of new information.
For now, this update doesn’t affect our immediate funding recommendations because GiveDirectly’s flagship program does not meet our current cost-effectiveness threshold, although we’ll continue looking into whether versions of the GiveDirectly program could be above our bar. We think GiveDirectly is a great organization and look forward to continuing our ongoing dialogue with them.
GiveWell supporters might note that this update has an additional implication. We use cash transfers as the benchmark for our cost-effectiveness estimates and define other programs in multiples of that benchmark. What does it mean for us if we now think cash transfers are significantly more cost-effective than we previously thought? This is a great question, and we want to give it the time and attention it deserves. Thus, we will be using our historic benchmark until we have thought it through. For now, you can think of our benchmark as “GiveWell’s pre-2024 estimate of the impacts of cash transfers in Kenya,” with GiveDirectly’s current programs in various countries coming in at 3 to 4 times as cost-effective as that benchmark.
We look forward to continuing to update our models and estimates as we seek out the most cost-effective programs to help people around the world.
Notes
↑1 | The basic eligibility requirements are: (1) recipients must permanently reside in the targeted village, (2) recipients must not have received a GiveDirectly transfer previously, and (3) at least one member of the household must be over 18. |
---|---|
↑2 | We use GiveDirectly’s unconditional cash transfers as a benchmark for comparing the cost-effectiveness of different funding opportunities, and had set the value of our estimate of cash transfers at 1. |
↑3 | We calculate spillover benefits as a percentage of recipient consumption benefits. In this table, the figures provided for spillover benefit updates were calculated based on recipient consumption benefits that incorporate our updated estimates of baseline consumption. |
↑4 | Our cost-effectiveness analysis includes a number of additional benefits and downward adjustments, such as reduced morbidity and risk of wastage, that we have opted not to explicitly model; instead, we incorporate them as rough best guesses and add them here. We do not discuss the supplementary benefits in this post; see this section of our full report for more information. |
↑5 | We spoke to the authors of the Egger et al. paper, as well as others who have worked on academic studies of cash transfer programs in low- and middle-income countries: Berk Ozler, Craig McIntosh, Rossa O’Keeffe O’Donovan, Jesse Cunha, and Eeshani Kandpal. |
↑6 | Egger et al. 2022 finds an expenditure multiplier of 2.6 and an income multiplier of 2.5. When they do a joint statistical test against a null hypothesis that the economic multiplier was less than or equal to 1 (what we’d expect if there were no spillovers), they reject this hypothesis at the 10% but not 5% or 1% statistical significance level. These multiplier estimates seem relatively less precise than the effect sizes that underpin our Top Charity recommendations, which are typically statistically significant at the 5% or 1% level. |
↑7 | With a transfer amount of approximately $1,000, factoring in around 20% in overhead costs, the cost per household is approximately $1,200. $1,000,000 / $1,200 = 833 households. |
↑8 | 833 households x 4.3 household members = 3,582 recipients. 13% of 3,440 = 466 children overall. |
↑9 | 5% of 466 = 23 children. |
Comments
Thank you for this detailed re-evaluation and the updated estimates on spillovers and other factors contributing to cost-effectiveness.
I do wonder, though, whether comparing GiveDirectly’s program to GiveWell’s top charities—estimated to be “~30-40% as cost-effective as our marginal funding opportunity”—might be a bit too gracious, given some methodological differences in evaluating impact. Specifically, it seems that GiveWell’s assessments of top charities don’t account for the same type of non-recipient spillovers when assessing cost-effectiveness. For instance, saving a life through vaccination not only preserves income but may also enable spending that generates broader economic benefits. These effects seem similar in spirit to the positive consumption spillovers identified in GiveDirectly’s program but don’t appear to be factored into all evaluations of health-focused interventions.
Maybe I missed something but would appreciate your thoughts!
Hi Jonas,
The question of how these spillover effects generalize to other income effects (from both health and other livelihoods programs) is something we’re interested in and will hopefully get to next year.
Our intuition is not to expect as large income spillovers from health-improving programs. Part of what’s driving the large spillovers of GiveDirectly are entire villages receiving large consumption shocks ~simultaneously, which helps kickstart local economic activity. We don’t see a similar mechanism at play with e.g. the future income effects of SMC, as the effects are smaller and more dispersed across the population. We’re hoping to interrogate this intuition next year by speaking with external experts, including the authors of the GiveDirectly spillover work.
Thank you, Chandler. That makes a lot of sense, and I’m sure you folks will do a deep dive into this. Just to share my two cents: Having a constant benchmark across time and settings might be a very intuitive and practical approach.
For instance, you could introduce something like a “cash value” metric, one that deliberately excludes spillovers. Under this framework, you could say that GiveDirectly’s program delivers 2.5x cash value, while your top charities might deliver 10x cash value. This would be easy to understand and keep the multipliers close to what you already have.
Best,
Jonas
Thanks for this idea, Jonas! I’ve shared it with our Research team for their consideration as we dive in further – we really appreciate you engaging with our work!