We’ve recently made a number of adjustments to improve our research process. Not all of them are easily visible outside of the organization.
This post is to highlight one of them: Publishing more frequent updates to our cost-effectiveness model throughout the year.
This post will explain:
We recently completed a small project to determine whether using subnational baseline malaria mortality estimates would make a difference to our estimates of the cost-effectiveness of two of our top charities, the Against Malaria Foundation and Malaria Consortium. We ultimately decided not to include these adjustments because they added complexity to our models and would require frequent updating, while only making a small difference (a 3-4% improvement) to our bottom line.
Though this post is on a fairly narrow topic, we believe this example illustrates the principles we use to make decisions about what to include in our cost-effectiveness model.
Many charities aim to influence how others (other donors, governments, or the private sector) allocate their funds. We call this influence on others “leverage.” Expenditure on a program can also crowd out funding that would otherwise have come from other sources. We call this “funging” (from “fungibility”).
In GiveWell’s early years, we didn’t account for leverage in our cost-effectiveness analysis; we counted all costs of an intervention equally, no matter who paid for them.((For example, see row 3 of our 2013 cost-effectiveness analysis for Against Malaria Foundation.)) For example, for the Schistosomiasis Control Initiative (SCI), a charity that treats intestinal parasites (deworming), we counted both drug and delivery costs, even when the drugs were donated. We did this because we felt it was the simplest approach, least prone to significant error or manipulation.
Over the last few years, our approach has evolved, and we made some adjustments for leverage and funging to our cost-effectiveness analyses where we felt they were clearly warranted.
In our top charities update at the end of 2017, we made a major change to how we dealt with the question of leverage by incorporating explicit, formal leverage estimates for every charity we recommend.
This change made our cost-effectiveness estimates of deworming charities (which typically leverage substantial government funding) look more cost-effective than our previous method. For example, our new method makes SCI look 1.2x more cost-effective than in the previous cost-effectiveness update. More details are in the table at the end of this post.
We also think the change makes our reasoning more transparent and more consistent across organizations.
In this post, we:
Details follow.
When our cost-effectiveness analysis finds robust and meaningful differences between charities, it plays a large role in our recommendations (more on the role it plays in this post).
But while our cost-effectiveness analysis represents our best guess, it’s also subject to substantial uncertainty; some of its results are a function of highly debatable, difficult-to-estimate inputs.
Sometimes these inputs are largely subjective, such as the moral weight we assign to charities achieving different good outcomes (e.g. improving health vs. increasing income). But even objective inputs are uncertain; a key input for anti-malaria interventions is malaria mortality, but the Institute for Health Metrics and Evaluation estimates 1.6 times more people died in Africa from malaria in 2016 (641,000) than the World Health Organization does (407,000; pg. 41).((Differences in their methodologies have been discussed, with older figures, in a 2012 blog post by the Center for Global Development.))
Before we finalized the charity recommendations we released in November, we determined how sensitive our results were to some of our most uncertain parameters.
In a previous blog post, we described how we use cost-effectiveness analyses when deciding which charities to recommend to donors.
Today, we published a report that discusses how GiveWell and other actors, such as governments and global health organizations, approach one of the most subjective and uncertain inputs into cost-effectiveness analyses: how to morally value different good outcomes.
Our cost-effectiveness analysis plays a critical role in the recommendations we make to donors. For example, as a direct result of our cost-effectiveness calculations, we place a higher priority on filling funding gaps at the charities we recommend that work on deworming programs and distributing malaria nets than we do directing funding to GiveDirectly, a GiveWell top charity that distributes direct cash transfers. We believe that GiveDirectly is the strongest organization we’ve ever seen, but according to our analysis, cash transfers are less cost-effective in terms of impact per dollar donated than deworming treatments and malaria nets.
Accordingly, cost-effectiveness analysis is a major part of GiveWell’s research process. We dedicate a large part of a full-time staff member (Christian Smith)’s capacity to this work and others involved with GiveWell research spend a considerable amount of time engaging with our cost-effectiveness model throughout the year. We consider this analysis a key part of our output and publish our model online so that anyone can check our calculations, enter their own inputs, and see if they agree with our approach and outputs.
This post will provide some basic information about how our cost-effectiveness analyses inform our charity recommendations.