In the fourth quarter of 2018, donors gave a combined $7.6 million in funding to GiveWell for making grants at our discretion. In this post, we discuss the process we used to decide how to allocate this $7.6 million, as well as an additional $0.8 million designated for grants at GiveWell’s discretion held by the Centre for Effective Altruism and $1.7 million in the EA Fund for Global Health and Development (which is managed by GiveWell Executive Director Elie Hassenfeld), for a total of $10.1 million in funding. We’re so grateful to have a community of supporters that relies on our work and is open to allowing us to allocate funding to the top charity or charities we believe need it most.
We noted in November 2018 that we would use funds received for making grants at our discretion to fill the next highest priority funding gaps among our top charities. At the time, we wrote:
If we had additional funds to allocate now, the most likely recipient would be Malaria Consortium to scale up its work providing seasonal malaria chemoprevention.
Based on our analysis in 2018 as well as updates we have received from our top charities since that time, we have decided to allocate this $10.1 million in funding to Malaria Consortium’s seasonal malaria chemoprevention (SMC) program. The SMC program consists of treating children with a course of preventive antimalarial drugs during the time of year when malaria transmission is greatest.
We continue to recommend that donors giving to GiveWell choose the option on our donation form for “grants to recommended charities at GiveWell’s discretion” so that we can direct the funding to the top charity or charities with the most pressing funding needs. For donors who prefer to give to a specific charity, we note that if we had additional funds to allocate at this time, we would very likely allocate them to Malaria Consortium’s seasonal malaria chemoprevention program, which we believe could use additional funding for highly cost-effective work, even after receiving the $10.1 million in funding mentioned above.
What Malaria Consortium will do with additional funding
We wrote in detail about Malaria Consortium’s room for additional funding for its SMC program as of November 2018 here. We also spoke with Malaria Consortium for an update in early 2019. Our understanding of what Malaria Consortium will do with additional funding for its SMC program (including this $10.1 million), in order of priority, is as follows:
- Contribute to filling a potential funding gap in Burkina Faso, the existence of which depends on the actions of other funders. If the gap materializes, filling it could require up to $3 million in addition to the $5 million that Malaria Consortium expects to have remaining on hand after what’s currently budgeted for 2019 and 2020.
- Scale up further in Nigeria and Chad in 2020. Our impression is that, given drug production constraints and the length of time needed to plan for the implementation of a campaign, receiving additional funding now rather than in late 2019 (when we plan to make our next recommendation to Good Ventures to fund top charities) increases the likelihood that Malaria Consortium can use the funding for 2020 programs.
- Fund the continuation of programs into 2021. Malaria Consortium has received enough funding to maintain its programs through 2020, but has not allocated funding to maintain programs beyond 2020. To maintain the 2019 program scale in 2021, Malaria Consortium would require an additional $14.8 million in funding, assuming no unbudgeted costs (e.g., additional scale-up) are incurred before then. Our impression is that there is little difference between receiving funding now and in late 2019 in terms of Malaria Consortium’s ability to use it to fund 2021 programs.
Overview of our decision-making process
In early 2019, we checked in with each of our top charities that seemed like plausible recipients of this funding, based on our assessment of their funding needs in late 2018. In general, these check-ins indicated that there weren’t updates in the marginal funding opportunities at our top charities. More details follow in the rest of this post. We refer below to “funding gaps,” which we use to describe the amount of additional funding that we believe could be used effectively (the gap between what charities could use and what they have on hand).
After considering each funding opportunity, we came to believe that the two most promising funding gaps are Malaria Consortium’s for SMC and the Against Malaria Foundation’s. The Against Malaria Foundation (AMF), which distributes insecticide-treated nets to prevent malaria, currently has the opportunity to fund nets in the Democratic Republic of Congo (DRC); we expect a high level of cost-effectiveness for this opportunity due to high malaria rates in DRC.
We discuss the comparison between these two funding opportunities in the next section. We followed the six principles described in this post in deciding between these two opportunities and ultimately decided to grant these funds to Malaria Consortium’s SMC program.
Comparing Malaria Consortium and AMF
What AMF would do with additional funding
In February 2019, AMF told us it had $62.8 million in uncommitted funds, which it plans to commit to a few 2020 net distributions (these are not yet formal commitments—as of February, AMF had not yet signed agreements with government partners to fund these distributions). AMF told us that if it had additional funding at this time, it would allocate those funds toward closing the gap in funding for nets in DRC for 2020. AMF has also shared more detailed information with us about its plans for the funds it holds and its negotiations with country governments; that information is confidential at this time. AMF reports that the total need for funding in DRC for a universal coverage campaign across eight provinces is between $35 million and $45 million.
Comparison using our principles
Principle 1: Put significant weight on our cost-effectiveness estimates.
We estimate that Malaria Consortium’s SMC program and AMF are similar in cost-effectiveness but that AMF is somewhat more cost-effective on the margin.
The most recent version of our published cost-effectiveness model at the time we made this decision (2019 version 2) estimates that Malaria Consortium is 8.5 times as cost-effective as unconditional cash transfers (“8.5x cash” for short) and AMF’s work in DRC is 10x cash (calculated by making a copy of the spreadsheet and selecting DRC in the “Country selection” tab for AMF).
Our best guess of the cost-effectiveness of these two opportunities incorporates several additional adjustments. See this footnote for details.1We adjust for our guess about how factors that are not formally modeled would change the results. For details, see column AB of this spreadsheet, sheet “Consolidated funding gaps.” This adjustment replicates what we did to arrive at our recommendations at the end of 2018. More in this blog post.
For both AMF and Malaria Consortium, we update the country-specific malaria mortality data to be more recent (2017 instead of 2016 figures). For Malaria Consortium, we correct what we believe to be an error in our model (which makes a roughly 5% difference in the final cost-effectiveness estimate), and we have also used an updated method (compared to what we used previously) to account for the fact that the age range of children targeted for SMC differs slightly from the age ranges given in the available age-specific mortality data (3 to 12 months vs. 1 to 12 months). We plan to incorporate these changes into the published model in the future.
For AMF we make several additional adjustments:
– We use DRC-specific cost data and adjustment for insecticide resistance. Our published cost-effectiveness model uses average data for these two parameters when a specific country is selected in the “Country selection” tab.
– We adjust the lifespan of a net downward by 10% for DRC. This is a rough guess based on findings from AMF’s past monitoring in DRC that suggested that nets wore out more quickly than in other locations where AMF has funded nets.
– We use a smaller fungibility adjustment than we do for other countries to capture the lower probability (compared to other countries where AMF operates) that DRC would reallocate funding that it receives from the Global Fund to Fight AIDS, Tuberculosis and Malaria to cover part of the funding gap for nets if AMF did not fund the distribution. Our understanding from conversations with AMF and the Global Fund is that DRC is relatively underfunded by the Global Fund, due to caps on how much it can spend in a single country and DRC’s large malaria burden, and so our guess is that there is less scope for reallocating funds from other malaria interventions to nets.
– We model most marginal funding as going to DRC, with some funding going to other countries. We do so firstly because we believe having additional funding on hand may lead AMF to commit more funding to other countries than it otherwise might, and secondly because of the possibility of AMF deciding not to commit additional funding or to cap the amount it provides to DRC if it has concerns about the quality of the 2019 distributions it is funding in DRC.
– We adjust AMF’s cost-effectiveness downward by 5% to account for the fact we recently learned that AMF has skipped some post-distribution surveys, leading us to update our estimate of potential misappropriation given missing monitoring results. See this spreadsheet.
With these updates, our best guess of the cost-effectiveness of these two opportunities is that additional funding to Malaria Consortium is 8.3x cash and to AMF is 10.0x cash, implying that AMF is 21% more cost-effective.
This estimate has not yet been vetted, so is more likely to contain errors than our published cost-effectiveness model. To enable us to pursue other research work throughout the year, we thoroughly revisit our comparisons between top charities once per year for our annual recommendations refresh in November. When making recommendations at other times of year, we ask ourselves “Have there been any major changes that should lead us to reconsider what we concluded last November?” In this case, we adjusted some of the inputs into our cost-effectiveness model to reflect what we have learned since November and found that the results were broadly similar to our published model. At this level of difference in estimated cost-effectiveness, which is small in relation to the uncertainty in the model, we are inclined to put substantial weight on the other principles discussed below, and particularly on Principle 2.
We are also somewhat concerned that funding AMF may create an incentive for AMF to prioritize less cost-effective spending opportunities over more cost-effective ones, thus reducing AMF’s overall cost-effectiveness in the long run. We estimate that the three other countries AMF is in negotiations with are less cost-effective places to work than DRC. If we were to provide funding to AMF for work in DRC, we could be indicating that a “gaming” strategy—in which an organization tells us that marginal funds would go to a more cost-effective opportunity because its funds on hand have been allocated to less cost-effective opportunities—results in additional funding beyond what it would receive if it allocated funding to more cost-effective opportunities first. We don’t want to create an incentive for organizations to prioritize funding less cost-effective opportunities ahead of more cost-effective ones. We haven’t estimated the potential impact of this factor quantitatively.
Principle 2: Consider additional information about an organization that we have not explicitly modeled.
While we incorporate many subjective factors into our cost-effectiveness models, there are additional costs and benefits that we believe may affect the true cost-effectiveness and that we do not believe are adequately captured by our models. Such uncaptured factors might include, for example: information that charities have and we lack about how to best to allocate funding among different locations; beneficiary experiences with the program that affect how much they benefit from it; and the degree to which charities have indirect impact through conducting research, acting as leaders in their fields, or bringing in new sources of funding.
As we generally do not have the opportunity to observe or measure these costs and benefits directly, we consider them qualitatively through proxies. Such proxies include: our perception of how thoughtfully charities answer our questions; whether they are transparent about mistakes they make; how successful they have been in meeting operational goals (such as hiring, geographic expansion, and instituting new technical systems); whether they conduct and publish research; the frequency of errors in the information they share with us; and whether they meet agreed-upon timelines for sharing information.
We plan to write more about factors that we consider outside of our CEA model in the next few months, as well as assessments of each of our top charities on the proxies we use.
Overall, we assess Malaria Consortium as consistently stronger on the above qualitative proxies than AMF. Both organizations stand out from the vast majority of organizations we have considered for their transparency about both positive and negative results and their track record of collecting information about how their programs are performing. They have both spent a large number of hours over several years (for Malaria Consortium) or over a decade (for AMF) responding to our questions and document requests. This comparison is a relative one, and one that we have not fully justified publicly (but plan to shortly). Based on our experiences working with both organizations, we believe that Malaria Consortium has shown signs of having stronger organizational management.
Principle 3: Assess charities’ funding gaps at the margin, i.e., where they would spend additional funding, where possible.
We’ve accounted for what Malaria Consortium and AMF are likely to do with marginal funding in our cost-effectiveness estimates, above.
Principle 4: Default towards not imposing restrictions on charity spending.
On this principle, there’s no difference between the two opportunities. Funding provided by GiveWell to either program would not be restricted.
Principle 5: Fund on a three-year horizon, unless we are particularly uncertain whether we will want to continue recommending a program in the future.
On this principle, there’s no difference between the two opportunities.
Principle 6: Ensure charities are incentivized to engage with our process.
This principle favors Malaria Consortium, which has consistently provided requested information that aids us in understanding and evaluating their program. AMF has more often been delayed or inconsistent in providing the information we’ve requested.
Other options we decided against (our other six top charities)
Schistosomiasis Control Initiative
The Schistosomiasis Control Initiative (SCI)’s room for additional funding is highly dependent on how much funding it receives from the UK’s Department for International Development (DFID) over the next three years. As of the time we were making this decision, we had not yet received an update on the level of funding that DFID plans to provide. More information is available in our review.
Helen Keller International’s vitamin A supplementation program
Helen Keller International (HKI) told us that it plans to use the funding it has already received for vitamin A supplementation as we expected: to continue its work in Mali, Burkina Faso, Guinea, and Côte d’Ivoire and to restart work in Niger. With additional funding it would prioritize work in:
- Kenya, where it could spend about $2 million over three years.
- Cameroon, where it could spend about $4.2 million over three years.
- Nigeria, where it could spend $0.6 million to conduct a study of the impact of technical assistance work.
- DRC, where it could spend about $9 million to reopen a country office and fund vitamin A supplementation over three years.
In November 2018, we estimated that these opportunities were less cost-effective than Malaria Consortium’s SMC program.2For HKI’s programs, see this spreadsheet, sheet “Consolidated funding gaps,” column AB. For Malaria Consortium’s overall SMC program, see same spreadsheet, sheet “Cost-effectiveness results,” row 6. We did not revisit those calculations as part of the quarterly allocation process.
Evidence Action’s Deworm the World Initiative
Deworm the World has told us that it plans to follow the prioritization laid out in our recommendation to Good Ventures. That prioritization leaves the following opportunities unfunded:
- Extending its funding runway beyond 2020 to 2021.
- Holding sufficient funding for 2020 programming in India that is currently supported by other funders.
- Improving financial stability via increased reserves.
- Expanding to new locations (two states in India and one state in Nigeria).
At the end of 2018, we estimated that these opportunities were 15.0x cash on average; however, that average was largely driven by the opportunity to expand to two new states in India, which is relatively low priority for Deworm the World because it is prioritizing financial stability over further expansion. With that in mind, we prefer to allocate funding to Malaria Consortium.
Sightsavers’ deworming program
Sightsavers indicated to us that it plans to follow the funding priorities it presented in 2018, with the exception of one area where there is no longer room for more funding. Sightsavers has sufficient funding for all remaining opportunities to fund deworming mass drug administration in 2019 that it currently has capacity to implement or that local security issues allow. Sightsavers is currently exploring gaps in deworming provision for other areas in 2020.
END Fund’s deworming program
We didn’t ask the END Fund for an update on its funding needs in early 2019, as we didn’t expect that an update would lead us to allocate discretionary funding to its deworming program. More context for this decision is available here.
GiveDirectly
We didn’t ask GiveDirectly for an update on its funding needs in early 2019, as we didn’t expect that an update would lead us to allocate discretionary funding to its work. More context for this decision is available here.
Notes
| ↑1 | We adjust for our guess about how factors that are not formally modeled would change the results. For details, see column AB of this spreadsheet, sheet “Consolidated funding gaps.” This adjustment replicates what we did to arrive at our recommendations at the end of 2018. More in this blog post. For both AMF and Malaria Consortium, we update the country-specific malaria mortality data to be more recent (2017 instead of 2016 figures). For Malaria Consortium, we correct what we believe to be an error in our model (which makes a roughly 5% difference in the final cost-effectiveness estimate), and we have also used an updated method (compared to what we used previously) to account for the fact that the age range of children targeted for SMC differs slightly from the age ranges given in the available age-specific mortality data (3 to 12 months vs. 1 to 12 months). We plan to incorporate these changes into the published model in the future. For AMF we make several additional adjustments: – We use DRC-specific cost data and adjustment for insecticide resistance. Our published cost-effectiveness model uses average data for these two parameters when a specific country is selected in the “Country selection” tab. – We adjust the lifespan of a net downward by 10% for DRC. This is a rough guess based on findings from AMF’s past monitoring in DRC that suggested that nets wore out more quickly than in other locations where AMF has funded nets. – We use a smaller fungibility adjustment than we do for other countries to capture the lower probability (compared to other countries where AMF operates) that DRC would reallocate funding that it receives from the Global Fund to Fight AIDS, Tuberculosis and Malaria to cover part of the funding gap for nets if AMF did not fund the distribution. Our understanding from conversations with AMF and the Global Fund is that DRC is relatively underfunded by the Global Fund, due to caps on how much it can spend in a single country and DRC’s large malaria burden, and so our guess is that there is less scope for reallocating funds from other malaria interventions to nets. – We model most marginal funding as going to DRC, with some funding going to other countries. We do so firstly because we believe having additional funding on hand may lead AMF to commit more funding to other countries than it otherwise might, and secondly because of the possibility of AMF deciding not to commit additional funding or to cap the amount it provides to DRC if it has concerns about the quality of the 2019 distributions it is funding in DRC. – We adjust AMF’s cost-effectiveness downward by 5% to account for the fact we recently learned that AMF has skipped some post-distribution surveys, leading us to update our estimate of potential misappropriation given missing monitoring results. See this spreadsheet. |
|---|---|
| ↑2 | For HKI’s programs, see this spreadsheet, sheet “Consolidated funding gaps,” column AB. For Malaria Consortium’s overall SMC program, see same spreadsheet, sheet “Cost-effectiveness results,” row 6. |
Comments
Thanks for the update! I’m curious about the “additional $0.8 million designated for grants at GiveWell’s discretion held by the Centre for Effective Altruism”. Is that going to be part of EA Grant but decided by GiveWell rather than CEA?
I appreciate GiveWell framing cost-effectiveness in terms of multiples of cash. It is intuitive and easy to remember. Malaria Consortium’s, AMF’s, and Deworm the World Initiative’s effectiveness are “8.3x”, “10.0x”, and “15.0x” cash, respectively. These cost-effectiveness estimates, however, rely on “extremely limited information and are therefore extremely rough,” using “subjective inputs” and “educated guesses.” As such, use of 2-3 significant digits is likely unwarranted. Is GiveWell truly confident that Deworm the World delivers 15.0x cash rather than 15.1x? If not, sticking to 15x should be sufficient. Similarly, Malaria Consortium’s and AMF’s cost-effectiveness should be expressed as 8x and 10x cash, with the former’s being characterized as “likely better” rather than the much more precise but unwarranted “21%.” GiveWell’s cost-effectiveness estimates play a very important role and its publication is useful and helpful. But using too many significant digits over-claims their precision and is thus misleading (especially when bolded as in this blog) and should be avoided.
Hi Reader, thanks for the question. Donors can donate to GiveWell, our top charities, and some of our standout charities via the Centre for Effective Altruism’s Effective Altruism (EA) Funds (see here). This option enables U.K.-based donors to donate in a way that is tax-advantaged; more information is available here. Donors who give via EA Funds can designate their donations to be allocated to “grants to charities at GiveWell’s discretion;” that pot of funding, which is explicitly designated to be allocated at GiveWell’s discretion, is what this $0.8 million refers to. Those funds are then allocated in the same way as the funding for recommended charities at our discretion that is given directly to GiveWell, rather than through EA Funds.
– Isabel (GiveWell)
Hi Wayne, thank you for the thoughtful feedback. We believe it’s difficult to communicate clearly and accurately about cost-effectiveness and we welcome input on how we could improve going forward.
The fact that our cost-effectiveness estimates are based on limited information, educated guesses, and some subjective inputs limits our confidence in them and leads us to avoid relying heavily on relatively small differences in estimates when making decisions. At the same time, we believe that our best guesses, while rough, do provide meaningful information.
When writing about our cost-effectiveness analysis, we aim to provide a level of precision that captures what we believe to be meaningful differences in figures while avoiding implying that we have a greater degree of confidence in the accuracy of those figures that we actually do. For example, when discussing our estimate of the cost-per-death averted for a given charity, we generally round to the nearest hundred dollars in order to indicate that these are rough estimates rather than exact figures.
In general, we find it useful to discuss variations in cost-effectiveness (measured in multiples of cash) rounded to the nearest tenth-of-a-multiple rather than the nearest multiple. If we were to round to “8x” and “10x,” I believe we would begin to lose some meaningful information. As an illustration, 8x and 10x could be rounded from best guesses of:
1. 8.4x and 9.5x, for an estimated difference of approximately 13% in cost-effectiveness, or
2. 7.5x and 10.4x, for an estimated difference of approximately 39% in cost-effectiveness.
We would likely treat those estimated differences (i.e., 13% and 39%) differently when deciding how to weight qualitative factors against our quantitative estimates, making it useful to discuss these estimates to the nearest tenth-of-a-multiple in order to capture that level of precision. (In contrast, I don’t believe we generally lose useful information by rounding to the nearest tenth-of-a-multiple rather than the nearest hundredth-of-a-multiple.)
Additionally, we choose to use language like “our best guess [is] … that AMF is 21% more cost-effective” rather than more general statements in order to more clearly articulate our view; language like “somewhat better” could be interpreted differently by different people (and “likely better” shares that problem, as well as having a slightly different meaning from what we intend, i.e. it speaks to the likelihood that AMF is more cost-effective than Malaria Consortium’s SMC program rather than addressing our best guess of the magnitude of the difference in cost-effectiveness).
We’ve bolded “our best guess of the cost-effectiveness of these two opportunities is that additional funding to Malaria Consortium is 8.3x cash and to AMF is 10.0x cash, implying that AMF is 21% more cost-effective” in this post to draw attention to the bottom line in a fairly long and detailed discussion; at the same time, I do hear your concern that emphasizing it in that way might indicate that the figures are more accurate at that level of precision than we believe they actually are.
We’ve written about how we use cost-effectiveness analyses in our assessments of charities, and that we don’t view them as highly precise, here. We also have written about how sensitive our cost-effectiveness analyses are to certain inputs here.
I hope this is helpful, and please let me know if you have additional thoughts.
Hi Isabel,
Thank you for taking the time to respond to my suggestion that GiveWell use only one significant digit in its communications, and not two or three. Note that one significant digit implies that only differences with magnitudes of 10% ~ 100% are significant (10 vs 9 and 2 vs 1, respectively) while two significant digits implies differences of 1% ~ 10% are significant. This delineation is consistent with your email response saying that expressing cash multiples with only 1 significant digit is too rough for the purposes of your blog since GiveWell “would likely treat [differences of 13% and 39%] differently when deciding how to weight qualitative factors against our quantitative estimates.” The links you provided in your response are very helpful, but I found them to be supportive of my suggestion.
GiveWell’s Jun-2017 blog states that “when comparing charities’ relative cost-effectiveness, we look for differences of 2-3x or more. If we find a difference of less than 2-3x, we feel unsure whether such a difference truly exists.” Note that 2-3x translates to differences of 200% ~ 300%. GiveWell’s Dec-2017 blog summarizes key uncertainties for each intervention. Let’s avoid the largest uncertainties arising from deworming or inter-outcome translations and focus only at malaria interventions. AMF has uncertainties in the range of 20% ~ 30% (net use adjustment; relative efficacy of ITNs) while the Malaria Consortium’s are in the range of 20% ~ 25% (external validity adjustment; ratio of malaria mortality to incidence). These are lower bounds since the final estimate’s uncertainty should be at least as large as individual input’s (assuming no negative correlations across inputs). In short, these past blogs suggest no basis for the use of two significant digits or more since differences of 10% are simply not meaningful. For the purpose of decision-making, reliance on too much precision is dangerous because it uses GiveWell’s cost-effectiveness model in a way the model is not designed for. For the purpose of communication, it is simply inappropriate and misleading by conveying precision where none exists.
In your response, you may have implied that the “best guess” phrasing used in your blog justifies the presence of decimals. I believe this defense inappropriately conflates risk with uncertainty. There is a difference between a “best guess” of 10.0 and a “best guess” of 10 even when both may have the same expected values. The latter conveys uncertainty that may not be distillable into probabilities or distributions. This distinction is important because one ought to respond to risk and uncertainty differently. One can quantify risk explicitly, but uncertainty may be better dealt with through additional research or through more intuition-based approaches. The June 2017 blog concludes by wondering how GiveWell can better communicate its model uncertainties. I believe consistent use of one significant digit is an easy win-win by properly conveying the appropriate level of uncertainty directly and by doing so not with more complexity but actually, with greater simplicity.
Hi Wayne,
Thank you for your response; we really appreciate the engagement.
The way we think about differences in cost-effectiveness varies by the nature of the project. In the June 2017 blog post, we’re writing specifically about considering new top charities, though the wording at the top of the post is broader, so it’s somewhat ambiguous. Later in the post, we write:
At that stage of our research process, we would be looking for programs that seem likely to be robustly more cost-effective than cash transfers. Our initial cost-effectiveness estimates are less well-developed than our published cost-effectiveness analysis of our top charities and therefore more uncertain.
We believe that our estimates of the cost-effectiveness of AMF and Malaria Consortium are developed enough that it’s useful to discuss them rounded to the nearest tenth-of-a-multiple for the reasons I wrote about above, namely that we would treat best guesses of e.g. 8.4x and 9.5x differently than best guesses of e.g. 7.5x and 10.4x. We trust our comparison of AMF and Malaria Consortium more than we’d trust a comparison of a program we’re relatively unfamiliar with and one of our top charities for two main reasons:
1. We’re comparing one malaria prevention program to another, so some of the considerations that go into the model are more similar than they would otherwise be. (You note the challenges of inter-outcome comparisons above.)
2. We’ve put many hours into developing and revising our cost-effectiveness estimates of both AMF and Malaria Consortium, so we’re more willing to rely on them than we would otherwise be.
In summary, while our estimates are uncertain, we believe the number of significant figures we use in this post enables us to best explain our decision-making. We do see your concern that we may inadvertently convey that our cost-effectiveness estimates are more precise than they are.
We’d also be happy to continue this conversation on a call, if you’d like. Please let us know!
Best,
Isabel
Hi Isabel, thanks so much for taking the time to respond, and I promise this is my last comment. I believe I’ve made clear my case about the uncertainty around the malaria interventions’ cost-effectiveness so won’t continue discussing it.
I do at least hope you’d reconsider the use of 3 significant digits for deworming’s cost-effectiveness, which you say is “15.0x cash” in your original blog. GiveWell’s Deworming Intervention report states that “cost-effectiveness estimates such as these should not be taken literally, due to the significant uncertainty around them.” A blog post dated Jan-2017 states that “GiveWell discounts [deworming] by some 99% out of doubts about generalizability.” Indeed, these discounts are based entirely on GiveWell employees’ subjective inputs and are thus largely arbitrary. A cost-effectiveness range between 0x – 100x cash for deworming would not be too broad since it would merely reflect minor adjustments to the 99% discount. This makes the use of 3 significant digits or even 2 (when GiveWell rounds to the nearest hundred dollars for cost-per-death) highly misleading and inappropriate.
If GiveWell is serious about conveying its model estimates’ uncertainty, a consistent policy on the appropriate use of significant digits would be great. Thank you for all the good work that you and GiveWell do. I am a big fan!
I agree with everything Wayne said. Being overly precise in these estimates actually detracts from GiveWell’s credibility by about 0.323%.
Hi Wayne, thanks again for laying out & expanding on your concerns. We appreciate your perspective and look forward to keeping it in mind as we communicate about our cost-effectiveness analysis going forward.
All the best,
Isabel
Comments are closed.