We’re extremely excited to be announcing the Change Our Mind Contest to encourage critiques of our cost-effectiveness analyses that could lead to substantial improvements of our overall allocation of funds. For all the details, see this page.
Cost-effectiveness is the single most important input in our decisions about what programs to recommend, and we believe it’s possible that we’re missing important considerations or making mistakes that lead us to allocate funding suboptimally. We’ve been excited to see people engaging with our cost-effectiveness analyses, and we’d like to inspire more of that engagement.
With that in mind, we’re inviting you to identify potentially important mistakes or weaknesses in our existing cost-effectiveness analyses and tell us about them!
The first-place winning entry will receive $20,000, the runner-up will receive $10,000, and the honorable mention will receive $5,000. We may offer multiple runner-up and honorable mention prizes if the quality of submissions is particularly high. All other entries that meet our criteria will receive a participation prize of $500, capped at a total of 50 participation prizes for the first 50 submissions.
In addition to the monetary prizes, excellent entries may lead to changes in how we allocate millions of dollars of funding, leading to more lives saved or improved.
Entries must be received by October 31, 2022, and the requirements are described in detail on the contest page. We will announce winners by December 15, 2022, and will publish the winning entries online.
If you have any questions, please leave a comment on this post or email change-our-mind@givewell.org.
We’re running this contest because the recommendation decisions we make are extremely important, and we want to incentivize feedback that will improve our work, thereby enabling us to do more good. We hope you’ll consider participating!
Comments
In current page :
“Cost-effectiveness is the single most important input in our decisions about what programs to recommend”
In http://www.givewell.org/how-we-work/our-criteria/cost-effectiveness :
“we give estimated cost-effectiveness only limited weight in recommending charities”
Hi, x,
Thanks for bringing this to our attention! The “Cost-Effectiveness” page, though it got some limited updates in August of this year, is a bit out of date and doesn’t do a good job of conveying how we currently use cost-effectiveness in our recommendations and grant decisions. We’re now working on some improvements to the wording on that page.
In short, it’s true that cost-effectiveness is the single most important input into our decision-making, though it’s not the only thing we consider. Saying cost-effectiveness carries “limited weight” isn’t exactly wrong, but it doesn’t accurately express how important it is to us.
Our goal is to maximize impact with the finite amount of funding we have (particularly this year), so we use our minimum cost-effectiveness threshold (currently 10 times as cost-effective as unconditional cash transfers, or “10x cash”) as the starting point to determine what programs we’ll fund. Generally, if our estimates tell us that a program clearly doesn’t meet that bar, we won’t consider it for immediate funding, although we might think about funding it in the future. If a program comes close to meeting the bar, but doesn’t quite get there, other factors could bend us toward funding it anyway. Those factors could include strength of the implementing organization, or the desire to retain the option of future funding—e.g., we might fund operational research on a program to gather more information about it, or we might give “bridge funding” to a program to keep it going while we refine our cost-effectiveness model.
We do try to incorporate as many relevant inputs as we can think of into our cost-effectiveness analyses, including things like “value of information” (example here), which tries to account for the knowledge we think we’ll get from funding research. That kind of input is necessarily very subjective, as are our moral weights, adjustments for leverage and funging, and other factors. So, although we put a lot of work into our cost-effectiveness estimates, they do have serious limitations, and we use them mostly for purposes of comparing to other grants we have made or might make.
The above describes how we use cost-effectiveness estimates in all of our grant decisions. When deciding whether to name a program a top charity (and thus cast a wide net for donations to support it), we consider other criteria, such as the organization’s track record and whether we’ve directed significant funding to it in the past, in addition to whether it has funding opportunities that meet our cost-effectiveness bar.
I hope that’s helpful!
Best,
Miranda
Many of the charities you currently recommend are very incremental in nature (the medicine costs X per patient and there is little overhead). Which is fine and good, but the cost benefit analysis may preclude other charities that require larger upfront costs. A budget impact analysis, if I understand it correctly, could help project the cost/benefit of a future project (including considerations like the costs/savings of displacing the status quo, etc…). There is more information on it in this report: https://www.ispor.org/heor-resources/good-practices/article/principles-of-good-practice-for-budget-impact-analysis
Hi, Luke,
Thank you for your comment and the recommendation!
We don’t have a philosophical preference for programs with low start-up costs, but it is true that our current top charities generally have low start-up costs compared with their ongoing costs. We’re not sure whether this might represent a bias in our process that tends to eliminate more programs with different cost profiles. One suggestion is that larger start-up costs could make a program less amenable to study by randomized controlled trial, which is the type of evidence we focus on seeking out when evaluating programs.
There are some programs that we’ve directed funding to with higher start-up costs. For instance, Dispensers for Safe Water and in-line chlorination tend to have relatively high start-up costs. We are comfortable with the idea of spreading out start-up costs over many years of benefit, although it can be tricky to estimate how many years of benefit we should attribute to the program at the outset.
We also fund research into programs to help fill in gaps in our knowledge (and in the general body of knowledge around an intervention), and so help us make better decisions about what to prioritize and fund. Research can be a significant up-front cost that we think will yield benefits over many years. Some examples of research grants we’ve recommended recently include this grant to fund an evaluation of mobile conditional cash transfers and this grant to plan a follow-up study on deworming. Additionally, we’ve recommended grants for work that’s aimed at changing public health policy (examples here and here), which represent up-front investments that we believe will result in benefits over many years.
I hope that’s helpful!
Best,
Miranda
Comments are closed.