Before publishing this post, I sent a draft to Alan Fenwick, Director of the Schistosomiasis Control Initiative who asked colleagues of his to comment. We asked each person for their permission to post their comments, and we’ve posted those for which we received permission here:
- Dr. Dan Colley’s comments (PDF). Dr. Colley is the Director of the Center for Tropical and Emerging Global Diseases at the University of Georgia.
- Dr. Alaa Hamed’s comments (PDF). Dr. Hamed is Senior Health Specialist at The World Bank office in Cairo.
- Dr. Lorenzo Savioli’s comments (PDF). Dr. Savioli is the Director of the Neglected Tropical Diseases Department at the World Health Organization.
The Schistosomiasis Control Initiative (SCI) is an outstanding giving opportunity compared to nearly every other option out there. It’s a challenge of the work GiveWell does that we have to communicate about differences between outstanding giving opportunities because these differences matter to us (in deciding where we would give) and to our audience.
This post explains why I think the gap between SCI and our top two charities is substantial. I have enough reservations about SCI that (uniquely among GiveWell staff members,) I did not allocate any of my personal giving to it last giving season. (I gave 75% of my gift to AMF and 25% to GiveDirectly.)
While my thoughts have been alluded to in already-public content (see our discussion of the relative merits of our top three charities as well as our review of SCI), conversations with donors have given me the continuing sense that the weight I, personally, put on these considerations hasn’t been made fully clear. I think it’s important to do so, and that’s the intent of this post. I think SCI is an outstanding giving opportunity in the scheme of things, but I want to be as clear as I can be about how I think about giving to them, in the spirit of transparency and open dialogue about the best giving opportunities. These views do not represent a change in GiveWell’s ranking or suggested allocation.
My position is not a function of doubts about the strength of the evidence for deworming or SCI’s track record. Deworming is an outstanding intervention, and I am on board with the analysis we’ve published about its relative cost-effectiveness. SCI has an impressive track record. As far as we can tell, it has repeatedly been involved in large-scale, successful deworming programs.
So why did I decide to give to other organizations instead of SCI?
Though I, personally, have spent tens of hours speaking with Professor Fenwick and other SCI staff and reviewing SCI documents (and other GiveWell staff have spent hundreds of hours speaking with Professor Fenwick and analyzing SCI’s documents) over the past 4 years, I still do not have a concrete, specific understanding of how SCI has allocated funds and its specific value added. My understanding could be summarized in the following way: deworming is an outstanding program; SCI is involved in deworming programs; the programs with which it has been involved with in the past have had good results; it requests additional funds.
In general, I feel that I’ve experienced a strong pattern in which uncovering new information about an organization or intervention (which I previously understood only at a superficial level) tends to lower rather than raise my confidence in it. As a result, I’ve started to adjust my confidence downward for organizations that I understand less well, where I have questions about how they work or spend money.
Good examples of this dynamic are organizations GiveWell rated highly earlier in its history but no longer recommends. Although in some cases, the change in ranking was due to a change in GiveWell’s approach, in most cases, continued analysis of a charity led to new information that shifted our view about the likely impact of their program.
The fact that I still have a relatively limited understanding of SCI’s use of funds (a) contrasts with our other two top organizations, and makes me relatively more concerned about SCI’s overall capabilities as an organization (for an example of the sort of thing I’m concerned about, see this exchange); (b) leads me to believe there’s a higher probability that we’ve missed important information about SCI that would lower our confidence if we had it.
The experience of having important unanswered questions also applies to following SCI’s progress since we gave it a top ranking. We have now recommended SCI for over a year (and have been carefully following it since mid-2009), and I feel that we’ve learned relatively little about its progress in that time. (See our updates on SCI.)
This experience is also relevant because I see limited opportunity to learn from SCI in the future, which undermines the argument we’ve given for supporting multiple charities. In my view, learning should be a key goal of giving, especially to organizations that are not #1. While the rest of GiveWell staff are not fully on board with the points I’m raising in this post, they do agree that thinking about our prospects for learning from SCI in the future will play an important role in deciding whether we should aim to direct more funding to it in the future.
While I’m not able to pin down more specific concerns about SCI – i.e., I’m concerned because I haven’t been able to answer important questions and in the past answering previously unanswered questions has led organizations to move off our top-rated list – I have some theories about what we might be missing.
Although SCI has an impressive track record, it’s worth noting that its major achievements have been in partnership with major funders (Gates Foundation, USAID, DFID, Geneva Global) and it is not clear to me how large a role these funders played and how much credit they deserve for SCI’s past successes. (See the relevant section of our review of SCI.) One can easily imagine a model in which countries agree to work with SCI and implement a deworming program largely because a major funder is behind the program. This could simply be because the funder can commit all the funding a program needs (so the country knows the program will move forward) or because a major funder exerts its influence over a country to convince it to implement a program.
SCI is now using unrestricted funding to try to start major programs without the backing of a major funder. Its largest use of unrestricted funding to date is in attempts to start a deworming program in Ethiopia. SCI is attempting this on its own, with fewer financial resources and diminished non-financial major funder support relative to what it had in its past successes.
To me, the best argument for supporting SCI is that (a) deworming is an excellent intervention; (b) SCI is a large, long-standing, credible organization that focuses on deworming and has no red flags. However, I also think this line of argument might apply to many other organizations that we’ve looked into briefly but stopped investigating because we found it challenging to get sufficient information about their track records or had questions about their room for more funding. These organizations include Deworm the World, the Center for Neglected Tropical Diseases, the African Programme for Onchocerciasis Control, the Measles and Rubella Initiative, and UNICEF’s Maternal and Neonatal Tetanus program.
The same was true about SCI when we first approached it in 2009, but because of the nature of our research process at that time, we were more willing to spend significant time trying to convince an organization to share information with us. SCI shared a significant amount of information with us about its past activities, but my intuition is that were we to spend the type of time on these other organizations, we would ultimately reach a similar level of understanding about their activities.
It’s true that we have carefully analyzed deworming as a program and found it to be among the most cost-effective programs we’ve considered. We have yet to assess the programs run by the organizations listed above, but my intuition again is that were we to analyze these programs – measles immunization, maternal and neonatal tetanus immunization, lymphatic filariasis control, onchocerciasis control – we would find programs that are as strong or nearly as strong as deworming.
Thanks for this post. It’s raised my confidence in GiveWell.
What could SCI do to allay your concerns? Why haven’t they done so?
Thank you for that comment, and indeed SCI realises that we must be doing something wrong if Elie is deserting us – We will work very hard in the coming months to win his support back by noting his criticisms. Alan Fenwick
Hi Sam, thanks for the comment and the kind words.
SCI could provide more detailed accounts of its activities, expenses, and the role of unrestricted funds in each region, and describe how these activities do and don’t relate to the activities it has carried out in support of past (apparently successful) deworming programs. I have a very strong understanding of exactly how AMF has spent funds, exactly how it plans to spend funds, and what all the steps of its process and activities are; I feel similarly about GiveDirectly.
The bottom line is that we haven’t been able to reach the same kind of concrete understanding of what SCI has done in the past with funds it has received as we have for AMF and GiveDirectly. Because we’ve spent so much time with SCI trying to understand this, it’s possible that either we are unable to understand it well enough or that SCI is unable to communicate it to us.
A thought that occurs to me while reading this is whether you think your aggregate contributions are large enough that you ought to take into account value-of-info effects. In other words, if SCI is trying this relatively new sort of project, do you think the amount of funding you can contribute or direct is high enough to potentially make a difference to whether the project is attempted, continued, or repeated? And hence to take into account the value of SCI repeating the project in future years if it works the first year?
I mention this because whenever I consider trying anything for the first time, my immediate thought is “value of information!” rather than just local expected cost and expected benefits. Lots of things with higher expected cost than expected benefit are worth trying once if your expectations are unstable and the action is potentially repeatable.
Eliezer: Thanks for the question. I’d say the answer is “in between.” We don’t think that donations to SCI are likely to lead to global learning in the same sense that funding GiveDirectly, which is trying relatively new approaches to an intervention, might, but we do think that “value of information” is a big part of the case for supporting SCI in the sense that GiveWell will learn from following up with additional charities over time. Holden describes this rationale in more depth here (see the section “How much should you give to each charity?”).
I have just returned from Ethiopia where we are investing $1 million over the next 3 months on mapping at least half of the country and then treatment of schistosomiasis and deworming children. Government commitment is fantastic, and there will be a launch of their NTD national plan on June 12th – 14th in the form of an NTD symposium. I would be thrilled if Elie could attend. Alan
i was very surprised to find SCI on your list of top 10 charities. As you say, deworming is an incredibly effective and proven intervention, but without accompanying improvements in sanitation and environmental modifications to reduce snail populations it is ultimately unsustainable. Can you guys recommend any charities that take a horizontal approach to healthcare improvements, rather than the primarily vertical programs which can be easier to evaluate and which you seem more likely to recommend?
Jessica, if you can make a convincing case that the benefits of these “horizontal” approaches provide better bang for buck than the SCI, I’d be eager to hear them. I’m sure Elie, Holden et al would too.
After reading this critique, I just perused the SCI website, and I see that they are giving quite a bit of detail on the strategy and, once programs have been in motion for a while, impact per country. Link glitches notwithstanding (the Burundi impact link leads to Burkina Faso impact data), in my opinion they are giving quite a lot of information on exactly what they’re doing in a given country. And measuring the outcomes, which makes me very happy!
Is this level of detail new, in response to Elie’s concerns, or did I misunderstand and Elie wants more of a financial breakdown on which step of the intervention cost what? The latter doesn’t seem to be available on the website, but I wouldn’t expect it to be. If the former, perhaps a note should be made at the top of this post that SCI has made some changes since the original post.
Amanda – thanks for the question. Though SCI reports historical data for some of the countries it has worked in, it is not exhaustive and doesn’t cover the current focus of its work (e.g., Ethiopia). It’s also not clear to us whether the level of monitoring it has done in the past is attributable to SCI’s initiative or the requirements of its institutional funders.
The type of information we’re looking for and haven’t seen is (a) up-to-date monitoring and evaluation information for all countries in which it works and (b) more detailed accounts of its activities, expenses, and the role of unrestricted funds in each region, and describe how these activities do and don’t relate to the activities it has carried out in support of past (apparently successful) deworming programs.
Comments are closed.