Since GiveWell recommended the Schistosomiasis Control Initiative (SCI) in November 2011, SCI has received about $1.4 million in unrestricted funds ($500,000 of which we directly attribute to GiveWell’s recommendation), of which $1.1 million remains to be spent. We have spoken with and met with SCI to discuss its plans for using these funds.

Funds spent to date

• SCI has made grants of $100,000 and $80,000 to Yemen and Senegal respectively for deworming drug delivery.
• SCI has granted about $25,000 to Tanzania to treat 153,000 children in one region.
• As part of discussions with the government about starting a national deworming program, SCI spent approximately $13,000 to support a conference on the deworming in Ethiopia.
• SCI has spent about $99,000 of the unrestricted funding on various organizational expenses such as travel and one staff member’s salary.

Plans for funds raised due to GiveWell’s research

SCI is currently planning to fund the delivery of donated schistosomiasis drugs to 1.5 million children in two regions of Ethiopia, and support disease mapping in other parts of the country. SCI is currently waiting on a budget from Ethiopia, but we estimate this project will cost a few hundred thousand dollars. SCI has told us that it expects these treatments to be delivered by the end of 2012.

SCI has also committed $100,000 to fund treatment of adults in one district of Burundi.

SCI is exploring the possibility of supporting deworming programs in Zimbabwe and the Democratic Republic of Congo. In addition, it may use unrestricted funding to provide additional treatments in Tanzania and Malawi, which are primarily supported by a large grant from the British government.

Comparing current plans to past plans

In November 2011, SCI told us that it would primarily use additional funds to expand deworming programs in Mozambique, Malawi, and Senegal. Recently, SCI told us that these are no longer the countries it expects to focus on with the funds raised through GiveWell. What has changed:

• SCI told us that it raised sufficient funding from other sources to support the Senegal program and that it never intended to expand the program beyond paying for delivery of drugs that were already available from the World Health Organization. We had been under the impression that SCI would expand this program further if it raised the money to do so, but it appears that we had a miscommunication with SCI on this point.
• In 2010, SCI received funding from the British government to support deworming programs in 8 countries, including Mozambique and Malawi. SCI recently decided that programs planned for two of the countries weren’t feasible, and shifted the money it expected to spend in these two countries to the Mozambique program.
• There may be other changes that we are not including here. We are not confident in our understanding of why SCI changed its plans.

Comments on SCI’s plans

First, SCI expects to spend almost all of the funds it has received due to GiveWell’s recommendation for a single round of treatment (save 15% of the funds, which it will hold for future treatments). Because multiple deworming treatments appear necessary for long-term impact (though the evidence on how many treatments are needed is thin), we are concerned that spending nearly all the funds now, could reduce SCI’s expected impact.

Second, our position is that treating children accounts for the majority of impacts from deworming. We are not confident in the impact of treating adults in Burundi.

We feel very good about recommending that donors give to our top charities, and they’re where all GiveWell staff are – happily and confidently – giving our personal donations this year.

That said, we’re skeptical by nature, and internally, we keep thinking about and discussing the ways in which our top charities might not work out: i.e., the scenarios/circumstances in which giving to these charities doesn’t accomplish what we expect or our views change substantially in the next year about what donations to these charities accomplish.

If I were going to write the equivalent of a “dissenting opinion” for our top charities, it would be based on the issues below.

AMF: insecticide-treated net distributions

Our #1-ranked organization is the Against Malaria Foundation (AMF). Our recommendation relies on the fact that there have been numerous randomized controlled trials (RCTs, i.e., high-quality studies) tying net distributions to reductions in childhood mortality, but there’s a question of external validity for these RCTs. Is it reasonable to assume that net distributions of the type AMF carries out will have the same sort of effects they’ve had in the studies? There are a couple of reasons to question whether they will.

In our analysis of the RCTs, we found one trial where researchers conducted random spot checks to people’s houses at 5 in the morning to make sure that nets were being used. In another case, researchers, themselves, installed the nets in participants’ homes. In general, we had the impression that researchers were working hard to make sure that nets were used as intended. While AMF’s distributions (and large-scale distributions in general) involve some measures for community education and monitoring of usage, we don’t think they’re comparable to what was done in the RCTs.

For nets to be effective, people need to use them appropriately and consistently. It’s not hard to imagine someone receiving a net one day, but over time not consistently using it well.

The best available data seems to indicate that on a large scale, people use their nets relatively consistently - in line with the usage rates seen in the RCTs (which did not, themselves, reach 100% usage). But the best available data isn’t as robust as we’d like it to be, particularly when it comes to the distinction between “reported usage” and “actual usage” (our best attempts to adjust for this indicate that it is not a major issue, but we aren’t working off much information).

Another way in which large-scale net distributions could fail to live up to the small-scale studies has to do with malaria transmission dynamics. It’s possible that mosquito populations and transmission dynamics have changed in the several years since the last small-scale study was done; it’s possible that the studies were done in areas that were unusually well-suited to seeing big drops in malaria come from nets; it’s even possible that widespread use of nets could lead to increasing resistance in mosquito populations over time. We haven’t seen any strong reasons to believe that any of these things are the case; in particular, we don’t believe that data on transmission dynamics is strong enough for the small-scale studies to have picked out particularly favorable settings. But it’s a definite point of uncertainty.

One point of reassurance here: as we observed previously, the malaria control community spends a huge amount of money on distributing nets. So if one of these concerns ends up being a huge problem, finding out about it (as we hope AMF will) could be hugely beneficial in and of itself, by helping the malaria community better execute distributions and/or better allocate resources.

SCI: deworming

Our evidence review of deworming concludes that there isn’t a particularly strong case that deworming has a significant impact on individuals’ life outcomes. The evidence boils down to (a) two well-respected but far-from-ironclad research papers on the long-term effects of deworming and (b) a large degree of evidence that deworming has a consistent (but limited) impact on haemoglobin levels (which are measured to determine anemia). Our positive view of deworming stems from the fact that although there is limited evidence of quality-of-life significance, the intervention is so cheap that it is likely to be a relatively good buy and may be a great buy.

Our view of deworming is highly susceptible to change based on evidence. A single pre-registered, large-sample RCT that did not find significant impacts on development for dewormed children would significantly adjust our view about the likely impacts of deworming. Combing through the data of one of the existing papers on deworming – which we plan to do in 2012 – could easily adjust our view of the intervention. In addition, despite the fact that SCI’s evidence of impact is as strong as we’ve ever seen in a charity, we still have our doubts about it.

Given this situation, it’s not hard to picture a world in which our view shifts substantially by the end of next year.

Both organizations

Both AMF and SCI partner with governments and NGOs to implement their prorgrams. This means that neither is in complete control of their activities and they could run into problems caused by, among other things, (a) lack of funding for the components of their programs that they don’t directly fund (more on this in Holden’s recent post about charitable leverage) or (b) governments changing their minds about working with each. Note that SCI has already had one program (in Zambia) where results have been quite disappointing and data remains thin.

Difficulty in dealing with partners is a significant risk in aid, and one we’ve learned about this past year from some of the challenges VillageReach is facing in its scaleup in Mozambique.

Putting the dissents in context

The above concerns are very real, but they’re much smaller than the concerns we have for any other charity in existence.

Some seem to believe that concerns ought to be hidden from donors, to encourage them to give. This may make sense for many donors. When dealing with our own audience, we feel much better asking you to give with eyes wide open: recognize that there are no guarantees, but that the things your money can accomplish - in terms of improving others’ lives - are incredible. We think that’s more than enough reason to give without regret.

We’ve recently published our updated charity recommendations, featuring two top charities (Against Malaria Foundation and Schistosomiasis Control Initiative) that score well on all of our criteria. In this post, we discuss how we decided which of these two charities to rank #1 and which to rank #2.

Both charities are executing health programs that deliver significant and very cheap help to people in the developing world. Both have strong track records and transparency, as well as concrete plans for how to use future donations.

Here’s what we see as the major relative pros and cons:

SCI has a more complete and convincing case that its past activities have had the intended outcomes.

• AMF has consistently gotten nets delivered to communities - and given the strong evidence on the impact of nets, this in itself is stronger evidence of impact than for nearly any other charity we’ve seen - but there are still some gaps in the picture. We aren’t sure whether, or for how long, nets are used properly, and we don’t have data on what has happened to malaria prevalence (though our research on nets in general has led us to believe that neither of these is a huge concern). AMF has made credible commitments to future data collection on both of these fronts (and has collected some data for the former).
• By contrast, SCI’s evidence shows substantial drops in disease prevalence. This evidence has some issues (which we discuss in the review), but overall we find it convincing.

This consideration is balanced somewhat by the fact that we are more confident in the quality-of-life significance of reducing malaria than of reducing parasitic infections.

AMF has more upside.

• It’s smaller, and appears to be earlier in its development (having just begun its first larger-scale distribution); the chance that GiveWell-influenced money can be crucial in its development is therefore higher.
• It’s working in an area - distribution of nets - where (a) an enormous amount of money is spent each year* (b) data on long-term usage and malaria prevalence following distributions still looks to us to be pretty thin. Well-executed and well-documented distributions could be valuable as pilots and as information for the hundreds of millions of dollars worth of other distributions going on.

We have more confidence in AMF as an organization. Both AMF and SCI are outstanding on this front: both are transparent and accountable with strong track records, and both have answered all our questions well. However,

• We’ve consistently (for more than a year now) found AMF noticeably easier to communicate with, and found it to address our concerns noticeably more clearly and directly. With AMF, we are more confident that we have gotten our questions fully answered, that we won’t later hear about something we should have heard about before, and that we will be able to learn about how our funds end up being used and whether things end up going well or poorly.
• SCI’s evaluation is outstanding, but may have been driven by its major funders (the Gates Foundation; DFID). With AMF, we are more convinced that the organization itself is committed to skeptical self-questioning, evaluation and improvement based on evaluation.
• Very broadly, all GiveWell staff agree that we have more general confidence in AMF’s operations and management than SCI’s. This is a completely subjective judgment call that isn’t attributable to any particular event - it’s just a general feeling based on the hours of conversations we’ve had with both organizations. This leads us to be more confident that AMF would make decisions we would ultimately agree with or understand in the face of new circumstances.

We are sufficiently confident in the people behind both SCI and AMF to feature them as top charities, but our confidence for AMF is higher, and if we kept this information to ourselves we wouldn’t feel that we’re telling donors the whole story. Ultimately, it’s hard to be 100% sure of how your money will be used before you give it; confidence in the people you’re giving to is an important factor.

We are more confident in malaria-related research than in deworming-related research. This is as topic we’ll be writing about more in the future. In brief,

• We have done extensive research on both nets and deworming. Studies on the former have consistently raised fewer unanswered questions and red flags than studies on the latter.
• Despite the work we’ve done, we still have many unanswered questions about both deworming and nets.
• We would guess that our unanswered questions will result in fewer negative adjustments for the nets, because we find the research - and by extension, the researchers - around nets to be more reliable.

The most important deciding factor for us comes down to a combination of cost-effectiveness and room for more funding.

• We believe that in general, the vast bulk of SCI’s expenditures go toward deworming children rather than adults (see the example of Yemen), and that this is a good thing because a major part of the case for deworming is the possibility of developmental impacts for people treated in childhood.
• We believe that deworming children is cost-effective - perhaps not quite as cost-effective (by our estimations) as net distribution, but close enough to make it a non-obvious call between the two.
• However, the activities that SCI would fund with additional dollars (in the range of what we’re likely to be able to send their way) look a bit different. Note that in Mozambique, the plan is to take children who have already been selected for planned every-other-year deworming and instead deworm them every year; we have little information to shed light on the likely marginal benefit here. Other potential activities include deworming selected and particularly at-risk adults. Overall, we feel that these activities will still accomplish substantial good, but that they’re unlikely to be as cost-effective as standard deworming of children.

Bottom line. SCI is among the best giving opportunities we’ve ever seen, and we recommend it to donors. However, GiveWell staff unanimously find AMF to be an even stronger opportunity.

There are obviously a lot of judgment calls here, and we are hoping to move substantial donations to each organization so that we can follow the progress of each and learn more for the future (we see this opportunity to learn as a major value in and of itself, in terms of making us better able to maximize the impact of future donations).

*See pages 12-13 of the World Malaria Report: in 2009-2010, the Global Fund and PMI alone spent ~$1.5 billion a year on malaria control, of which about 1/3 was for nets specifically.

GiveWell has published our annual update on how to accomplish as much good as possible with your donations.

Our top two charities - out of hundreds we’ve examined - are (1) the Against Malaria Foundation, which fights malaria using insecticide-treated bednets, and (2) the Schistosomiasis Control Initiative, which treats children for intestinal worms.

Our update is the result of a full year of intensive research: examining hundreds of charities, contacting the most promising ones, and completing in-depth investigations that include

• Conversations with representatives
• Examination of internal documentation including monitoring and evaluation reports, budgets, and plans for using additional funding
• Reviewing independent literature and evidence of effectiveness of the charities’ programs
• Site visits to charities’ work in the field

We have published the full details of our process, including a list of all charities examined and reviews for those examined in-depth.

Our top two charities are outstanding on all fronts. They execute proven, cost-effective programs for helping people. They have strong track records. They have concrete future plans and room for more funding. They are transparent and accountable to donors.

We also have identified five other standout organizations for donors interested in other causes. These are GiveDirectly (cash grants to poor households in Kenya), Innovations for Poverty Action (research on how to fight poverty and promote development), Nyaya Health (healthcare in rural Nepal), Pratham (primary education in India), and Small Enterprise Foundation (microfinance in South Africa).

Note that last year’s top-rated charity, VillageReach, does not have projected short-term funding needs (it expects to be able to meet these needs with funds not driven by GiveWell), as discussed previously.

The charities above all work in the developing world. Our top recommendation for donors who want to support causes in the United States is KIPP Houston, an outstanding charter schools facing budget cuts.

Over the last year, we drove over $1.6 million to our top-rated charities. We hope to drive substantially more over the coming year.

There are a lot of reasons to be interested in charities focused on neglected tropical diseases (NTDs), and particularly on deworming schoolchildren.

• Deworming schoolchildren, as discussed in a recent post, is ranked among the top programs in terms of “cost-effectiveness,” i.e., health benefits (in DALY terms) per dollar.
• Deworm the World has in the past been featured and, seemingly, implicitly endorsed by Poverty Action Lab,* which is one of the top sources of rigorous research on aid.
• Schistosomiasis Control Initiative and Deworm the World are rated as the top charities of Giving What We Can (a group that promotes both more generous and more intelligent giving).

We are positive on deworming as an intervention. But we feel that for individual donors, it is necessary to form confidence in charities, not just interventions. This is where our interest in NTD control hits major obstacles. This post goes through our questions and the difficulty we’ve had answering them, and focuses on charities focusing partly or fully on deworming.

Key questions

As far as we can tell (details below), the major organizations focused on deworming work heavily with developing-world governments. They spend money on both advocacy and subsidies for government control programs. We therefore have the following questions:

• How do the charity’s funds break down between subsidies and advocacy?
• For subsidies:
  • How much of the total spending on control programs has been covered by the charity, as opposed to the government?
  • How has the government been audited to ensure compliance with terms and conditions? Note that the Stop Tuberculosis Partnership serves in a similar role to NTD charities, subsidizing (through drug grants) government tuberculosis control programs, and it has an extremely thorough auditing process to ensure that drugs are used appropriately, that reported statistics are meaningful, etc. It has in the past discontinued funding for noncompliance. How do NTD control charities compare?
  • Does the charity intend to keep subsidies at the same level indefinitely, or does it intend for the government to take over activities? If the latter, what is its track record in accomplishing this very difficult task? Note that while we consider “sustainability” optional for some programs, we believe it is crucial for deworming, for reasons discussed previously.
• For advocacy activities:
  • How much has been spent, what programs (and where) have been advocated, and to what results? What does the future advocacy plan and budget look like, and what might be expected from it? Cost-effectiveness of control programs is one question; the cost-effectiveness of advocacy funds could be another question entirely.
  • To the extent that advocacy has succeeded, has it resulted in (a) more government funding for NTD control with no offsetting cuts, i.e., increases in total medical budgets; (b) more government funding for NTD control at the expense of other health spending; (c) more funding for NTD control, contingent on more donor subsidies?
  • If (a), how has the increase in funding been financed? If (b), where have cuts been made, and to what extent has funding shifted from other worthy health spending? If (c), what are the answers to the key questions about subsidies (above)?

Schistosomiasis Control Initiative (SCI)

A key roadblock to evaluating SCI is that it does not share any information about its budget by program (past or planned). Last year, after failing to find this information on its website, we contacted Prof. Alan Fenwick, and after significant back-and-forth we were told that no budget information could be shared. This makes it impossible to answer key questions about the role of advocacy vs. subsidies, and where future donations are likely to go.

SCI also declined to answer our questions about impact of past work directly, instead suggesting that we search PubMed. We did so, and have also recently reviewed SCI’s updated website, summarizing what we found in this XLS file. Our conclusions:

• There is serious evidence of the impact of past projects, a credit to SCI and something that is rare among charities in general.
• SCI appears to have provided both funding and technical assistance in the past (and from phone conversations we also understand that it has done advocacy work). Without budget information, we do not know how its funds have broken down between these activities or the specifics of what has been paid for by SCI vs. governments.
• SCI appears to have exited at least 3 countries (of 8 that we have information on), and the extent to which its programming has been sustained - both in terms of finances and quality - by the government is unclear.
• We do not have information on how SCI audits government data and practices (though the direct evidence of impact is encouraging), or on the specifics of its advocacy work.

Deworm The World

We have essentially no information about Deworm the World.

• Its website includes no reports on expenses or on specific projects, past or present.
• It is not clear to us where Deworm the World’s focus lies. It states that it can “link” funders to projects, while linking to a page where people can donate directly to it. It has another link implying that donations directly fund projects on the ground. But Poverty Action Lab’s description* makes it sound like the organization focuses significantly on advocacy and public information as well.
• Last year, after failing to find substantive information on the website, we emailed the organization twice and received no response.

We consider Poverty Action Lab itself to be a model of transparency, posting exhaustive information on studies both completed and in progress. But Deworm the World is on the opposite end of the spectrum, providing no substantive public information as far as we can tell.

We have in the past been tempted to recommend Deworm the World simply on the strength of our respect for Poverty Action Lab. But ultimately, conducting research projects is a different enough challenge from working with governments on large-scale programs that we think that doing so would be the wrong move - both in terms of the incentives it would provide and the good it would accomplish.

*At the time this post was drafted, Poverty Action Lab discussed Deworm the World in various places on its site. However, the links have since broken as Poverty Action Lab redid its website. We have not been able to find any mention of Deworm the World on Poverty Action Lab’s new website.