Highlights from our update of AMF follow. For those who want more information, please see our full AMF update.

Background

Historically, AMF had distributed bednets in the following way: on-the-ground organizations applied to AMF for nets; after AMF reviewed and approved proposals, it would ship nets to the charity; finally, AMF would check in to see whether the nets had been distributed and were being used.

In early 2011, AMF changed its model. It received significant funding, which allowed it to proactively identify opportunities for net distributions rather than reacting to charities’ requests. In mid-2011, it decided to provide about 250,000 nets (at a cost of a little over $1 million to AMF) to the Ntcheu district in Malawi as part of a national net distribution.

Note: In addition to the costs incurred by AMF, Concern Universal, AMF’s distribution partner in Malawi also incurred costs. We estimated these costs in our review, but are currently working to update our estimate of Concern’s costs because we believe that we underestimated Concern Universal’s costs in our original AMF review. We don’t believe our estimate of total costs per net will change significantly.

The Malawi distribution

In October 2011, three GiveWell staff members visited the Ntcheu district while pre-distribution activities were taking place (such as surveying local households to determine who needed nets and how many each household needed). Concern Universal, the organization distributing AMF-provided nets in Ntcheu, started the distribution in mid-December and now (as of mid-March) Concern Universal has reported that it had distributed 242,745 nets to households in the district, out of the 251,720 nets provided by AMF. For its final distribution location in late March/early April, Concern Universal requires another 16,574 nets, which AMF is sending. (More information in our full, detailed update.)

Concern has posted weekly reports on its progress on AMF’s website, which not only share data on weekly nets distributed but also share problems identified during distributions. Concern reports highlight a number of instances of attempted theft and fraud by the health workers (HSAs) Concern has employed to distribute nets. In one case, Concern believes that an HSA fabricated the existence of two villages in order to steal the nets. Neither we nor AMF are surprised by attempted fraud, and we are glad that Concern’s process has identified such instances.

We cannot be sure whether Concern has identified all instances of malfeasance. That said, we believe that Concern’s process for identifying attempted fraud (along with AMF’s oversight of Concern and our monitoring of the entire process) is robust and therefore, we would be aware of significant problems. (More information in our full, detailed update.)

AMF’s 2012 plans

• AMF currently has access to approximately $3.5 million in received (or committed and soon to be received) funding. Approximately $2.3 million of this comes from GiveWell-directed donors.
• Malawi is currently in the midst of a national net distribution, which has been delayed. (AMF-funded distributions are not delayed; the rest of the national distribution is.) Rob Mather told us that Malawi estimates it needs 5.8 million nets, of which it has access to 5.2 million (4.7 million from the Global Fund and approximately 500,000 from the President’s Malaria Initiative). It therefore has a gap of 600,000 nets, and AMF is in discussions with the NMCP to see if AMF will provide these nets.
• Rob Mather has told us that he is not yet ready to provide these nets. He (a) has not yet seen sufficient data to convince him that there is, indeed, a gap of 600,000 nets, and (b) he is not yet convinced that AMF funding, as opposed to funding from other partners (e.g., Global Fund), is needed. In the event that Malawi does provide this information, AMF would provide additional nets for the Malawi distribution. AMF has therefore reserved $2.5 million of its funding while it assesses the need for nets in Malawi.
• AMF will likely reach a decision about Malawi in either the next few weeks – if the decision is not to provide further nets for Malawi - or in the next 3-4 months if it looks like funding further nets is possible. Three to four months is the timeframe required to collect data from the National Malaria Control Program and from the field to determine the net gap. Coordination with the Global Fund and President’s Malaria Initiative is also required. In the event of a positive outcome, Mr. Mather expects that the nets would be distributed before the rainy season in November. In the event that AMF decides not to contribute further to the Malawi effort, it believes that there are strong needs for nets exist in campaigns in Ghana, Mali, Togo and several other countries.
• One of AMF’s broad goals in its interactions with the Global Fund and the National Malaria Control Program in Malawi is improving the way that net distributions are conducted. Mr. Mather believes the AMF/Concern Universal approach in the Ntcheu district distribution follows best practice for pre-distribution data collection, distribution oversight, and post-distribution monitoring (described in our AMF review). As AMF’s influence increases (due to increased funding ability), it hopes to influence other players involved in bednet distribution to adopt the Ntcheu distribution as a model.

Expected future revenue

AMF told us that it has no current commitments for significant future revenue, but is in discussions with donors who could potentially make 7-figure donations. We are currently comfortable with AMF receiving up to ~$15 million over the next year. This figure would be about half of what would be needed to close the net gaps for Malawi, Mali and Ghana (according to our most conservative estimates published in our coverage analysis), and would constitute a significant “step up” for AMF’s size - a somewhat risky but overall strong opportunity for donors. We would have to revisit the question of AMF’s room for more funding if it were to raise more than $15 million.

We learned last week that we underestimated the total cost of the Against Malaria Foundation’s insecticide-treated net distribution.

A nonprofit interested in reducing costs for its own model read our AMF review and contacted AMF and Concern Universal (AMF’s distribution partner in Malawi). The nonprofit representative noticed that Concern’s costs seemed lower than he expected, and he hoped that he could find a way to reduce his own organization’s costs.

Concern Universal’s response to the nonprofit representative’s request listed several categories of costs that we excluded in our cost estimate. We will release a full report about these costs once we have all the details and have put together a revised estimate.

In our view, it is a huge benefit of our transparency that people outside GiveWell are able to scrutinize the material we publish on charities and ask questions that lead us to catch errors. This allows us to improve our charity reviews as well as our overall process.

Update: A few people have asked us, “How likely is this to change AMF’s #1-ranking?” It almost definitely will not change based on this. Our back of the envelope estimate of the costs puts them at less than $1 per net, which would not change our bottom line.

We feel very good about recommending that donors give to our top charities, and they’re where all GiveWell staff are – happily and confidently – giving our personal donations this year.

That said, we’re skeptical by nature, and internally, we keep thinking about and discussing the ways in which our top charities might not work out: i.e., the scenarios/circumstances in which giving to these charities doesn’t accomplish what we expect or our views change substantially in the next year about what donations to these charities accomplish.

If I were going to write the equivalent of a “dissenting opinion” for our top charities, it would be based on the issues below.

AMF: insecticide-treated net distributions

Our #1-ranked organization is the Against Malaria Foundation (AMF). Our recommendation relies on the fact that there have been numerous randomized controlled trials (RCTs, i.e., high-quality studies) tying net distributions to reductions in childhood mortality, but there’s a question of external validity for these RCTs. Is it reasonable to assume that net distributions of the type AMF carries out will have the same sort of effects they’ve had in the studies? There are a couple of reasons to question whether they will.

In our analysis of the RCTs, we found one trial where researchers conducted random spot checks to people’s houses at 5 in the morning to make sure that nets were being used. In another case, researchers, themselves, installed the nets in participants’ homes. In general, we had the impression that researchers were working hard to make sure that nets were used as intended. While AMF’s distributions (and large-scale distributions in general) involve some measures for community education and monitoring of usage, we don’t think they’re comparable to what was done in the RCTs.

For nets to be effective, people need to use them appropriately and consistently. It’s not hard to imagine someone receiving a net one day, but over time not consistently using it well.

The best available data seems to indicate that on a large scale, people use their nets relatively consistently - in line with the usage rates seen in the RCTs (which did not, themselves, reach 100% usage). But the best available data isn’t as robust as we’d like it to be, particularly when it comes to the distinction between “reported usage” and “actual usage” (our best attempts to adjust for this indicate that it is not a major issue, but we aren’t working off much information).

Another way in which large-scale net distributions could fail to live up to the small-scale studies has to do with malaria transmission dynamics. It’s possible that mosquito populations and transmission dynamics have changed in the several years since the last small-scale study was done; it’s possible that the studies were done in areas that were unusually well-suited to seeing big drops in malaria come from nets; it’s even possible that widespread use of nets could lead to increasing resistance in mosquito populations over time. We haven’t seen any strong reasons to believe that any of these things are the case; in particular, we don’t believe that data on transmission dynamics is strong enough for the small-scale studies to have picked out particularly favorable settings. But it’s a definite point of uncertainty.

One point of reassurance here: as we observed previously, the malaria control community spends a huge amount of money on distributing nets. So if one of these concerns ends up being a huge problem, finding out about it (as we hope AMF will) could be hugely beneficial in and of itself, by helping the malaria community better execute distributions and/or better allocate resources.

SCI: deworming

Our evidence review of deworming concludes that there isn’t a particularly strong case that deworming has a significant impact on individuals’ life outcomes. The evidence boils down to (a) two well-respected but far-from-ironclad research papers on the long-term effects of deworming and (b) a large degree of evidence that deworming has a consistent (but limited) impact on haemoglobin levels (which are measured to determine anemia). Our positive view of deworming stems from the fact that although there is limited evidence of quality-of-life significance, the intervention is so cheap that it is likely to be a relatively good buy and may be a great buy.

Our view of deworming is highly susceptible to change based on evidence. A single pre-registered, large-sample RCT that did not find significant impacts on development for dewormed children would significantly adjust our view about the likely impacts of deworming. Combing through the data of one of the existing papers on deworming – which we plan to do in 2012 – could easily adjust our view of the intervention. In addition, despite the fact that SCI’s evidence of impact is as strong as we’ve ever seen in a charity, we still have our doubts about it.

Given this situation, it’s not hard to picture a world in which our view shifts substantially by the end of next year.

Both organizations

Both AMF and SCI partner with governments and NGOs to implement their prorgrams. This means that neither is in complete control of their activities and they could run into problems caused by, among other things, (a) lack of funding for the components of their programs that they don’t directly fund (more on this in Holden’s recent post about charitable leverage) or (b) governments changing their minds about working with each. Note that SCI has already had one program (in Zambia) where results have been quite disappointing and data remains thin.

Difficulty in dealing with partners is a significant risk in aid, and one we’ve learned about this past year from some of the challenges VillageReach is facing in its scaleup in Mozambique.

Putting the dissents in context

The above concerns are very real, but they’re much smaller than the concerns we have for any other charity in existence.

Some seem to believe that concerns ought to be hidden from donors, to encourage them to give. This may make sense for many donors. When dealing with our own audience, we feel much better asking you to give with eyes wide open: recognize that there are no guarantees, but that the things your money can accomplish - in terms of improving others’ lives - are incredible. We think that’s more than enough reason to give without regret.

We’ve recently published our updated charity recommendations, featuring two top charities (Against Malaria Foundation and Schistosomiasis Control Initiative) that score well on all of our criteria. In this post, we discuss how we decided which of these two charities to rank #1 and which to rank #2.

Both charities are executing health programs that deliver significant and very cheap help to people in the developing world. Both have strong track records and transparency, as well as concrete plans for how to use future donations.

Here’s what we see as the major relative pros and cons:

SCI has a more complete and convincing case that its past activities have had the intended outcomes.

• AMF has consistently gotten nets delivered to communities - and given the strong evidence on the impact of nets, this in itself is stronger evidence of impact than for nearly any other charity we’ve seen - but there are still some gaps in the picture. We aren’t sure whether, or for how long, nets are used properly, and we don’t have data on what has happened to malaria prevalence (though our research on nets in general has led us to believe that neither of these is a huge concern). AMF has made credible commitments to future data collection on both of these fronts (and has collected some data for the former).
• By contrast, SCI’s evidence shows substantial drops in disease prevalence. This evidence has some issues (which we discuss in the review), but overall we find it convincing.

This consideration is balanced somewhat by the fact that we are more confident in the quality-of-life significance of reducing malaria than of reducing parasitic infections.

AMF has more upside.

• It’s smaller, and appears to be earlier in its development (having just begun its first larger-scale distribution); the chance that GiveWell-influenced money can be crucial in its development is therefore higher.
• It’s working in an area - distribution of nets - where (a) an enormous amount of money is spent each year* (b) data on long-term usage and malaria prevalence following distributions still looks to us to be pretty thin. Well-executed and well-documented distributions could be valuable as pilots and as information for the hundreds of millions of dollars worth of other distributions going on.

We have more confidence in AMF as an organization. Both AMF and SCI are outstanding on this front: both are transparent and accountable with strong track records, and both have answered all our questions well. However,

• We’ve consistently (for more than a year now) found AMF noticeably easier to communicate with, and found it to address our concerns noticeably more clearly and directly. With AMF, we are more confident that we have gotten our questions fully answered, that we won’t later hear about something we should have heard about before, and that we will be able to learn about how our funds end up being used and whether things end up going well or poorly.
• SCI’s evaluation is outstanding, but may have been driven by its major funders (the Gates Foundation; DFID). With AMF, we are more convinced that the organization itself is committed to skeptical self-questioning, evaluation and improvement based on evaluation.
• Very broadly, all GiveWell staff agree that we have more general confidence in AMF’s operations and management than SCI’s. This is a completely subjective judgment call that isn’t attributable to any particular event - it’s just a general feeling based on the hours of conversations we’ve had with both organizations. This leads us to be more confident that AMF would make decisions we would ultimately agree with or understand in the face of new circumstances.

We are sufficiently confident in the people behind both SCI and AMF to feature them as top charities, but our confidence for AMF is higher, and if we kept this information to ourselves we wouldn’t feel that we’re telling donors the whole story. Ultimately, it’s hard to be 100% sure of how your money will be used before you give it; confidence in the people you’re giving to is an important factor.

We are more confident in malaria-related research than in deworming-related research. This is as topic we’ll be writing about more in the future. In brief,

• We have done extensive research on both nets and deworming. Studies on the former have consistently raised fewer unanswered questions and red flags than studies on the latter.
• Despite the work we’ve done, we still have many unanswered questions about both deworming and nets.
• We would guess that our unanswered questions will result in fewer negative adjustments for the nets, because we find the research - and by extension, the researchers - around nets to be more reliable.

The most important deciding factor for us comes down to a combination of cost-effectiveness and room for more funding.

• We believe that in general, the vast bulk of SCI’s expenditures go toward deworming children rather than adults (see the example of Yemen), and that this is a good thing because a major part of the case for deworming is the possibility of developmental impacts for people treated in childhood.
• We believe that deworming children is cost-effective - perhaps not quite as cost-effective (by our estimations) as net distribution, but close enough to make it a non-obvious call between the two.
• However, the activities that SCI would fund with additional dollars (in the range of what we’re likely to be able to send their way) look a bit different. Note that in Mozambique, the plan is to take children who have already been selected for planned every-other-year deworming and instead deworm them every year; we have little information to shed light on the likely marginal benefit here. Other potential activities include deworming selected and particularly at-risk adults. Overall, we feel that these activities will still accomplish substantial good, but that they’re unlikely to be as cost-effective as standard deworming of children.

Bottom line. SCI is among the best giving opportunities we’ve ever seen, and we recommend it to donors. However, GiveWell staff unanimously find AMF to be an even stronger opportunity.

There are obviously a lot of judgment calls here, and we are hoping to move substantial donations to each organization so that we can follow the progress of each and learn more for the future (we see this opportunity to learn as a major value in and of itself, in terms of making us better able to maximize the impact of future donations).

*See pages 12-13 of the World Malaria Report: in 2009-2010, the Global Fund and PMI alone spent ~$1.5 billion a year on malaria control, of which about 1/3 was for nets specifically.

GiveWell has published our annual update on how to accomplish as much good as possible with your donations.

Our top two charities - out of hundreds we’ve examined - are (1) the Against Malaria Foundation, which fights malaria using insecticide-treated bednets, and (2) the Schistosomiasis Control Initiative, which treats children for intestinal worms.

Our update is the result of a full year of intensive research: examining hundreds of charities, contacting the most promising ones, and completing in-depth investigations that include

• Conversations with representatives
• Examination of internal documentation including monitoring and evaluation reports, budgets, and plans for using additional funding
• Reviewing independent literature and evidence of effectiveness of the charities’ programs
• Site visits to charities’ work in the field

We have published the full details of our process, including a list of all charities examined and reviews for those examined in-depth.

Our top two charities are outstanding on all fronts. They execute proven, cost-effective programs for helping people. They have strong track records. They have concrete future plans and room for more funding. They are transparent and accountable to donors.

We also have identified five other standout organizations for donors interested in other causes. These are GiveDirectly (cash grants to poor households in Kenya), Innovations for Poverty Action (research on how to fight poverty and promote development), Nyaya Health (healthcare in rural Nepal), Pratham (primary education in India), and Small Enterprise Foundation (microfinance in South Africa).

Note that last year’s top-rated charity, VillageReach, does not have projected short-term funding needs (it expects to be able to meet these needs with funds not driven by GiveWell), as discussed previously.

The charities above all work in the developing world. Our top recommendation for donors who want to support causes in the United States is KIPP Houston, an outstanding charter schools facing budget cuts.

Over the last year, we drove over $1.6 million to our top-rated charities. We hope to drive substantially more over the coming year.