The GiveWell Blog

Weighing organizational strength vs. estimated cost-effectiveness

(updated on: ) | by Sean

A major question we’ve asked ourselves internally over the last few years is how we should weigh organizational quality versus the value of the intervention that the organization is carrying out.

In particular, is it better to recommend an organization we’re very impressed by and confident in that’s carrying out a good program, or better to recommend an organization we’re much less confident in that’s carrying out an exceptional program? This question has been most salient when deciding how to rank giving to GiveDirectly vs giving to the Schistosomiasis Control Initiative.

GiveDirectly vs SCI

GiveDirectly is an organization that we’re very impressed by and confident in, more so than any other charity we’ve come across in our history. Reasons for this:

But, we estimate that marginal dollars to the program it implements — direct cash transfers — are significantly less cost-effective than bednets and deworming programs. Excluding organizational factors, our best guess is that deworming programs — which SCI supports — are roughly 5 times as cost-effective as cash transfers. As discussed further below, our cost effectiveness estimates are generally based on extremely limited information and are therefore extremely rough, so we are cautious in assigning too much weight to them.

Despite the better cost-effectiveness of deworming, we’ve had significant issues with SCI as an organization. The two most important:

  • We originally relied on a set of studies showing dramatic drops in worm infection coinciding with SCI-run deworming programs to evaluate SCI’s track record; we later discovered flaws in the study methodology that led us to conclude that they did not demonstrate that SCI had a strong track record. We wrote about these flaws in 2013 and 2014.
  • We’ve seen limited and at times erroneous financial information from SCI over the years. We have seen some improvements in SCI’s financial reporting in 2016, but we still have some concerns, as detailed in our most recent report.

More broadly, both of these cases are examples of general problems we’ve had communicating with SCI over the years. And we don’t believe SCI’s trajectory has generated evidence of overall impressiveness comparable to GiveDirectly’s, discussed above.

Which should we recommend?

One argument is that GiveWell should only recommend exceptional organizations, and so the issues we’ve seen with SCI should disqualify them.

But, we think that the ~5x difference in cost-effectiveness is meaningful. There’s a large degree of uncertainty in our cost-effectiveness analyses, which is something we’ve written a lot about in the past, but this multiplier appears somewhat stable (it has persisted in this range over time, and currently is consistent with the individual estimates of many staff members), and a ~5x difference gives a fair amount of room for SCI to do more good even accounting both for possible errors in our analysis and for differences in organizational efficiency.

A separate argument that we’ve made in the past is that great organizations have upside that goes beyond the value of conducting the specific program they’re implementing. For example, early funding to a great organization may have allow it to grow faster and increase the amount of money going to their program globally, either through proving the model or through their own fundraising. And GiveDirectly has shown some propensity for potentially innovative projects, as discussed above.

We think that earlier funding to GiveDirectly had this benefit, but it’s less of a consideration now that GiveDirectly is a more mature organization.  We believe this upside exists for what we’ve called “capacity-relevant” funding, which is the type of funding need that we consider to be most valuable when ranking the importance of marginal dollars to each of our top charities, and refers to funding gaps that we expect will allow organizations to grow in an outsized way in the future, for instance by going into a new country.

Bottom line

Our most recent recommendations ranked SCI’s funding gap higher than GiveDirectly’s due to SCI’s cost-effectiveness. We think that SCI is a strong organization overall, despite the issues we’ve noted, and we think that the “upside” for GiveDirectly is limited on the margin, so ultimately our estimated 5x multiplier looks meaningful enough to be determinative.

We remain conflicted about this tradeoff and regularly debate it internally, and we think reasonable donors may disagree about which organization to support.

Why we don’t currently recommend charities focused on vaccine distribution

(updated on: ) | by Catherine Hollander

GiveWell does not recommend any charities focused on vaccine funding and distribution. But we remain excited about vaccinations as a health intervention. The vaccination programs we’ve researched have been backed by strong, independent evidence of effectiveness and appear likely to be competitive with our top charities in their cost-effectiveness. We’d be excited to support a charity to implement these programs. This post will describe why we don’t right now.

In brief, we’ve been looking for vaccination giving opportunities over the last few years, but have continued to fail to find them. This is due to (a) lack of room for more funding and (b) UNICEF’s decision not to participate in our review process.

In particular, over the past 1-2 years, we’ve been looking for funding opportunities for measles, meningitis A, and maternal and neonatal tetanus vaccination. Each of these is discussed in greater detail below.

Measles and Rubella Initiative

We have nearly completed an assessment of the evidence and cost-effectiveness for supplementary measles and rubella campaigns. (We’ve summarized our current take below; more detail will be available in our full intervention report, which we hope to publish on our website this year.) These campaigns supplement routine, childhood immunization and aim to vaccinate all children against measles and rubella. In particular, children between the ages of 9 months to 14 or 15 years are targeted. The evidence that such campaigns — when targeting children under age 5, for whom the disease is most likely to be fatal — are effective appears to be strong and the cost-effectiveness (per measles death averted) is competitive with that of our top charities.

However, we don’t believe that the Measles and Rubella Initiative (the primary entity that supports these campaigns) has room for more funding to vaccinate children under age 5. We spoke with M&RI in January 2016 and representatives there told us (p. 4 at that link) that M&RI has a funding gap of approximately $36 million in 2016. Of the $36 million, $31 million would fund a campaign targeting 5-14 year olds in Ethiopia. Vaccinating children age 5-14 would, by reducing the number of people who could contract and transmit the disease, reduce infections in children under 5 and could potentially save lives. We have not pursued this opportunity because our guess is that this will be less cost-effective than our top charities. Gavi, a large alliance which funds vaccinations, has fully funded M&RI’s gap for vaccinations of <5-year-olds in Ethiopia but has not fully filled its 5-14 year-old gap. Though we don’t have information about why Gavi made this funding decision, it is consistent with our impression that filling the 5-14 year-old gap may be less cost-effective than the <5-year-old gap.

We believe the remaining $5 million gap for 2016 is very small compared to the size of M&RI’s total budget. As of November 2015, M&RI estimated (p. 15 at that link) $662.6 million in resource requirements for 2016. Note that this does not include the large campaign in Ethiopia, which has been carried over from 2015 to 2016. (Details here (p. 35-36) and here (p. 3).)

Thus, our impression is that the $5 million funding gap beyond the Ethiopia campaign represents an extremely small fraction of the total M&RI budget for 2016. The $5 million gap is so small relative to the total that we would guess that M&RI would be able to raise funding for it if it represented a pressing need, either from Gavi or another source. (We would guess that it doesn’t, since Gavi funds so much of M&RI’s work that it seems very unlikely that it would leave such a small gap.)

Meningitis A

It also appears to us that there is no room for more funding in meningitis A vaccinations. We didn’t complete a full assessment of the evidence of effectiveness and cost-effectiveness for meningitis A vaccines because we learned early on in our investigation that there was unlikely to be room for more funding. However, our guess is that this intervention would be competitive with our top charities’ cost-effectiveness. Our meningitis A write-up concludes:

…[I]t seems unlikely that there will be room for more funding to support additional mass campaigns (or related immunization activities) in the meningitis belt in the near future. Gavi, a large funding vehicle for vaccinations, appears to have enough funding to fulfill its commitment to support all such activities in all 26 countries in the meningitis belt.

We’re not aware of any organizations other than Gavi funding meningitis A vaccine programs.

Maternal and neonatal tetanus campaigns

Vaccination campaigns to prevent maternal and neonatal tetanus appear potentially as cost-effective as our top charities. (More details are in our full report on this intervention.)

We have been following UNICEF’s work in this area, the Maternal and Neonatal Tetanus Elimination Initiative, since 2012. UNICEF recently informed us that it was declining to participate in our review process. We plan to write more about our understanding of UNICEF’s decision in the future. Our impression is that UNICEF is the primary funding vehicle for maternal and neonatal tetanus campaigns.

This is another example where we tried but failed to find a way to fund vaccinations. We are not aware of organizations conducting similar work, but would be interested in considering a similar opportunity if we identified one.

Challenges in finding a great Vitamin A charity

| by Sean

Vitamin A supplementation involves giving Vitamin A to children at risk of deficiency to prevent death and other negative health impacts. We’d be interested in supporting a charity to carry out this program, but so far we have not found one we’d like to recommend.

The evidence on the effectiveness of the program raises a number of questions that we’d need a charity to answer, and we haven’t found one that can satisfactorily answer them. This post lays out the state of the evidence regarding Vitamin A, and the questions charities would need to answer to receive our recommendation.

This is a summary of our full report on Vitamin A Supplementation as a charitable program. As with our recent post about water quality interventions, we’re interested in providing more accessible summaries of our research to illustrate the challenges of identifying the most effective charities.

The key points of this post:

  • Vitamin A supplementation has a mixed evidence base that seems to suggest that the program is particularly effective in certain circumstances.
  • Because of the mixed evidence base, we have a set of questions any Vitamin A charity would need to answer before we would be willing to make them one of our top charities.

What is the problem?

Vitamin A deficiency (VAD) is a common, potentially deadly, condition in the developing world. Symptoms include:

  • stunting
  • anemia
  • dry eyes (the leading cause of preventable childhood blindness)
  • susceptibility to infection
  • death

Providing children with doses of Vitamin A two to three times per year can combat Vitamin A deficiency, and is typically relatively inexpensive. Doses cost less than a dollar per person per year, although distribution can be more costly.

Does Vitamin A supplementation work?

Studies of Vitamin A supplementation primarily focus on whether giving vitamin A pills to children can reduce their risk of dying.

Many large, randomized controlled trials (which we consider a particularly strong method of evaluating global health programs) have been conducted to determine the impact of Vitamin A supplementation, in which children are randomly chosen to receive the pills or not, and the mortality rate between recipients and non-recipients can be compared.

Most of the results look very promising: a Cochrane Collaboration review of seventeen randomized studies, mostly conducted in the 1980s and 1990s, found that Vitamin A supplementation reduces all-cause mortality by 24%.[1]

However, one major study, with four times as many participants as all the studies included in the Cochrane review combined, contradicts these results.

The Deworming and Enhanced Vitamin A Study (DEVTA) was published after the Cochrane review, and did not find a statistically significant effect of giving children Vitamin A pills.[2]

Reconciling DEVTA and earlier trials

What should we think about whether Vitamin A is an effective program to support given that 17 trials found a large, significant effect of giving children Vitamin A supplements, while DEVTA found no statistically significant effect? There are a few possibilities:

  1. The world changed between the time the initial 17 studies were conducted and DEVTA, and Vitamin A supplementation is no longer as effective. Vitamin A supplementation may only be effective in areas with extremely high Vitamin A deficiency or child mortality – if so, worldwide improvements in health may mean Vitamin A supplementation is not as impactful as it once was, on average.
  2. The best guide to impact is averaging the effects of DEVTA and the other 17 trials. If we believe DEVTA’s results were due to chance, we could assume that the average effect of Vitamin A supplementation will be a 12% reduction in childhood mortality, based on the Cochrane meta-analysis that includes DEVTA. This is about half the size of the effect that was estimated prior to DEVTA.
  3. DEVTA’s lack of results were due to specific features of DEVTA, and charities conducting work under conditions more similar to the original 17 trials are likely to be highly impactful. Some differences between DEVTA and the original trials include (read more):
      1. DEVTA may have failed to reach enough children. DEVTA reported treating 86% of the children in the study, close to the rate achieved in previous trials, but some researchers have called this number into question, believing the study was not implemented as rigorously as previous trials and that it is implausible to achieve such high coverage at such low cost.
      2. DEVTA may have treated a population with less severe or less prevalent Vitamin A deficiency than in previous trials. However, DEVTA reports that a similar percentage of children had Vitamin A deficiency as in previous trials, although reliable, comparable data on Vitamin A deficiency is scarce.
      3. The population treated by DEVTA may have had better overall health than previously studied populations. Deaths prevented by Vitamin A Supplementation may be due to reduced mortality from diarrhea or measles, so if DEVTA participants were less vulnerable to dying from these diseases than participants in other studies, we would expect Vitamin A to have a smaller effect on mortality. The mortality rate in the control group in DEVTA was lower than control group mortality in 4 of the 5 trials that account for 80% of the weight in the Cochrane review, although the prevalence of diarrhea and measles does not appear to be very different.

We would guess that the best available explanation for the discrepancy between DEVTA and the 17 earlier trials is the lower baseline child mortality rate – and possibly better overall health – among DEVTA participants.

Finding a Vitamin A charity to recommend

For this reason, we would be interested in recommending a charity that could demonstrate that the conditions of the area it targets are similar to those of the original 17 studies considered in the Cochrane review, and so we have a list of questions that a charity would have to answer in order for us to consider them as a top charity.

Our list of questions includes:

  • Is the charity working in an area with high child mortality?
  • To what extent do children in the targeted area suffer from Vitamin A deficiency?
  • Can the charity provide evidence that it successfully reaches the children it targets?

Our current conclusion

Considering all of these factors, we believe that Vitamin A supplementation may be one of the most cost-effective ways to save lives when the program is high quality and delivered in locations with high child mortality rates. Before recommending a charity, we’d need to see compelling evidence to satisfy these concerns about the conditions under which providing children with Vitamin A supplements is likely to be effective. Our concerns have not been satisfactorily addressed for any charity we’ve spoken with so far.

Notes

[1]

95% confidence interval: 17% to 31%

[2]

The study found a 4% reduction in child mortality, but with a 95% confidence interval between a 3% increase in child mortality and an 11% decrease in child mortality, so the study leaves open the possibility that giving children Vitamin A supplements has no effect on their risk of death.

Mid-year update to top charity recommendations

(updated on: ) | by Natalie Crispin

This post provides an update on what we’ve learned about our top charities in the first half of 2016.

We continue to recommend all four of our top charities. Our recommendation for donors seeking to directly follow our advice remains the same: we recommend they give to the Against Malaria Foundation (AMF), which we believe has the most valuable current funding gap.

Below, we provide:

  • Updates on our view about AMF, which we consider the most important information we’ve learned in the last half-year (More)
  • Updates on other top charities (More)
  • A discussion of the reasoning behind our current recommendation to donors (More)

Updates on AMF

 

Background

AMF (www.againstmalaria.com) provides funding for long-lasting insecticide-treated net distributions (for protection against malaria) in developing countries. There is strong evidence that distributing nets reduces child mortality and malaria cases. AMF has relatively strong reporting requirements for its distribution partners and provides a level of public disclosure and tracking of distributions that we have not seen from any other net distribution charity. Overall, AMF is the best giving opportunity we are currently aware of. That said, we have concerns about AMF’s recent monitoring and transparency that we plan to focus on in the second half of the year.

Updates from the last six months

We are more confident than we were before in AMF’s ability to successfully complete deals with most countries it engages with. Over the past few years, our key concern about AMF has been whether it would be able to effectively absorb additional funding and sign distribution agreements with governments and other partners. At the end of 2013, we stopped recommending AMF because we felt it did not require additional funding, and our end-of-year analyses in 2014 and 2015 discussed this issue in depth. In early 2016, AMF signed agreements to fund two large distributions (totaling $37 million) of insecticide-treated nets in countries it has not previously worked in. We now believe that AMF has effectively addressed this concern.

AMF is in discussions for several additional large distributions. AMF currently holds approximately $23.3 million, and we believe that it is very likely to have to slow its work if it receives less than an additional $11 million very quickly. It is possible that it could also use up to an additional (approximately) $18 million more during this calendar year.

It may be more valuable to give to AMF now than it will be later this year or next year. AMF’s funding gap may be time-sensitive because:

  1. AMF is in several discussions about distributions that would take place in 2017. It has told us that it needs to make decisions within a month or two about which discussions to pursue. We don’t have a clear sense for how long before a distribution AMF needs to be able to commit funding, and note that, for example, AMF committed in February 2016 to a distribution in Ghana taking place in June to August 2016. That said, it seems quite plausible that AMF needs to commit soon to distributions taking place in 2017.
  2. We don’t know whether there will be large funding gaps for nets in 2018 and beyond. The price of nets has been decreasing and the size of grants from the two largest funders of nets, the Global Fund to fight AIDS, TB, and Malaria and the President’s Malaria Initiative, is not yet known. (The Global Fund is holding its replenishment conference in September, in which donor governments are asked to make three-year pledges, so we may know more before the end of the year.) It’s possible that these funders will fund all or nearly all of the net needs in countries other than those that are particularly hard to work in for 2018. If that happens, gifts to AMF in late 2016 could be less valuable than gifts in the next couple of months. (This could also mean that, if AMF fills gaps in 2017 that would have been filled by other funders in 2018, gifts now are less valuable than they have been in the past. We have added an adjustment for this to our cost-effectiveness analysis, but given the high degree of uncertainty, this could be a more important factor than we are currently adjusting for.)


Notwithstanding the above, we have important questions about AMF that we plan to continue to investigate. None of these developments caused us to change our recommendation about giving to AMF, but they are important considerations for donors:

  1. Monitoring data: We have new concerns about AMF’s monitoring of its distributions, particularly its post-distribution check-up (PDCU) surveys. These surveys are a key part of our confidence in the quality of AMF’s distributions. For Malawi, where most of the PDCUs completed to date have been done, our key concern is that villages that surveyors visit are not selected randomly, but are instead selected by hand by staff of the organization that both implements and monitors the distributions, which seems fairly likely to lead to bias in the results. We have also seen results from the first two PDCUs from DRC. We have not yet looked at the DRC results in-depth or discussed them with AMF, but there appear to be major problems in how the surveys were carried out (particularly a high percentage of internally inconsistent data – around 40%-50%) and, if we believe the remaining data, fairly high rates of missing or unhung nets (~20% at 6-months) and nets that deteriorated quickly (65% were in ‘very good’ or ‘good’ condition at 6-months).
  2. Transparency: Recently, AMF has been slower to share documentation from some distributions. AMF has told us that it has this documentation and we are concerned that AMF is not being as transparent as it could be. We believe this documentation is important for monitoring the quality of AMF’s distributions; it includes PDCUs, results from re-surveying 5% of households in during pre-distribution registrations (AMF has told us that this is a standard part of its process, but we have not seen results from any distributions), and malaria case rate data from Malawi that AMF has told us it has on hand. AMF attributes the delays to lack of staff capacity. We plan to write more about monitoring and transparency in a future post.
  3. Insecticide resistance: Insecticide resistance (defined broadly as “any ways in which populations of mosquitoes adapt to the presence of insecticide-treated nets (ITNs) in order to make them less effective”) is a major threat to the effectiveness of ITNs. Insecticide resistance seems to be fairly common across sub-Saharan Africa, and it seems that resistance is increasing. It remains difficult to quantify the impact of resistance, but our very rough best guess (methodology described in more detail below) is that ITNs are roughly one-third less effective in the areas where AMF is working than they would be in the absence of insecticide resistance. We continue to believe, despite resistance, ITNs remain a highly cost-effective intervention. See our full report for more detail.

Other updates on AMF

  • To better understand whether AMF is providing nets that would not otherwise have been funded, we considered five cases where AMF considered funding a distribution and did not ultimately provide funding. We then looked at whether other funders stepped in and how long of a delay resulted from having to wait for other funders. We published the details here. In short, most distributions took place later than they would have if AMF had funded them (on average over a year), which probably means that the people were not protected with nets during that time. We feel that these case studies provide some evidence that nets that AMF buys do not simply displace nets from other funding sources.
  • We’ve noted in the past that the delays in AMF signing agreements for distributions may have been due to AMF’s hesitation about paying for the costs of a distribution other than the purchase price of nets. For the distributions that AMF has signed this year, AMF has agreed to pay for some non-net costs, particularly the costs of PDCUs. The Global Fund to fight AIDS, TB, and Malaria is paying for the other non-net costs of the distribution. AMF’s willingness to fund some of the non-net costs may have made it easier for it to sign distribution agreements and put funds to use more quickly.

Updates on our other top charities

 

Schistosomiasis Control Initiative (full report)

Background

SCI (www3.imperial.ac.uk/schisto) works with governments in sub-Saharan Africa to create or scale up deworming programs (treating children for schistosomiasis and other intestinal parasites). SCI’s role has primarily been to identify recipient countries, provide funding to governments for government-implemented programs, provide advisory support, and conduct research on the process and outcomes of the programs.

In past years, we’ve written that we had significant concerns about SCI’s financial reporting and financial management that meant we lacked high-quality, basic information about how SCI was spending funding and how much funding it had available to allocate to programs. We decided to focus our work in the first half of 2016 on this issue. We felt that seeing significant improvements in the quality of SCI’s finances was necessary for us to continue recommending SCI.

We believe that deworming is a program backed by relatively strong evidence. We have reservations about the evidence, but we think the potential benefits are great enough, and costs low enough, to outweigh these reservations. SCI has conducted studies in about half of the countries it works in (including the countries with the largest programs) to determine whether its programs have reached a large proportion of children targeted. These studies have generally found moderately positive results, but have major methodological limitations. We have not asked SCI for monitoring results since last year.

Updates from the last six months

We published a separate blog post on our work on SCI so far this year. Our main takeaways:

  • SCI has begun producing higher-quality financial documents that allow us to learn some basic financial information about SCI.
  • We learned of two substantial errors in SCI’s financial management and reporting. 1) a July 2015 grant from GiveWell for about $333,000 was misallocated within Imperial College, which houses SCI, until we noticed it was missing from SCI’s revenue in March 2016; and (2) in 2015, SCI underreported how much funding it would have from other sources in 2016, leading us to overestimate its room for more funding by $1.5 million.
  • The clarity of our communication with SCI about its finances has improved, but there is still substantial room for further improvement.

We feel that SCI has improved, but we would still rank our other top charities ahead of it in terms of our ability to communicate and understand their work. Given this situation, we continue to recommend SCI now and think that SCI is reasonably likely to retain its top charity status at the end of 2016. We plan, in the second half of 2016, to expand the scope of our research on SCI.

We have not asked SCI for an update on its room for more funding (due to our focus on financial documents in the first half of the year). It’s our understanding that funds that SCI receives in the next six months will be allocated to work in 2017 and beyond. Because of this, we don’t believe that SCI has a pressing need for additional funds, though our guess is that it will have room for more funding when we next update our recommendations in November and that funds given before then will help fund gaps for the next budget year.

GiveDirectly (full report)

Background

GiveDirectly (www.givedirectly.org) transfers cash to households in developing countries via mobile phone-linked payment services. It targets extremely low-income households. The proportion of total expenses that GiveDirectly has delivered directly to recipients is approximately 83% overall. We believe that this approach faces an unusually low burden of proof, and that the available evidence supports the idea that unconditional cash transfers significantly help people.

We believe GiveDirectly to be an exceptionally strong and effective organization, even more so than our other top charities. It has invested heavily in self-evaluation from the start, scaled up quickly, and communicated with us clearly. It appears that GiveDirectly has been effective at delivering cash to low-income households. GiveDirectly has one major randomized controlled trial (RCT) of its impact and took the unusual step of making the details of this study public before data was collected (more). It continues to experiment heavily.

Updates from the last six months

  • GiveDirectly announced an initiative to test a “basic income guarantee” to provide long-term, ongoing cash transfers sufficient for basic needs. The cost-effectiveness of providing this form of cash transfers may be different from the one-time transfers GiveDirectly has made in the past.
  • GiveDirectly continues to have more room for more funding than we expect GiveWell-influenced donors to fill in the next six months. Its top priority is funding the basic income guarantee project.
  • In late 2015 and early 2016, when GiveDirectly began enrolling participants in Homa Bay county, Kenya, it experienced a high rate of people refusing to be enrolled in the program. The reason for this is not fully clear, though GiveDirectly believes in some cases local leaders advised people to not trust the program. While GiveDirectly has temporarily dealt with this setback by moving its operations to a different location in Homa Bay county, it is possible that similar future challenges could reduce GiveDirectly’s ability to commit as much as it currently projects.
  • GiveDirectly has reached an agreement with a major funder which provides a mechanism through which multiple benchmarking projects (projects comparing cash transfers to other types of aid programs) can be launched. The major funder may fund up to $15 million for four different benchmarking projects with GiveDirectly. GiveDirectly plans to make available up to $15 million of the grant it received from Good Ventures in 2015 to match funds committed by the major funder. GiveDirectly and its partner have not yet determined which aid programs will be evaluated or how the evaluations will be carried out.
  • We are reasonably confident that GiveDirectly could effectively use significantly more funding than we expect it to receive, including an additional $30 million for additional cash transfers in 2016, though scaling up to this size would require a major acceleration in the second half of the year. We have not asked GiveDirectly how funding above this amount would affect its activities and plans (because we think it is very unlikely that GiveDirectly will receive more than $30 million from GiveWell-influenced supporters before our next update in November).

Deworm the World (full report)

Background

Deworm the World (www.evidenceaction.org/deworming), led by Evidence Action, advocates for, supports, and evaluates government-run school-based deworming programs (treating children for intestinal parasites).

We believe that deworming is a program backed by relatively strong evidence. We have reservations about the evidence, but we think the potential benefits are great enough, and costs low enough, to outweigh these reservations. Deworm the World retains monitors whose reports indicate that the deworming programs it supports successfully deworm children.

Updates from the last six months

  • We asked Deworm the World whether additional funding in the next six months would change its activities or plans. It told us that it does not expect funding to be the bottleneck to any work in that time. We’d guess that there is a very small chance that it will encounter an unexpected opportunity and be bottlenecked by funding before our next update in November.
  • Deworm the World appears to be making progress expanding to new countries. It has made a multi-year commitment to provide technical assistance and resources to Cross River state, Nigeria for its school-based deworming program (the first deworming is scheduled for the end of this month), and are undertaking a nationwide prevalence survey in Pakistan.
  • In the past, we have focused our review of Deworm the World on its work in India. We are in the process of learning more about its work in other locations, particularly Kenya. The monitoring we have seen from Kenya appears to be high quality.

Summary of key considerations for top charities

 

The table below summarizes the key considerations for our four top charities. With the exception of modest changes to room for more funding, our high-level view of our top charities, as summarized in the table below, is the same as at our last update in November 2015.

Consideration AMF Deworm the World GiveDirectly SCI
Program estimated cost-effectiveness (relative to cash transfers) ~10x ~10x Baseline ~5x
Directness and robustness of the case for impact Strong Moderate Strongest Moderate
Transparency and communication Strong Strong Strongest Weakest
Ongoing monitoring and likelihood of detecting future problems Strong Strong Strongest Weakest
Organizational track record of rolling out program Moderate Moderate Strong Strong
Room for more funding High Limited High Likely moderate (not investigated)

 

Reasoning behind our current recommendation to donors

 

Our recommendation for donors seeking to directly follow our advice is to give to AMF, which we believe has the most valuable current funding gap. We believe AMF will likely have opportunities to fund distributions this year which it will not be able to fund without additional funding. Due to the excellent cost-effectiveness of AMF’s work, we consider this a highly valuable funding gap to fill. Our current estimate is that on average AMF saves a life for about every $3,500 that it spends; this is an increase from our November 2015 estimate and reflects changes to our cost-effectiveness model as well as some of our inputs into bed nets’ cost-effectiveness. As always, we advise against taking cost-effectiveness estimates literally and view them as highly uncertain.

The below table lays out our ranking of funding gaps for June to November 2016. The first million dollars to a charity can have a very different impact from, e.g., the 20th million dollars. Accordingly, our ranking of individual funding gaps accounts for both (a) the quality of the charity and the good accomplished by its program, per dollar, and (b) whether a given level of funding is highly or only marginally likely to be needed in the next six months.
We consider funding that allows a charity to implement more of its core program (without substantial benefits beyond the direct good accomplished by this program) to be “execution funding.” We’ve separated this funding into three levels:

  • Level 1: the amount we expect a charity to need in the coming year. If a charity has less funding than this level, we think it is more likely than not that it will be bottlenecked (or unable to carry out its core program to the fullest extent) by funding in the coming year. For this mid-year update, we have focused on funds that are needed before our next update in November, with the exception of SCI where we believe funds will not affect its work until next year.
  • Level 2: if a charity has this amount, we think there is an ~80% chance that it will not be bottlenecked by funding.
  • Level 3: if a charity has this amount, we think there is a ~95% chance that it will not be bottlenecked by funding.

(Our rankings can also take into account whether a gap is “capacity-relevant” or providing an incentive to engage in our process. We do not currently believe that our top charities have capacity-relevant gaps and are not planning to make mid-year incentive grants, so we haven’t gone into detail on that here. More details on how we think about capacity-relevant and execution gaps in this post.)

Priority Charity Amount (millions) Type Description Comment
1 AMF $11.3 Execution level 1 Fund distributions in two countries that AMF is in discussions with but does not have sufficient funding for AMF is strongest overall
2 AMF $7.3 Execution level 2 Fund the next largest gap on the list of remaining 2016-17 gaps in African countries –
3 SCI $10.1 Execution level 1 Very rough because we haven’t discussed this with SCI; further gaps not estimated Not as strong as AMF in isolation, so ranked below for same type of gap
4 AMF $10.5 Execution level 3 Fund the final two AMF-relevant gaps on the list of remaining 2016-17 gaps in African countries –
5 GiveDirectly $22.2 Execution level 1 Basic income guarantee program and additional standard transfers Not as cost-effective as bednets or deworming, so lower priority
6 Deworm the World $6.0 Execution level 3 A rough guess at the funding needed to cover a 3-year deworming program in a new country Strong cost-effectiveness, but unlikely to need funds in the short-term
6 GiveDirectly $7.8 Execution level 2 Funding for additional structured projects; further gaps not estimated –

 

We are not recommending that Good Ventures make grants to our top charities for this mid-year refresh. In November 2015, we recommended that Good Ventures fund 50% of our top charities’ highest-value funding gaps for the year and Good Ventures gave $44.4 million to our top four charities. We felt this approach resulted in Good Ventures funding its “fair share” while avoiding creating incentives for other donors to avoid the causes we’re interested in, which could lead to less overall funding for these causes in the long run. (More on this reasoning available here.)

What we’ve learned about SCI this year

(updated on: ) | by Andrew Martin

In past years, we’ve written that we had significant concerns about the financial reporting and financial management of the Schistosomiasis Control Initiative (SCI), one of our top charities since 2011. Our concerns have included:

  • We had not been able to learn important and basic financial information about SCI. Despite substantial effort, before 2016 we were not able to determine the total amount of funding that SCI held at any one time. We also had very little information on what SCI’s funds were spent on within country programs.
  • We found that SCI’s financial reports were prone to containing errors.

Due to these concerns, we decided to focus our research on SCI in preparation for our June 2016 top charities update only on the quality of its financial reporting and financial management. We felt that seeing significant improvements in the quality of SCI’s finances was necessary for us to continue recommending SCI.

Our main takeaways from our research on SCI so far in 2016:

  • SCI has begun producing higher-quality financial documents that allow us to learn some basic financial information about SCI, including the total amount of funding it holds, how much funding has been allocated to its upcoming budget year, and how it spent restricted and unrestricted funds by country in the previous budget year. We have also been able to learn somewhat more about how its funds are spent within national deworming programs.
  • We learned of two substantial errors in SCI’s financial management and reporting. 1) a July 2015 grant from GiveWell for about $333,000 was misallocated within Imperial College, which houses SCI, until we noticed it was missing from SCI’s revenue in March 2016; and (2) in 2015, SCI provided inaccurate information about how much funding it would have from other sources in 2016, leading us to overestimate its room for more funding by $1.5 million.
  • The clarity of our communication with SCI about these financial errors and its plans for the upcoming year has improved in comparison with previous years.

Details follow.

SCI’s financial documents in 2016

As of the beginning of April 2016, SCI had $15.8 million ($8.6 million in restricted funding and $7.2 million in unrestricted funding) available to allocate to its April 2016 to March 2017 budget year, according to its recent financial documents. Despite our discovery of additional financial errors this year (discussed below), we feel fairly confident that this information is accurate. We’ve seen transaction-level detail for each of SCI’s accounts, asked SCI’s new Finance and Operations Manager questions about the data, and largely received clear and reasonable answers.

SCI also sent us detailed breakdowns of in-country spending in its 2015-16 budget year for six of its country programs. Although this spending data gives us some information about what SCI’s funds were spent on within country programs last year, we note that we have not seen spending breakdowns for the eleven other deworming programs supported by SCI in 2015-16 (additional concerns about this spending data are discussed in our full review of SCI.)

Despite the improvements in SCI’s financial reporting that have allowed us to learn some basic financial information, we remain concerned about SCI’s use of Imperial College’s accounting system, which seems ill-suited to SCI’s needs. SCI has told us that it began using new accounting software in April 2016; we’re uncertain about the degree to which this will alleviate our concerns.

Financial errors we learned about in 2016

We’ve learned about two financial errors this year:

  • Not realizing that it had not received a transfer of funds from GiveWell: In July 2015, we granted $333,414 to SCI, which included all donations we received designated for supporting SCI between February and May 2015. After reviewing SCI’s financial documents in March 2016, we informed SCI that the July 2015 funding did not appear to be accounted for. After investigating the issue, SCI found that the funds had been misallocated by Imperial College to a different part of the college. SCI did not receive the funds until April 2016. SCI has asked Imperial College why the error occurred, but has not yet received a substantive response.
  • Underreporting available funding from DFID: In October 2015, SCI sent us its target treatment numbers for each national deworming program it supports, amounts of funding available from DFID and other large donors, and the amounts of additional funding required to deliver the targeted number of treatments and cover central expenditures for its April 2016 to March 2017 budget year. In March 2016, SCI sent GiveWell documents that indicated that around $1.5 million more funding was available from DFID to allocate to SCI’s 2016-2017 budget year than indicated in the October 2015 document. SCI told us that the October 2015 document included funding that was available from DFID to allocate to national deworming programs, but omitted $1.5 million in funding available from DFID to allocate to SCI’s central expenditures.

We consider both of these errors to be substantial. We are uncertain whether SCI would have ever received the funding from donations we collected on SCI’s behalf between February and May 2015 if we had not brought the issue to SCI’s attention. Our room for more funding analysis is a major factor in determining our funding recommendations to donors and to Good Ventures; an overestimation of SCI’s room for more funding by $1.5 million could have caused us to recommend donations to SCI that would have been better allocated to filling other funding gaps.

Our communication with SCI

Although we think that the financial errors we learned about in 2016 were substantial, we believe that it is a good sign that we were able to learn of these errors by communicating with SCI. In the past, we’ve noted that we’ve struggled to communicate effectively with SCI’s representatives, which sometimes meant that we were unable to clear up our confusion about inconsistencies we found in SCI’s documents.

We also feel that we’ve communicated clearly with SCI about its plans for the upcoming year and gained a better understanding of the factors that limit the delivery of additional deworming treatments in different contexts.

Bottom line

Given the improvements, we continue to recommend SCI now and think that SCI is a contender for a top charity recommendation at the end of 2016. We plan, in the second half of 2016, to expand the scope of our research on SCI to include looking at recent monitoring and evaluation, cost per treatment, and room for more funding in 2017 and beyond. We continue to have some concerns about SCI’s financial reporting and management (most notably, the errors noted above) and will be following up with SCI about our outstanding questions.

June 2016 open thread

(updated on: ) | by Catherine Hollander

Our goal with hosting quarterly open threads is to give blog readers an opportunity to publicly raise comments or questions about GiveWell or related topics (in the comments section below). As always, you’re also welcome to email us at info@givewell.org or to request a call with GiveWell staff if you have feedback or questions you’d prefer to discuss privately. We’ll try to respond promptly to questions or comments.

If you have questions related to the Open Philanthropy Project, you can post those in the Open Philanthropy Project’s open thread.

You can view our March 2016 open thread here.