What we’ve learned about SCI this year

In past years, we’ve written that we had significant concerns about the financial reporting and financial management of the Schistosomiasis Control Initiative (SCI), one of our top charities since 2011. Our concerns have included:

  • We had not been able to learn important and basic financial information about SCI. Despite substantial effort, before 2016 we were not able to determine the total amount of funding that SCI held at any one time. We also had very little information on what SCI’s funds were spent on within country programs.
  • We found that SCI’s financial reports were prone to containing errors.

Due to these concerns, we decided to focus our research on SCI in preparation for our June 2016 top charities update only on the quality of its financial reporting and financial management. We felt that seeing significant improvements in the quality of SCI’s finances was necessary for us to continue recommending SCI.

Our main takeaways from our research on SCI so far in 2016:

  • SCI has begun producing higher-quality financial documents that allow us to learn some basic financial information about SCI, including the total amount of funding it holds, how much funding has been allocated to its upcoming budget year, and how it spent restricted and unrestricted funds by country in the previous budget year. We have also been able to learn somewhat more about how its funds are spent within national deworming programs.
  • We learned of two substantial errors in SCI’s financial management and reporting. 1) a July 2015 grant from GiveWell for about $333,000 was misallocated within Imperial College, which houses SCI, until we noticed it was missing from SCI’s revenue in March 2016; and (2) in 2015, SCI provided inaccurate information about how much funding it would have from other sources in 2016, leading us to overestimate its room for more funding by $1.5 million.
  • The clarity of our communication with SCI about these financial errors and its plans for the upcoming year has improved in comparison with previous years.

Details follow.

SCI’s financial documents in 2016

As of the beginning of April 2016, SCI had $15.8 million ($8.6 million in restricted funding and $7.2 million in unrestricted funding) available to allocate to its April 2016 to March 2017 budget year, according to its recent financial documents. Despite our discovery of additional financial errors this year (discussed below), we feel fairly confident that this information is accurate. We’ve seen transaction-level detail for each of SCI’s accounts, asked SCI’s new Finance and Operations Manager questions about the data, and largely received clear and reasonable answers.

SCI also sent us detailed breakdowns of in-country spending in its 2015-16 budget year for six of its country programs. Although this spending data gives us some information about what SCI’s funds were spent on within country programs last year, we note that we have not seen spending breakdowns for the eleven other deworming programs supported by SCI in 2015-16 (additional concerns about this spending data are discussed in our full review of SCI.)

Despite the improvements in SCI’s financial reporting that have allowed us to learn some basic financial information, we remain concerned about SCI’s use of Imperial College’s accounting system, which seems ill-suited to SCI’s needs. SCI has told us that it began using new accounting software in April 2016; we’re uncertain about the degree to which this will alleviate our concerns.

Financial errors we learned about in 2016

We’ve learned about two financial errors this year:

  • Not realizing that it had not received a transfer of funds from GiveWell: In July 2015, we granted $333,414 to SCI, which included all donations we received designated for supporting SCI between February and May 2015. After reviewing SCI’s financial documents in March 2016, we informed SCI that the July 2015 funding did not appear to be accounted for. After investigating the issue, SCI found that the funds had been misallocated by Imperial College to a different part of the college. SCI did not receive the funds until April 2016. SCI has asked Imperial College why the error occurred, but has not yet received a substantive response.
  • Underreporting available funding from DFID: In October 2015, SCI sent us its target treatment numbers for each national deworming program it supports, amounts of funding available from DFID and other large donors, and the amounts of additional funding required to deliver the targeted number of treatments and cover central expenditures for its April 2016 to March 2017 budget year. In March 2016, SCI sent GiveWell documents that indicated that around $1.5 million more funding was available from DFID to allocate to SCI’s 2016-2017 budget year than indicated in the October 2015 document. SCI told us that the October 2015 document included funding that was available from DFID to allocate to national deworming programs, but omitted $1.5 million in funding available from DFID to allocate to SCI’s central expenditures.

We consider both of these errors to be substantial. We are uncertain whether SCI would have ever received the funding from donations we collected on SCI’s behalf between February and May 2015 if we had not brought the issue to SCI’s attention. Our room for more funding analysis is a major factor in determining our funding recommendations to donors and to Good Ventures; an overestimation of SCI’s room for more funding by $1.5 million could have caused us to recommend donations to SCI that would have been better allocated to filling other funding gaps.

Our communication with SCI

Although we think that the financial errors we learned about in 2016 were substantial, we believe that it is a good sign that we were able to learn of these errors by communicating with SCI. In the past, we’ve noted that we’ve struggled to communicate effectively with SCI’s representatives, which sometimes meant that we were unable to clear up our confusion about inconsistencies we found in SCI’s documents.

We also feel that we’ve communicated clearly with SCI about its plans for the upcoming year and gained a better understanding of the factors that limit the delivery of additional deworming treatments in different contexts.

Bottom line

Given the improvements, we continue to recommend SCI now and think that SCI is a contender for a top charity recommendation at the end of 2016. We plan, in the second half of 2016, to expand the scope of our research on SCI to include looking at recent monitoring and evaluation, cost per treatment, and room for more funding in 2017 and beyond. We continue to have some concerns about SCI’s financial reporting and management (most notably, the errors noted above) and will be following up with SCI about our outstanding questions.

June 2016 open thread

Our goal with hosting quarterly open threads is to give blog readers an opportunity to publicly raise comments or questions about GiveWell or related topics (in the comments section below). As always, you’re also welcome to email us at info@givewell.org or to request a call with GiveWell staff if you have feedback or questions you’d prefer to discuss privately. We’ll try to respond promptly to questions or comments.

If you have questions related to the Open Philanthropy Project, you can post those in the Open Philanthropy Project’s open thread.

You can view our March 2016 open thread here.

We’re hiring a Director of Operations

GiveWell’s Operations team is responsible for all finance, accounting, HR, legal and tech functions at GiveWell and the Open Philanthropy Project. We started to build this team over the past year, and we’re now looking for a proven leader to lead our current team and grow it for the future. Additional details about the role are in the job posting.

At a high level, the person we hire will:

  • lead a team of 5-10 people to ensure that necessary business activities continue to function.
  • improve the systems we have and, where necessary, create new ones so that our operational work runs excellently.
  • over time, potentially grow into a senior leadership role at GiveWell and Open Philanthropy, helping the leaders of the organization set strategy and the future direction for both organizations.

We’re looking for someone with a demonstrably strong track record as a manager, ideally someone who has successfully a managed a team of at least several people for at least 1-3 years.

Instructions for applicants are available in the Director of Operations job posting.

Trying out a new format: GiveWell podcast

June 17, 2016 update: we’ve received relatively few responses to our survey about the podcast, so we’re currently not planning on making more episodes. If you listened, feel free to fill out the survey or contact us.

As part of our increased outreach efforts this year, we decided to make three test episodes of a podcast. We’re aiming to quickly and easily share our research in a new form that might be more accessible for many people.

We’ve recorded three episodes, and we’re releasing them now to gauge our audience’s reaction. If there’s strong interest, we’ll keep making episodes.

The three episodes:

  • Episode 1 – About GiveWell

  • Episode 2 – New employees and working at GiveWell (we suspect this may be interesting to people considering working at GiveWell, and of limited interest to others)

  • Episode 3 – The Against Malaria Foundation

The podcasts can be downloaded or streamed from the links above, and are also available for podcast apps via iTunes or by copying our RSS feed into your favorite podcast program:

http://feeds.feedburner.com/GivewellPodcast

Some notes about this podcast:

  • The goal of this podcast is not to be a highly produced, well-polished product; rather, we aimed to create recordings which would be informative and interesting to our audience but don’t require a large time investment from us.
  • The format of this podcast is roughly half hour-long conversations about research we’ve done or other topics relevant to our work, with minimal scripting or editing.
  • If we continue making these, it’s possible we’d put some resources into improving the sound quality, but in general the format would be similar: casual, largely unscripted conversations with GiveWell and Open Philanthropy Project staff about our research findings, their areas of expertise, or other topics relevant to our work.

If you do listen to the podcast, we’d appreciate it if you would answer this very short survey about it. Doing so will help us decide whether to continue producing it.

GiveWell’s money moved and web traffic in 2015

GiveWell is dedicated to finding outstanding giving opportunities and publishing the full details of our analysis. In addition to evaluations of other charities, GiveWell publishes substantial evaluation of our own work. This post lays out highlights from our 2015 metrics report, which reviews what we know about how our research impacted donors. Please note:

  • We report on “metrics years” that run from February through January; for example, our 2015 data cover February 1, 2015 through January 31, 2016.
  • We differentiate between our traditional charity recommendations, our work on the Open Philanthropy Project, and other charitable giving.
  • More context on the relationship between Good Ventures and GiveWell can be found here.

Summary of influence: In 2015, GiveWell influenced charitable giving in several ways. The following table summarizes our understanding of this influence.
Table_Summary.png

Total money moved: In 2015, GiveWell tracked $110.1 million in money moved to our recommended charities. Our money moved only includes donations that we are confident were influenced by our recommendations.

Chart_MoneyMoved.png

Open Philanthropy Project: As part of our work on the Open Philanthropy Project, we advised Good Ventures to make grants totaling $6.4 million. This was in addition to Good Ventures’ support for our recommended charities.

Money moved by charity: Our four top charities received the majority of our money moved. Our four standout charities received a total of $2.2 million.

Table_ByCharity.png

Money moved by size of donor: In 2015, the number of donors and amount donated increased across each donor size category. In 2015, 95% of our money moved (excluding Good Ventures) came from about 15% of our donors, each of whom gave $1,000 or more.

Table_ByDonorSize.png

Donor retention: The total number of donors who gave to our recommended charities or to GiveWell unrestricted increased about 60% year-over-year to 15,274 in 2015. This included 10,669 donors who gave to our recommended charities for the first time. Among all donors who gave in the previous year, about 40% gave again in 2015, up from about 33% who gave again in 2014.

Table_Retention.png

Our retention was stronger among donors who gave larger amounts or who first gave to our recommendations prior to 2013. Of larger donors (those who gave $10,000 or more in either of the last two years), about 80% who gave in 2014 gave again in 2015.

Table_Retention10k.png

GiveWell’s expenses: GiveWell’s total operating expenses in 2015 were $3.4 million. Our expenses increased from about $1.8 million in 2014 as the size of our staff grew, largely to support the Open Philanthropy Project. We estimate that about one-third of our total expenses ($1.1 million) supported our traditional top charity work and about two-thirds supported the Open Philanthropy Project. In 2014, we estimated that expenses for our traditional charity work were about $900,000.

Donations supporting GiveWell’s operations: Prior to 2013, GiveWell relied on a small number of donors to provide unrestricted support for our operations. Since 2013, we have asked more donors to support our operational costs and asked donors to support us at a higher level than we had in previous years. In 2015, we raised $4.9 million, up from $3.0 million in 2014. Several institutions and the six largest individual donors contributed about two-thirds of GiveWell’s funding in 2015.

Table_Unrestricted.png

Donor demographics: We continued to collect information on our donors. We found the picture of our 2015 donors to be broadly consistent with previous information. Based on reports from donors who gave $2,000 or more, we found:

  • The most common ways that donors found us were via Peter Singer and personal referrals.
  • About 70% of our donors are under 40, and about 60% work in technology or finance.

Web traffic: Unique visitors to our website increased by 12% in 2015 compared to 2014 (when excluding visitors driven by AdWords, Google’s online advertising product).

Chart_WebTraffic.png

For more detail, see our full metrics report (PDF).

Reservations about water quality interventions

When we started researching water quality interventions, we originally guessed that there was clear evidence that the programs were effective at improving people’s health based on our general intuitions about the programs and our initial read of the relevant Cochrane review’s bottom line. Once we dug into the details of the research, though, we realized that the evidence regarding the program is mixed — a major reason that we don’t currently recommend charities that work on improving water quality.

This post describes that research, and is drawn from our full report on water quality interventions, which provides more detail and citations. The goal of this blog post is to summarize that report in a more accessible form.

The key points of this post:

  • We formed an initial impression that water quality interventions (e.g., chlorine tablets or dispensers; water filters; other types of treatment that clean drinking water in the developing world) improve people’s health, based on our understanding of the dangers of contaminated water. However, digging into the evidence regarding clean water led us to conclude that the evidence base is mixed.
  • We’re not confident that water quality interventions are not effective; rather, we believe that it’s unclear whether or not they’re effective, and so we don’t include water quality among our priority programs.

The intuitive case for water quality charities

Intuitively, providing a method for cleaning water seems like it should be an effective way to improve health. There is little debate that contaminated water causes diarrhea or that the methods used to treat water reduce contamination, and so it’s reasonable to expect that interventions that reduce water contamination ought to reduce diarrhea. And because diarrhea is deadly – the Global Burden of Disease estimated that in 2010, diarrheal disease caused about 10% of child deaths in developing countries – these interventions could be potentially life-saving.

Developing world randomized controlled trials

A number of independent studies have been done on water quality programs. The results of those studies look very promising: They largely find that providing clean water reduces diarrhea rates.

Specifically, 20 randomized controlled trials conducted in developing countries with around 30,000 participants found that interventions to improve water quality were effective at preventing diarrhea, with an average 35% reduction in the odds of getting diarrhea, according to a meta-analysis (study of studies) conducted by the Cochrane Collaboration. Relative to most interventions we’ve considered, twenty randomized controlled trials is an unusually high number of studies.

We typically consider randomized controlled trials to be a particularly strong method of studying global health outcomes. These studies compare a group that receives an intervention – in this case, some method for cleaning water, like chlorine tablets – with a control group that doesn’t. The intervention and control groups are randomly chosen to try to ensure that two groups are similar in every way other than whether or not they receive the intervention. This way, any differences between the groups ought to be due to the intervention.

The Cochrane Collaboration’s findings, then, strongly suggest that the cleaner water is responsible for the fall in diarrhea rates.

Possible weakness of the randomized controlled trials

When we examined these studies more closely, we noticed a potential weakness. As the Cochrane review states: “Only four of the 22 randomised controlled studies, however, were properly blinded”. (22 because the Cochrane review also considers 2 studies conducted in the United States, which we excluded from our analysis.) In other words, the researchers conducting these studies measured diarrhea rates by asking participants to report them, and the study participants knew whether or not they had received the intervention that was being tested.

It’s possible that this knowledge influenced how participants reported diarrhea cases. For example, participants who knew that their water was being treated may have been less likely to report cases of diarrhea, perhaps because they recalled using the treatment and misremembered how many cases they had, or because they anticipated what the surveyor wanted to hear. If that were the case, the study would find an inaccurately low number of diarrhea cases among people that received the clean water intervention, which would result in the study finding an inflated effect of the intervention.

To deal with the issue of participants knowing whether or not they received the intervention (and the possibility that this influences reporting), studies can use “blinding” to disguise who has received the intervention by distributing a placebo (for example, a fake, non-functional water filter) to the control group. When this method is used, participants don’t know whether they have the fake filter or the real one, and so any bias in how they report their cases of diarrhea ought to affect both groups equally, so we can still measure the difference between the two groups without this source of bias affecting the outcome.

Blinded randomized controlled trials should let us know whether the tested intervention is actually improving people’s health, or whether other sources of harmful pathogens are negating any positive effect of the clean water programs. If the latter were true, we may not expect supporting the intervention to have a beneficial impact on the population.

Blinded studies

We identified five blinded studies of water quality interventions that have been conducted in developing countries (which includes some published after the Cochrane review). These studies measured household interventions to clean water; four studied chlorination and one examined the impact of water filters. They did not find a statistically significant effect of the interventions on diarrhea. (Pooling the effects of the studies, we calculated a 5% reduction in the percent of days with diarrhea for children under 5, with a 95% confidence interval ranging from a 6% increase to a 14% decrease, and no reduction for all ages with a 95% confidence interval ranging from a 12% increase to an 11% decrease.)

This suggests that biased reporting from study participants who knew whether or not they were receiving an intervention may explain why the non-blinded studies in the Cochrane review found a statistically significant effect – that is, it suggests that people in the non-blinded studies may have underreported diarrhea rates when they knew that they were receiving a treatment, but when people were unsure whether or not they were receiving a treatment, there was no reporting bias and so no statistically significant effect was found.

Possible weaknesses of the blinded studies

However, these blinded studies have their own weaknesses. In one, the fake filter used for the control group was actually somewhat effective at cleaning the water, meaning that the control group (which wasn’t supposed to be receiving any clean-water benefits) received water that was at least partially cleaned. It’s not surprising, then, that the study didn’t find a noticeable difference between the diarrhea rates of the two groups.

Two of the other blinded studies were conducted in groups that already had low levels of diarrheal disease, which may have made it harder to detect a difference between the control and intervention groups. The other two had small sample sizes, of 112 (the smallest included in the Cochrane review) and 287 people. (Additional problems with the blinded studies are discussed here.)

These problems suggest that the blinded studies are not conclusive in determining that there’s no statistically significant effect of providing chlorine and water filters to reduce diarrhea rates – each of those problems might explain why the blinded studies found no statistically significant effect, even if the intervention itself is actually beneficial.

(There are a handful of other, different studies that we do not find compelling – you can read more about those here.)

How could these interventions fail to reduce diarrhea rates?

Considering the scientific consensus that contaminated water can cause diarrhea, one could reasonably ask whether it’s even plausible that water quality interventions have small or negligible effects (as opposed to large effects). Is it? There is a potential explanation: Other pathways for pathogens, such as from fecal matter on unwashed hands or from food via flies or other people’s hands, especially in an environment with poor sanitation, may cause a significant number of diarrhea cases, so it’s possible that the gains from simply providing clean water could be small if people are regularly infected via other sources. In fact, it’s possible that water quality interventions might have little effect on diarrhea unless all sources of pathogens are eliminated. The possibility that providing clean water will have little effect in an environment with other sources of pathogens is especially concerning because the blinded studies found no statistically significant effect of the intervention.

Bottom line

We concluded in our intervention report (emphasis included for this post),

The strength of the evidence relies on how one chooses to weigh a few, blinded studies with some methodological weaknesses that report no effect against a large number of unblinded studies, some of which may also have some methodological weaknesses, that, on average, report large effects… Overall, we are ambivalent about the effect of water quality interventions on diarrhea. We find plausible theories grounded in the available evidence for both believing that water quality interventions reduce diarrhea and for the more pessimistic conclusion that these interventions do not have an effect.

Based on the ambiguity of the results, and the fact that it’s plausible the interventions on their own do not have an effect, we currently do not consider water quality interventions to be evidence-backed and do not include them among our priority programs. This doesn’t mean we’re confident that the interventions don’t work, but it does mean that we’re not confident that they do.

Note that Dr. Alix Zwane, formerly the Executive Director of Evidence Action (which runs one of our top charities, the Deworm the World Initiative, in addition to a charity which conducts water quality programs, Dispensers for Safe Water) and now CEO of the Global Innovation Fund, disagrees with our conclusion and has a more positive view of the benefits of water quality intervention. Her comments on our report, along with comments by Thomas Clasen, one of the authors of the Cochrane report, can be found here.

Comparison to other programs we recommend

While the evidence for water quality interventions is mixed, it is stronger than for most other interventions that we’ve come across. Depending on what you believe about the weaknesses of the evidence base, it’s arguably as good as the evidence for deworming, which two of our top charities focus on. Deworming also has a complicated evidence base which we have written about extensively, most recently here.

Our initial estimates, however, found that water quality interventions wouldn’t be as cost-effective as deworming programs, and so recently we have not prioritized further research into water quality charities.

That said, because this evidence base is stronger than most, it’s possible that a strong water quality charity could qualify as one of our standout charities, even though we would expect to still recommend our current top charities more highly. A strong water quality charity could even one day become one of our top charity recommendations, especially if our current top charities run out of room for more funding.