Deworm the World Initiative (led by Evidence Action) update

Summary

The Deworm the World Initiative (DtWI), led by Evidence Action, received approximately $2.3 million as a result of GiveWell’s recommendation last year. While there were some deviations, it largely allocated these funds as we expected.

DtWI now has limited room for funding; it is currently seeking to raise an additional $1.3 million to support its activities in 2015 and 2016. We expect it to allocate approximately 30% of the additional funds it receives for work related to expanding school-based, mass deworming programs and funding related operating expenses (including impact evaluation related expenses), and will allocate other funds to priorities that are less directly connected to expanding and evaluating deworming programs (investigating ways to combine other evidence-based programs with deworming rollouts, supplementing a project supported by another funder).

We currently expect to release updated recommendations by December 1st. We think it is likely that the Deworm the World Initiative will remain on our top charities list.

How did DtWI spend the money it received due to GiveWell, and how does this compare to our expectations?

GiveWell directed approximately $2.3 million to the Deworm the World Initiative since we added it to our top charities list in December 2013.

At the time of our recommendation, we expected DtWI to spend additional funds in the following ways; we did not have precise estimates for how much it would spend in each category:

  • Some portion to provide reserves for DtWI, both to make the organization more resilient and to allow it to respond to high impact opportunities
  • Some portion to allow DtWI to offer a lower-intensity level of assistance to regions that didn’t require its standard level of assistance
  • Some portion to support expansion to new states in India

It has allocated these funds as follows (years when we expect funds to be spent in parentheses; 2014 means funds have been spent):

  • $881,000 – ongoing reserves. Our understanding is that DtWI does not have plans to spend these funds in the near future. Instead, these funds make DtWI more robust as an organization: for example, it is less likely to need to significantly shift priorities in order to fundraise and it is more likely to be able to respond quickly to high-impact opportunities it identifies.
  • $509,000 – expansion into new countries (2015 and 2016). This includes preliminary work in Ethiopia, Indonesia, Philippines to support possible future work and $104,000 for prevalence surveys and technical assistance to the government and partner organization in Vietnam.
  • $430,000 – ongoing work in India (2014 and early 2015). This will fund a follow-up prevalence survey in Bihar to assess the impact of three rounds of deworming on worm prevalence and intensity, and enable expansion to preschool children there, as well as contribute to the third round of the Rajasthan and Delhi programs.
  • $207,000 – contribution to elimination research primarily funded by the Children’s Investment Fund Foundation (CIFF) and the Bill and Melinda Gates Foundation (BMGF) (2015-2017). CIFF and BMGF provided approximately $1.6 million in funding to the Deworm the World Initiative and the London School of Hygiene and Tropical Medicine to conduct research on the feasibility and cost effectiveness of breaking transmission of soil-transmitted helminths. Breaking transmission would potentially require a different approach (likely covering more than just school-aged children) than DtWI’s standard school-based deworming model.
  • $151,000 – DtWI overhead (2014). These funds support DtWI as an organization but are not directly programmed (e.g., a portion of Alix Zwane’s, the Executive Director of Evidence Action salary, Evidence Action financial staff, etc.). Note that DtWI estimated $151,000 based on allocating 15% of programmed GiveWell-sourced funding to DtWI overhead. DtWI said it could more explicitly track these funds but would be time consuming to do so. We agreed that more detailed accounting was not necessary.
  • $129,000 – additional staff (2014). In 2014, DtWI hired (a) a deputy director to support its programming worldwide and (b) someone to focus on its impact evaluation. The latter hire is likely to be doing work on the breaking transmission work discussed below. We allocate some of this line item to expansion and related operating expenses and some to research.

Overall, DtWI’s funding decisions seem reasonable to us and are broadly consistent with what we anticipated.

  • 46% ($1,067,000) supported expanding deworming programs and funding related operating expenses (including impact evaluation related expenses). This includes the deputy director who supports the organization as a whole but is necessary to expanded work in India and other new countries and half of the salary for the impact-evaluation-focused new staffer since he works on programmatic and technical support across DtWI.
  • 38% ($881,000) supported ongoing reserves.
  • 10% ($241,000) supported research that we had not anticipated (including the other half of the new staffer since he is spending a significant part of his time on this research).
  • 6% (151,000) supported DtWI as a whole.

How would DtWI spend additional funds?

The Deworm the World Initiative seeks an additional $1.3 million to support its activities in 2015 and 2016. DtWI expects to spend $377,000 of the $1.3 million (29%) it seeks on work related to expanding school-based mass deworming programs and funding related operating expenses (including impact evaluation related expenses). More specifically, these activities would be:

  • $230,000: staff to support expansion in India, new countries, and related operating and evaluation expenses. This line item is the salary for the deputy director and part of the salary for the impact evaluation focused staff member described above.
  • $144,000: DtWI overhead (described above).
  • $500,000: evaluation of new evidence-based programs that leverage deworming. We have limited detail about what this would entail. One idea that DtWI has investigated is the possibility of distributing bednets along with deworming pills in schools as an alternative distribution mechanism to national net distributions. Another is including hand-washing educational programming alongside deworming days. This line item includes $50,000 to support DtWI’s evaluation of its hygiene and deworming program funded by Dubai Cares and $50,000 to enable DtWI to hire a senior epidemiologist.
  • $230,000: staff to support evaluation of DtWI’s work in Kenya. This work is primarily funded by CIFF. DtWI believes that additional resources can improve significantly the quality of the analysis done regarding the cost effectiveness of breaking transmission. This line item includes $100,000 to support the impact evaluation focused staff member described above.
  • $170,000: implementation support for the integrated deworming, sanitation and hygiene education program in Vietnam, in partnership with Thrive Networks.

Why is DtWI seeking additional funds primarily to support research and evaluation rather than scale up? What changed in the past year?

In 2014, two events affected DtWI’s projection of the additional funding it would require to scale up in India:

  1. The Children’s Investment Fund Foundation (CIFF), a major foundation that had supported DtWI’s programs in Kenya, agreed to a 6-year, $17.7 million grant to support DtWI’s expansion to additional states in India and technical assistance to the Government of India for a national deworming program. At the end of 2013, DtWI believed it was reasonably likely that it would not receive this grant and had not anticipated how quickly it would come through. With these funds, DtWI does not require significant additional funding to support its India expansion.
  2. The new Indian government expressed interest in conducting a single deworming day nationally with increased national attention and resources. Advocating for such a policy and assisting the national government in creating a plan became the major focus of DtWI’s India work in 2014, which both reduced the amount of time it was able to spend generating interest in heavy DtWI involvement in new states, and also required little funding since there were few costs of that project aside from staff time. DtWI believes that the first national deworming day will likely happen in February 2015.

Together, these changes led DtWI to the conclusion that funding is no longer the bottleneck to reaching more people in India.

More broadly, we believe that if donors close both the $1.3-million 2-year funding gap of DtWI and the ~5-$8-million funding gap of the Schistosomiasis Control Initiative (SCI), another deworming organization we recommend, funding will not be the primary bottleneck to deworming programs’ scaling in general. Overall, our impression is that there is currently more funding available for scaling up deworming programs than capacity at organizations to utilize funds for scaleup.

Dr. Zwane believes that DtWI’s research agenda is important for two reasons:

  1. She believes it is possible that this research will demonstrate that other approaches to deworming are more cost-effective, such as eliminating worms from areas to avoid the need for mass treatments, or combining deworming with other interventions such as bednet distributions or hygiene education.
  2. She would like DtWI to consistently provide useful information to funders and policymakers and undertaking this research will enable it to continue doing so.

Notes on other deworming implementers and funders

It is not unlikely that GiveWell-directed donors will close the funding gaps of both DtWI and Schistosomiasis Control Initiative in the coming few months. Because of this, we also asked Alix Zwane (Executive Director of Evidence Action) about other implementers and funders working on deworming.

Implementers

Dr. Zwane told us DtWI and SCI are the two primary organizations that focus primarily on expanding countrywide deworming programs. Other organizations work on deworming but are not as directly focused on scaleup with government partners to her knowledge. There are other NGOs that work on other neglected tropical diseases (e.g., SightSavers) and school health (e.g., Partnership for Child Development), but Dr. Zwane is less familiar with the reach and scope of the service delivery they support.

Organizations that do a smaller amount of deworming implementation include UNICEF, Micronutrient Initiative and Vitamin Angels, which have begun adding deworming pills to their vitamin A supplementation programs, and WaterAid, which adds deworming to some of its water and sanitation programs.

IMA World Health, Helen Keller International, Sight Savers International, The MENTOR Initiative, and possibly others implement deworming programs supported by the funders discussed below. We have yet to speak with these organizations and have little information about their deworming programs or funding needs.

According to Dr. Zwane, the Global Network for Neglected Tropical Diseases works primarily on advocacy and does not focus on deworming, specifically, while Children without Worms coordinates partners globally and does not work on providing technical assistance for program delivery directly currently to her knowledge.

Funders

Major funders of deworming service delivery include the following: Dubai Cares, The END Fund, CIFF, the UK government’s Department for International Development (DFID), Michael and Susan Dell Foundation, and the US government’s USAID.

According to Dr. Zwane, these funders are interested in supporting scale-up, and she believes that DtWI will be in a strong position to raise funds for scale up from them if and when funding becomes a bottleneck. These funders are less likely to fund the types of activities to which DtWI has allocated GiveWell-directed funding.

A longer list of organizations working on deworming is available in this document, from a recent meeting of groups that are part of the STH Coalition.

Inaugural open thread

A commenter recently suggested we try out a periodic open thread. Let’s give it a shot!

Our goal is to give blog readers an opportunity to publicly raise comments or questions about GiveWell or related topics (in the comments section below). As always, you’re also welcome to email us at info@givewell.org if there’s feedback or questions you’d prefer to discuss privately. We’ll try to respond promptly to questions or comments.

If this is helpful, we’ll continue to do them periodically in the future.

Quick update on Ebola (full update to come later)

At the moment, we don’t see highly time-sensitive giving opportunities that we are likely to recommend around responding to the Ebola outbreak in west Africa. We still plan to put out a detailed blog post describing what we’ve done to investigate the situation and what we think the best options for donors are, but we have now postponed this post in order to prioritize work on top charities. We hope to have the Ebola post out by early December.

Our ongoing review of GAIN

Note: GAIN has reviewed a draft of this post and added responses below.

The Global Alliance for Improved Nutrition (GAIN) focuses on reducing malnutrition by fortifying foods and condiments with essential nutrients and through other interventions. We have been considering GAIN for a 2014 recommendation for its work on Universal Salt Iodization (USI) because of our impression that iodization has strong evidence of effectiveness, cost-effectiveness, and room for more funding (our take on salt iodization is in our post on ICCIDD), and our understanding that GAIN is one of only a few organizations working on salt iodization on a large scale in many different countries. We decided to prioritize GAIN after speaking with Greg S. Garrett from GAIN as part of our investigation of ICCIDD.

We’ve now spent a considerable amount of time talking to GAIN and analyzing documents GAIN shared with us. This post shares what we’ve learned so far and what questions we’re planning to focus on throughout the rest of our investigation. (For more detail, see our detailed interim review.)

Broadly, GAIN’s USI work includes advocacy, technical assistance, supplying equipment and building supply systems for potassium iodate and iodized salt, training government officials and salt producers, and monitoring coverage and impact, among other activities.

GAIN has completed the first phase of our investigation process and we view it as a contender for a future recommendation. We now plan to make a $100,000 grant to GAIN (as part of our “top charity participation grants,” funded by Good Ventures). However, we are planning to prioritize work on other organizations first because we think that will yield a higher return on our time. GAIN is a challenging organization for us to evaluate, because its work (e.g., training/monitoring) is often several steps removed from increases in the impact we hope to see (e.g., improvements in iodization rates); we have evaluated and recommended organizations like this before, but with the information we currently have from GAIN, we feel we are still far from a full understanding of GAIN’s work and impact.

Questions we hope to answer in our ongoing analysis

We feel that we are still at a relatively early stage of our investigation of GAIN and have many important, open questions.

What’s GAIN’s track record at improving USI programs?

There are two components of this question, and we have only limited information on both: (1) in the countries that GAIN has worked, is a higher percentage of salt being iodized at the correct concentration than it was previously, and/or have measures of iodine levels in the population improved? and (2) what evidence is there that GAIN caused any improvements that are observed?

On (1), the trend seems quite mixed: in a set of nine countries, use of adequately iodized salt decreased in three countries, increased by a small amount in four countries, and increased substantially in two countries. Were we to rely on this data, we would need more information on the data sources. The data is not necessarily comparable within or across countries due to different survey design and salt iodine assessment methodology.

GAIN’s response: We have advocated globally and ensured that within our target countries a sub-sample of salt will be tested for iodine using a recognized quantitative method in any assessment taking place. This is reflected in the fact that eight of the nine update assessments measured salt iodine quantitatively, and all forthcoming (2015) end of project assessments will include quantitative analysis for the entire salt sample.

In the interim, the data we have access to (from surveys and other reliable sources of information since project start) for nine of our target countries show that baseline coverage of adequately iodized salt was 72.6%. This coverage has increased to 80.3% in these nine countries over the project period to date, representing an increase in the reach of adequately iodized salt to approximately 195 million people, mostly in China, Ethiopia, India and Pakistan.

We’ve looked more closely at iodization trends in four countries that GAIN highlighted to us in case studies of its work. In Bangladesh and India, rates of adequately iodized salt increased by a small amount (4 to 7 percentage points). In Ethiopia, GAIN reports that iodization rates have roughly doubled (from 20 to 40%) over a period that included GAIN’s involvement in the country (but follow up data is incomplete). In the fourth country, the Philippines, GAIN has only provided data for a group of salt producers it worked with, and found that samples meeting iodization standards increased from less than a third to over 80%.

We put limited weight on these claims - particularly in the last two examples - because we have not seen details of how the data was collected.

GAIN’s response: In India, which is self-sufficient in salt supply, the most recent data are based on supply QC reports from the national salt management information system, the development of which was a major achievement of GAIN support to the Salt Dept. This system has greatly facilitated obtaining timely and higher quality data, comparable over time, on iodized salt supply and quality. The system and rapid review time also increases accountability of salt producers as well as regulatory monitoring staff. How the supply QC data relate to household access to iodized salt will be determined after the national survey which will be conducted before the end of 2014.

2013 data for Ethiopia were presented at the Micronutrient Forum by the Ethiopian Public Health Institute. These salt iodine data are from a sample of over 5,000 households, selected from 354 enumeration areas to provide regionally and nationally representative data. The full national micronutrient survey, conducted by the end of 2014, will repeat the collection of salt samples and also include assessment of iodine status. It is expected that this will have improved considerably, in line with increased access to iodized salt, since the very low baseline. GAIN has been a key player working with the salt producers in challenging conditions to establish a sustainable KIO3 supply, develop iodization technology and is currently working on these as well as on improving quality control and regulatory monitoring of the salt supply with FMHACA.

For (2), linking GAIN to improvements in specific countries will be difficult because its activities (such as technical assistance, monitoring, training, etc.) tend to be a few steps removed from the outcomes of improved salt iodization and iodine consumption rates. Furthermore, GAIN generally works in collaboration with other groups (more below), making attribution to one party difficult or impossible. For more details on GAIN’s activities and reported accomplishments in these countries, see our interim review.

GAIN’s response: GAIN focuses its work primarily on proven drivers of supply which are mapped against the 2011 WHO/CDC logic model for micronutrient interventions in public health.  GAIN works closely with partners (government, private sector and other agencies in particular UNICEF) and therefore attributing progress to one agency alone and set of activities can be challenging. 

Do we have a comprehensive understanding of GAIN’s USI activities?

We have found it challenging to understand what, specifically, GAIN’s work on USI involves. This is because GAIN’s activities on USI vary considerably across countries and it works in 16 countries, and GAIN works closely with UNICEF, ICCIDD, and governments on this work, making it difficult to understand GAIN’s role. To date we have focused on understanding GAIN’s work in 4 countries because (a) GAIN has shared the most information for these countries, and (b) together they account for roughly 50% of its primary grant for USI.

Details in our interim review of GAIN.

GAIN’s response: As explained above, GAIN focuses its work primarily on proven drivers of supply which are mapped against the 2011 WHO/CDC logic model for micronutrient interventions in public health.  GAIN works closely with partners (government, private sector and other agencies in particular UNICEF) and therefore attributing progress to one agency alone and set of activities can be challenging.

Specifically, in addition to ongoing policy and advocacy and monitoring of national programs (e.g. designing and delivering coverage surveys) GAIN’s work can be summarized by the following four proven supply drivers: 1) improving quality assurance quality control/regulatory monitoring/improving testing capability; 2) integrating the use of adequately iodized salt in the food industry; 3) designing models for sustained KIO3 procurement; and 4) delivering innovations along the supply chain which apply positive pressure on iodized salt supply.

Room for more funding

GAIN’s USI work is mostly supported by a large, multi-year grant from the Gates Foundation that is ending in 2015. It is not yet clear if the Gates Foundation will renew this support. It seems likely that if it does not, GAIN will have a funding gap for its USI work. We have not yet discussed with GAIN how it would prioritize USI work at different levels of funding.

GAIN’s response: GAIN is concerned by the apparent lack of continued funding for iodine nutrition across the sector. GAIN is not alone in the sector and others are also struggling to fund what we believe to be critical activities towards improving global iodine intakes. This includes technical assistance, monitoring and evaluation and policy and advocacy towards continued improvement of national iodization programs and other proven iodine interventions. Global progress towards the virtual elimination of iodine deficiency disorders has been enormous to date but there is much work to be done. GAIN hopes to see commitments made by funders for this important work to ensure sustainability and to complete these efforts.

Update on Open Philanthropy Project

Our last major public updates on the Open Philanthropy Project were our May and June posts on global catastrophic risks and U.S. policy. This post summarizes our progress since then and where we currently stand on our goal of committing to causes.

Summary of currently ongoing activities

  • Deep dives on priority causes for U.S. policy. In June, we hired Shayna Strom as our Director of U.S. Policy. Shayna comes to GiveWell from the White House, where she was the Chief of Staff and Senior Counselor at the White House Office of Information and Regulatory Affairs. She is based on Washington, D.C., and she is investigating our two leading contenders for priority causes in U.S. policy - labor mobility and criminal justice reform – at a higher level of depth than any cause investigation we’ve done so far, aiming to get a full lay of the land and develop a preliminary strategy for where we expect to concentrate our grantmaking. She is also aiming to surface more potentially promising causes.
  • Shallow- and medium-depth investigations for U.S. policy. Alexander Berger, who previously was conducting shallow and medium investigations within both U.S. policy and global catastrophic risks, is now focused on U.S. policy. He has conducted shallow investigations on several causes (not yet written up) and is currently exploring the spaces of health care policy and U.S. poverty & inequality.
  • Global catastrophic risks. Howie Lempel, who previously was focused on investigating and writing about the Pew Charitable Trusts, has been transitioning to a role in which his main responsibility is leading our work on global catastrophic risks. Howie’s main responsibility at the moment is conducting relatively high-depth investigations, including some preliminary grantmaking if warranted, of biosecurity and geoengineering. He also is conducting or managing lower-depth investigations on several other causes, including risks associated with artificial intelligence. In addition, Nick Beckstead, currently at Future of Humanity Institute, is working on a contract basis to conduct shallow investigations of further possible global catastrophic risk areas. Nick will be joining us as a full-time employee in December.
  • Critical evidence reviews. We are now working with David Roodman as a contractor focused on critical evidence reviews relevant to the Open Philanthropy Project. He recently completed a draft writeup on immigration and current residents’ wages and is now working on a writeup attempting to quantify the risks of geomagnetic storms.
  • Preliminary work on scientific research funding. We have assembled a small team of junior scientific advisors that works with me on preliminary explorations of scientific research funding. Our last update on this front was in 2013, and more updates are forthcoming.
  • Other work. In light of our enlarged team, we have spent substantial time improving our procedures for efficiently coordinating on investigations (and particularly grant decisions). We recently finalized another grant that has come out of our (now de-prioritized) co-funding work. We have continued work on our history of philanthropy project and have two case studies pending publication.

Our current plans
Our current focus is on committing to causes within the broad categories of U.S. policy and global catastrophic risks. We’ve published a spreadsheet summary of our current stances on the causes we’ve investigated, and which ones we consider to be the strongest contenders based on what we know today. In brief:

  • Within U.S. policy, our top contenders for priority causes are labor mobility and criminal justice reform. We have also done substantial work (including some preliminary grants) on macroeconomic policy.
  • Within global catastrophic risks, our top contenders for priority causes are biosecurity and geoengineering, though there are several causes that we see as potential contenders and are still working on investigating, including risks associated with artificial intelligence.

Before committing to causes, we are hoping to:

  • Investigate more potential causes at shallow and/or medium depth.
  • Do relatively deep investigations of the top contenders, and gain a more tangible sense of what our strategy and grantmaking would likely look like if we committed to these causes.
  • Hold convenings in which a small number of people with relevant experience provides feedback on our thinking. We are planning a convening for November in Washington, D.C. on the subject of our priorities within U.S. policy. We are still deciding whether and how to conduct a convening around global catastrophic risks (currently leaning against doing so).
  • Put more thought into how many, and what level, of “commitments to causes” we should make given our current capacity.

We are still hoping to commit to causes in these categories by the end of the calendar year, though we still see this as a stretch goal and see a reasonable probability that we’ll delay by a couple of months. As a lower priority, I am working on investigating possible approaches to scientific research funding, and hope to provide more updates on this front fairly soon.

Grants
Good Ventures has made a number of grants within potential priority causes, largely as a way of exploring what giving opportunities in these causes might look like (more on this idea at a previous post). These grants have been in top contender areas for U.S. policy (labor mobility, criminal justice reform and macroeconomic policy); there are likely to be some grants within some global catastrophic risk areas as well in the near future. A list of grants to date is available here.

Update on SCI’s evidence of impact

Note: Consistent with our usual practices, SCI reviewed a draft of this post prior to publication, but the final product is ours.

We wrote last year about reevaluating studies we relied on in our evaluation of the Schistosomiasis Control Initiative (SCI), which is one of our top charities. We noted at the time that we were planning to continue learning about the studies. We have now revisited these studies and discussed them in depth with SCI. We have two main takeaways from this investigation:

  1. We now feel that some of the panel studies we previously relied on in our evaluation of SCI do not in fact provide evidence that SCI’s national programs have reached a large proportion of children targeted. We wrote last year that in one of the four studies, participants received extra deworming treatment from researchers if they were found to be infected with worms. We learned this year that participants in at least one of the other three studies received treatment separately from and under closer supervision than other students in the country. Where participants received more treatment or more careful treatment than other students in the country, we believe that the results of the studies do not reflect the treatment coverage achieved by the national programs.
  2. We are concerned about SCI’s external communications around these panel studies. We first published the view that all of SCI’s panel studies provided evidence of effective national programs in 2009, along with our interpretation of how the studies were carried out. It is concerning to us that (a) the published papers on the studies, which had SCI staff as coauthors, imply or explicitly say that the studies reflect the performance of the national programs, (b) SCI did not correct this interpretation, which we published and asked them to review, and (c) when we asked SCI about the methodological issue specifically, SCI leadership gave us information that SCI program staff later contradicted. 

SCI has told us that it has standardized and made significant improvements to its procedures for more recent monitoring, including treating children at sentinel schools as part of MDAs. SCI recently shared with us a more recent panel study and two studies of MDA coverage rates. We plan to write about these studies in our upcoming review of SCI.

On the whole, we continue to view SCI as an outstanding giving opportunity, and it will likely maintain its status as a top charity when we refresh our recommendations in December.

Our updated views on four SCI panel studies

In our 2011 and 2012 recommendations of SCI, we emphasized a set of four panel studies from Uganda, Burkina Faso, Niger, and Burundi. In each of the studies, a set of sentinel schools was chosen, and children at each sentinel school were tested for worm infections before the start of an SCI-led control program. The same children were followed up and tested again in subsequent years. The studies show a fairly consistent picture of dramatic drops in disease prevalence among the study participants. We acknowledged multiple limitations to these studies, but felt that they gave relatively strong suggestive evidence that the SCI-led control programs had reached a large proportion of the children they targeted.

Last year, we learned that in the Uganda study (Kabatereine et al. 2007), study participants who tested positive for infection in the course of the study were directly treated by the researchers. Thus, improvements measured over time could be attributable to the activities of the researchers carrying out the study rather than necessarily to the coverage of the national control program.

This year, we have revisited the methodology used in the other three studies through conversations with SCI. We have learned that the Burkina Faso study has similar methodological issues to the Uganda study. We are unsure whether researchers participated in treatment in the studies in the other two countries.

In the Burkina Faso study, which began in 2004, drug administration at the sentinel sites was not part of the national mass drug administration (MDA) program. Researchers working for the national schistosomiasis control program supervised teachers as they administered deworming treatment to all children in the sentinel schools on the day researchers took samples, a few weeks before the MDA. SCI told us that the national control program staff ran the study in this way to meet ethical guidelines laid out by the countries’ ethics boards as well as to produce more impressive results than might have been obtained by treating children as part of the national MDA. In addition to their ethical commitment and strong desire to ensure that all children tested were treated, the researchers were more knowledgeable about deworming than the district-level health workers who administered treatment in the MDA. Supervision from the researchers may have caused teachers to do a more thorough job administering deworming treatment. For this reason, and because drug administration in sentinel schools was logistically separate from the MDA, we do not consider this panel study to reflect the coverage achieved by the national MDA.

We are unsure whether participants in the Niger study received treatment in the same manner as the other students throughout the country, or whether they were treated separately under the supervision of the researchers as in Burkina Faso. SCI leadership originally told us that participants were treated in the same way as other students in the country. SCI’s current Program Manager for Niger and Burkina Faso (who was not the Program Manager at the time of the study) later said that participants were treated separately under the supervision of the researchers (note 1). After we discussed the Program Manager’s comments with SCI leadership, SCI contacted Dr. Amadou Garba, National Coordinator of the Schistosomiasis and Soil-transmitted Helminth Control Program in Niger at the beginning of the panel study. According to SCI, he stated that participants were treated as part of the MDA. We believe that participants were most likely treated as part of the MDA, but feel that we cannot be sure given the conflicting positions expressed by SCI staff.

Note that we believe that each SCI staff member we spoke to told us what they believed to be the truth about the methodology of the studies, and we do not think that any of them intentionally misled us.

SCI’s Program Manager in Burundi confirmed SCI leadership’s statement that the children in the sentinel schools there received treatment in the same manner as children throughout the country (note 2).

Note that other substantial concerns about these studies remain even in cases where children in sentinel schools were treated as part of the national MDAs. As we wrote in our 2012 review of SCI, we are concerned because the sentinel sites selected may not be representative of the whole country, and because only about half of the students initially surveyed were followed up. We are also concerned that treatment teams and teachers may have known which schools were sentinel schools and may have administered deworming treatment more carefully in the sentinel schools (note 3). We have had these concerns since first evaluating the studies and they should not be taken as an update, but it is important to note that they are still live concerns.

Additionally, given that it took us years to discover the issues regarding different protocols for treating children at sentinel sites, we feel that it is reasonably likely that the Niger and Burundi studies have weaknesses that we have not yet discovered.

We have seen little monitoring data outside of the panel studies that we believe shows that national programs supported by SCI reach a high proportion of the children they attempt to treat. Thus, we are unsure of the impact of SCI’s programs. Also, considering that the Burkina Faso study does not seem to reflect the coverage achieved by the MDA, we now have somewhat lower confidence that SCI has been able to effectively use its research results to improve its programs.

SCI’s external communications about the four studies

Our conversations with SCI staff about the panel studies have reduced our confidence in SCI’s external communications, particularly its communication with us. There are three reasons for this:

  1. The published papers on the studies, which had SCI staff as coauthors, either imply or explicitly say that the Uganda and Burkina Faso studies reflect the performance of the national programs, even though we now believe that the studies do not necessarily reflect the performance of the national programs (note 4).
  2. We treated the panel studies as central evidence of the impact of SCI’s MDAs in our 2011 and 2012 reviews of SCI. SCI vetted those reviews but did not note to us that in some studies, children in the sentinel schools were treated separately from the national MDAs.
  3. This year, we asked SCI’s leadership about whether the Niger and Burkina Faso studies had methodological issues similar to the Uganda study. They told us, “All students were treated as part of the national treatment program (at the same time and with the same treatment strategy) as the purpose of the sentinel sites was to assess the impact of the national control program.” When we later spoke with program staff who had been involved in the studies, they contradicted this picture (though we believe that all SCI staff told us what they believed to be the truth).

We previously noted difficulties communicating with SCI. We feel that our struggle to communicate effectively with SCI about the panel studies was more serious than previous difficulties. We credit SCI with connecting us with staff who could provide more detailed answers to our questions, even where those answers contradicted SCI leadership. Still, the fact that it took this much time and effort to gain this information reflects poorly on our ability to communicate with SCI.

More recent studies

The Uganda and Burkina Faso studies began in 2003 and 2004 respectively. SCI has continued to collect monitoring data as part of its ongoing programs. SCI has told us that it has standardized and made significant improvements to its procedures for more recent monitoring, including treating children at sentinel schools as part of MDAs. SCI recently shared with us a more recent panel study and two studies of MDA coverage rates. We have begun to analyze these new studies and feel that they likely provide some degree of additional evidence that national programs supported by SCI achieve relatively high coverage. We plan to write about these studies in our upcoming review of SCI.

Bottom line

Our concerns about SCI’s evidence of effectiveness and its external communications discussed in this post cause us to take a less positive view of SCI. However, we continue to believe that the program SCI supports, combination deworming, is among the most cost-effective programs we have considered and that the program has room for more funding globally. We recently wrote about a new study that seems to bolster the evidence for the long-term effects of deworming. In addition, SCI recently sent us the more recent studies discussed above and hosted us for three days of meetings with staff to update and expand our understanding of SCI’s work. When we refresh our top charity rankings later this year, we will likely include SCI.


Note 1: “In 2004 both Niger and Burkina Faso received funds from the Bill & Melinda Gates Foundation through the SCI to establish a national Schistosomiasis and STH control program. In addition to the financing of mass drug administration (MDA), a research lab was built in each country with the purpose of monitoring and evaluating the impact of the national deworming program. Assessment of the MDA was carried out through a six-year sentinel site survey from 2004-2010. Statistical power calculations were performed to ensure that the correct number of schools and children per school were sampled to be representative of all schools receiving the same treatment strategy. At baseline (in 2004), each of the sentinel schools were sampled by a team from the research lab who arrived in the morning to take samples from 150 students selected at random, which were then taken back to the lab for analysis. In the afternoon, one or two lab staff remained at the school to supervise the deworming drug administration and to ensure that all students in the school were treated. Ethical review boards in the countries mandated that the members of the lab team personally ensure treatment of children who had been sampled for the study. The lab staff who administered treatment were aware that the school was a sentinel site. These same children were then followed up each sub sequential year. If there were any children that were lost-to-follow up then new children who were entering into the first year of school at sentinel schools were recruited into the study. They were then representative of non-treated children. Children in the sentinel schools received treatment under the supervision of the lab staff each year. The lab staff were unaware of which students had been found to have worms to avoid preferential treatment. Children who were found to have worms were not given any extra treatment beyond the treatment given to the entire sentinel school. The sentinel sites were treated two weeks prior to the national MDA to ensure that the schools were not treated twice. SCI were able to then monitor registers from the national campaign showing which schools had received treatment in the MDA to ensure that the sentinel schools were not twice treated. For the non-sentinel schools, the National Schistosomiasis and STH Control Program were responsible for coordinating the mass treatment for all schools in each district. Cascade training was carried out where teachers were trained at the district level on how to administer the deworming drugs. The teachers in turn were responsible for treating the children with supervision from district level health staff. The district level health staff were in turn trained and supervised by the central level Ministry of Health.” Non-verbatim summary of a conversation with Anna Phillips on May 27, 2014.

Note 2: “The Schistosomiasis Control Initiative (SCI) ran a study on Burundi’s national control program from 2007-2011. During the study, researchers tested students at sentinel schools for schistosomiasis in mid-May every year. Researchers did not provide treatment to any students. Students in sentinel schools were supposed to receive schistosomiasis treatment at their schools as part of the Burundian government’s annual mass drug administration (MDA) in mid-June. The ethics review board in Burundi approved the option of treating children from sentinel schools as part of the MDA. The MDA was part of Mother-and-Child Health Week, a national program in Burundi that delivered vaccines and other medical interventions. The treatment delivery system was the same throughout the country, including in regions containing sentinel schools.” Non-verbatim summary of a conversation with Giuseppina Ortu on June 20, 2014.

Note 3: “It is unclear whether the treatment team knew which schools were sentinel schools. Researchers visiting the sentinel schools would have been highly visible, so teachers, students, and people living nearby likely knew if a school was a sentinel school. Conceivably, if the treatment team knew which schools were sentinel schools, it may have been particularly careful to provide treatment to the students in the sentinel schools. The best way to avoid this would have been for the researchers to sample different schools every year so that the treatment team could not predict which schools would be sampled next. However, switching schools every year would have prevented the researchers from following the same students from one year to the next. The treatment team did not know the medical test results of the individuals whom they were treating in sentinel schools, but the team might have been told that there were some students with positive test results in particular schools. The team leading the sentinel school study and the teams administering treatment were part of the Burundian government’s neglected tropical disease (NTD) control program. However, there was little overlap between the team leading the study and treatment teams, because the people leading the study worked for the central government, while the treatment teams consisted of workers from district health centers. On the other hand, it is possible that one of the leaders of the study also supervised the MDA.” Non-verbatim summary of a conversation with Giuseppina Ortu on June 20, 2014.

Note 4: For example:

  • 2007 paper on the Burkina Faso study with SCI staff Artemis Koukounari, Elisa Bosqué-Oliva, Yaobi Zhang, Christl Donnelly, Alan Fenwick, and Joanne Webster as coauthors was titled “Schistosoma haematobium Infection and Morbidity Before and After Large-Scale Administration of Praziquantel in Burkina Faso.”
  • 2007 paper on the Uganda study with SCI staff Artemis Koukounari, Fiona Fleming, Yaobi Zhang, Joanne Webster, and Alan Fenwick as coauthors states in the abstract, “We aimed to assess the health impact of a national control programme targeting schistosomiasis and intestinal nematodes in Uganda, which has provided population-based anthelmintic chemotherapy since 2003.” We previously wrote about other aspects of this paper that we found to present a confusing picture.