New Cochrane Review of the Effectiveness of Deworming

Update 07/20/12: Miguel and Kremer (and others) have responded to the characterization of their 2004 study by the updated Cochrane review here. We find many of their responses to the Cochrane authors’ objections (which are distinct from our reservations) persuasive, especially regarding attrition and sample selection in the haemoglobin data and baseline school attendance data. As we wrote last week, the Baird et al. 2011 follow-up to Miguel and Kremer 2004 remains especially important to our view on deworming; neither the updated Cochrane review nor the author’s response has changed that.

On Wednesday, the Cochrane Collaboration published a new systematic review of the effectiveness of deworming drugs in improving nutritional status, school performance, and cognitive test scores.

The new Cochrane review of deworming to kill soil-transmitted intestinal worms (STHs) finds almost no evidence of benefits on nutrition, cognitive development, or school performance in mass deworming studies, and small benefits on nutrition in small, screened studies; this is largely the same conclusion as the older Cochrane review, though the new one is updated with more studies and a persuasive response to criticisms. It excludes studies that treat both STHs and schistosomiasis, which is what the Schistosomiasis Control Initiative does, so it does not directly affect our assessment of them. However, the new review reinforces our skepticism about the quality of much of the evidence supporting deworming, and strengthens our view that the evidence in favor of distributing bednets is stronger. Accordingly, SCI continues to hold our #2 rating. We plan to continue to investigate the papers that are most crucial to our assessment of the benefits of deworming.

In a nutshell, the new Cochrane review does not directly challenge the case for SCI as our #2 charity, though we have somewhat less confidence than we did.

In the remainder of this post, we:

The new Cochrane review on STH deworming

In the new Cochrane review on STH deworming, Taylor-Robinson et al. examine randomized controlled trials (RCTs) of deworming to address soil-transmitted intestinal worms (STHs), looking at impacts on nutrition, cognitive skills, and educational outcomes. Excluding studies that treated both STHs and schistosomiasis, they find surprisingly limited evidence of nutritional benefits, and very little support for cognitive or educational benefits.

In particular, they find that:

  • in mass deworming programs that treated everyone without testing them first, there is no consistent evidence for any effect on nutrition, cognitive performance, or school performance (more);
  • in small pilot programs that screened for the presence of worms prior to treatment, treatment was associated with increased weight and haemoglobin, which implies a reduction in anemia (more).

The previous Cochrane review of STH deworming, also by Taylor-Robinson et al., reached many similar conclusions, but we believe the new one to be more robust (more). The older review did not separate out studies that screened for worm infection and look at their effects separately, as the new review does. Doing so sharpens our take on the evidence, without fundamentally changing the picture.

We write more below about how this affects our take on SCI, but it is worth noting that the new systematic review might affect our likelihood of recommending Deworm the World, another deworming charity that we have been investigating. Unlike SCI, which conducts combination deworming, we believe that Deworm the World does some STH-only deworming.

Changes since the last Cochrane review and response to critics

The new review differs from the previous Cochrane review of STH deworming in several ways. Most importantly, from our perspective:

  • it incorporates many additional studies, including more studies focused on haemoglobin/ anemia and Miguel and Kremer 2004, which was previously excluded;
  • it stratifies mass deworming studies by the prevalence of infections, so it can determine whether effects are consistently larger in higher-prevalence studies; and
  • it distinguishes between mass and screened deworming programs.

The new review also differs from several systematic reviews–Hall 2008, Albonico 2008, and Gulani 2007–that have been published since the last major update to the Cochrane review, all of which found statistically significant benefits to deworming.

Some of the changes since the last review were undertaken in order to respond to criticisms from deworming scholars. Taylor-Robinson et al. write:

Critics of a previous version of this review (Dickson 2000a) stated that the impact must be considered stratified by the intensity of the infection (Cooper 2000; Savioli 2000). We have done this comprehensively in this edition and no clear pattern of effect has emerged….

Other advocates of deworming, such as Bundy 2009, have argued that many of the underlying trials of deworming suffer from three critical methodological problems: treatment externalities in dynamic infection systems, inadequate measurement of cognitive outcomes and school attendance, and sample attrition. We agree with these points. However, externalities will be detected by large cluster-RCTs with a year or more follow up, and there are now five trials such as this included in this review.

We find these responses from Taylor-Robinson et al. compelling and we believe the new review to be a significant improvement over the older Cochrane review of deworming.

The new review’s take on mass-deworming programs

Unlike screened programs, mass deworming programs treat everyone with deworming drugs without testing whether they have a worm infection first (because doing so is costly relative to the price of the deworming drugs). The new Cochrane review finds that there is little evidence from studies of mass deworming programs to show that they improve nutrition, cognitive performance, or school outcomes.

Two studies in one location in Kenya with extremely high worm prevalence found that a single deworming treatment caused weight gain, but seven more studies in different areas found no effect, and larger studies with multiple doses were even more inconclusive: two found large and significant results, while ten others found small statistically insignificant results (pgs 19-21). There is essentially no evidence from studies of mass STH deworming to show that it improves haemoglobin status, height, cognitive test scores, or school performance; the evidence for an improvement in school attendance comes solely from the Miguel and Kremer 2004 study, with the other unscreened RCT finding no improvement in attendance. (See our update about this study above).

The older Cochrane review on STH deworming, which we wrote about in our intervention report on deworming, did not distinguish as sharply between mass and screened programs. Though a sensitivity analysis in the old review that focused on mass studies found no significant effect on weight, the main analysis found a small statistically significant benefit by combining screened and mass studies. The new review continues to find that mass deworming has no statistically significant benefit on weight, but it differs from the older review in that it foregrounds this result.

The new Cochrane review also includes haemoglobin status as a main outcome for the first time. It is the first systematic review we’ve seen that distinguishes between the haemoglobin outcomes of mass and screened deworming, finding no statistically significant effect of mass STH deworming.

The new review’s take on smaller programs that screened for worm infection

Despite finding little evidence from mass deworming studies to support deworming, the new Cochrane review does find some evidence from randomized controlled trials to indicate that STH deworming improves nutrition in programs that screen for worm infections (i.e. only give deworming drugs to infected people).

In three small RCTs with a total of 149 participants who were screened for STH infections prior to participation, deworming pills caused a statistically significant increase in weight of about .6 kilograms. In a few other small screened RCTs, deworming statistically significantly improved mid-upper arm circumference and skin fold thickness; similar studies found no effect on height, body mass index, or school attendance. Two screened RCTs with a total of 108 participants found that treating STH infections causes a statistically significant increase in haemoglobin of 3.7g/L (which implies a reduction in anemia).

What does it mean if smaller programs with screened participants show effects, while larger programs of mass deworming do not? One possibility is that STH deworming does have some impact on nutrition in infected individuals, but that the effect is too small to pick up in unscreened population studies. Another possibility is that the effects seen in smaller programs are spurious. The Cochrane review highlights the latter possibility, stating that “the data on targeted deworming is limited (three small trials, n = 149); the quality of the evidence is ’moderate’ for weight and ’low’ for haemoglobin.” (The Cochrane review also points to a third possibility: “the intervention itself is different … having been screened, and then told they have worms, children are more likely to comply with treatment, and alter their behaviour.” We find this possibility least likely.)

The overall quality of deworming research: publication bias, data-mining, and representativeness

One of our big take-aways from the Taylor-Robinson et al. review is that we should be really worried about publication bias, data-mining, and the representativeness of the research we rely on.

Publication bias

The best example of publication bias comes from the DEVTA study of deworming and Vitamin A supplementation, conducted on a population of more than a million children in Lucknow, India from 1999 to 2004, which remains unpublished to this day. We had already been aware of DEVTA from our research on Vitamin A supplementation, but the particulars of Taylor-Robinson et al.’s correspondence with the authors are new to us:

DEVTA: the world’s largest ever RCT, which includes over a million children randomized in a cluster design with mortality as the primary outcome, remains unpublished six years after completion. We have corresponded with the senior author on several occasions. We also wrote a letter to the Lancet in June 2011, asking for publication of this important study. When this letter was accepted, the authors submitted the manuscript to the Lancet within a week, and we withdrew our letter. However, at the time of writing (June 2012) the paper remains unpublished.

Results presented at a conference in 2007 (PPT) indicate that compliance was high but that the treatment did not cause a statistically significant reduction in mortality. Combining this results with other studies of Vitamin A, there still appears to be an effect on mortality, but the lack of formal publication means that the international consensus continues to overestimate the impact of Vitamin A on mortality.

We don’t think that STH deworming prevents a significant number of deaths, so whatever the impact of the deworming branch of the treatment in DEVTA on child mortality turns out to be is unlikely to affect our assessment of deworming. However, the fact that such a large and important study remains unpublished eight years after the trial was completed and five years after a conference presentation conveying the key results speaks to the power of publication bias.

Data mining

More generally, Taylor-Robinson et al. make it clear that studies have looked for potential impacts of deworming on a large number of different outcomes. (I count more than ten—weight, height, mid-upper arm circumference, skin-fold thickness, body mass index, measures of physical exertion like the Harvard Step Test, hemoglobin status, school attendance, school persistence, school exam performance, and cognitive test scores—with many potential sub-categories and measures each.) With so many different outcomes measured and little theoretical basis for determining which results are genuine, the potential for spurious results seems large, especially for outcomes which have been measured in only a few studies. (This would be a form of data-mining, and seems to have played a role in the previous systematic reviews that did find significant results.)

Representativeness

Taylor-Robinson et al. point to an additional concern about representativeness, which, while not really fitting the rubric of data-mining and publication bias, raises the specter of a set of rigorous research results that nonetheless don’t translate into practice. They write:

Evidence of benefit of deworming on nutrition appears to depend on three studies, all conducted more than 15 years ago, with two from the same area of Kenya where nearly all children were infected with worms and worm burdens were high. Later and much larger studies have failed to demonstrate the same effects. It may be that over time the intensity of infection has declined, and that the results from these few trials are simply not applicable to contemporary populations with lighter worm burdens.

This worry comports with our own reservations about the evidence from the Miguel and Kremer 2004 experiment, which was conducted during a period of abnormally elevated worm prevalences due to flooding caused by El Nino.

Together, these examples heighten our concern about the potential for bias and unrepresentativeness in the key studies we rely on in our assessment of the evidence for deworming.

The evidence in favor of the Schistosomiasis Control Initiative

Our intervention report on combination deworming, of the kind conducted by the Schistosomiasis Control Initiative, focuses on three kinds of benefits:

  • Subtle general health impacts, especially on haemoglobin. We drew our conclusions on haemoglobin effects from Smith and Brooker 2010‘s analysis of studies on combination deworming; since the new review examines STH-only deworming and not combination deworming, it does not address these studies.
  • Prevention of potentially severe effects, such as intestinal obstruction. These effects are rare and play a relatively small role in our position on deworming. The Cochrane review does not address these effects for the most part. (As stated above, it does discuss one study, with unavailable results, that examined mortality, but we believe mortality from STHs is rare enough that we wouldn’t expect it to show up in such a study.)
  • Developmental impacts, particularly on income later in life. The new review does not directly address the studies we used here. Bleakley 2004 is outside of the scope of the Cochrane review because it is not an experimental analysis, and Baird et al. 2011 is not mentioned, presumably because it has not yet been published. However, Taylor-Robinson do discuss Miguel and Kremer 2004, which underlies the Baird et al. 2011 follow-up; in their assessment of the risk of bias in included studies, Miguel and Kremer 2004 does poorly (it appears to be the worst-graded of the 42 included trials; Figure 3). (See our update about this study above.) Presumably, the follow-up is subject to most, if not all, of the same worries that characterize the initial study since it relies on the same underlying experiment. We have written before about our reservations about these studies, and the new Taylor-Robinson et al. review reinforces those reservations without adding substantial new information. We plan to continue to research the details of these papers, which are crucial to our assessment of deworming.

Conclusion

The new Cochrane review does not directly challenge the findings that are core to our view on combination deworming. That said, it does highlight general issues with research on deworming (e.g., potential publication bias and a case for benefit that is generally weaker than what many relevant academics and advocates seem to have believed). We therefore continue to recommend the Schistosomiasis Control Initiative as our #2 charity, though we have somewhat less confidence than we previously did.

Comments

New Cochrane Review of the Effectiveness of Deworming — 9 Comments

  1. > More generally, Taylor-Robinson et al. make it clear that studies have looked for potential impacts of deworming on a large number of different outcomes. (I count more than ten—weight, height, mid-upper arm circumference, skin-fold thickness, body mass index, measures of physical exertion like the Harvard Step Test, hemoglobin status, school attendance, school persistence, school exam performance, and cognitive test scores—with many potential sub-categories and measures each.) With so many different outcomes measured and little theoretical basis for determining which results are genuine, the potential for spurious results seems large, especially for outcomes which have been measured in only a few studies. (This would be a form of data-mining, and seems to have played a role in the previous systematic reviews that did find significant results.)

    Were the non-significant measures not reported or something? Otherwise I would have expected the Cochrane collaborators to have applied multiple correction and fixed that issue.

  2. Thank you for this detailed review. We all know that justice should be done and seen to be done – the Cochrane review is understandable but just feels wrong – the justice of the review is not seen to be done. Those billion kids with worms must be better off without them inside their stomachs and eating their food – and in extremely heavy infections causing more serious effects. And schistosomiasis is of course much more serious, Alan

  3. Re: “What does it mean if smaller programs with screened participants show effects, while larger programs of mass deworming do not?”

    Is it possible that giving deworming pills to healthy people has side effects that cancel out the benefits of giving deworming pills to sick people, which net out to zero in mass studies?

    Why do you think differential compliance is the least likely explanation?

    Following on gwern’s comment, what happens if you apply the correction on the assumption that all unreported measures were nonsignificant?

  4. Thanks for the comments.

    gwern: As far as I know, Cochrane does not adjust for multiple comparisons. Although statistical adjustments for multiple comparisons exist, I’m not sure that they’re helpful on the issue of “multiple potential definitions of outcome measures,” which aren’t necessarily reported in the underlying papers.

    Alan: We don’t doubt that deworming makes kids better off, but the question we face is whether it is one of the best possible ways to help people. The research showing limited evidence of benefit for mass STH deworming affects our view on the latter question.

    Aaron:

    • Although it is conceptually possible that giving deworming pills to healthy people causes some negative side effects that offset the benefits to treated populations, I haven’t seen any evidence of this. (They do sometimes appear to cause vomiting, but we’ve never seen a negative effect on, e.g., anemia.) My best guess is that the studies of STH to date just haven’t had the power to detect small improvements in nutritional status amongst the subset of infected people. I agree that this probably means that the effects of mass deworming are smaller than researchers expected (since they typically run experiments with adequate power to detect the anticipated benefits).
    • We think differential compliance is unlikely to explain the limited effects because deworming generally involves administrators watching the children take the pills. The pills are administered annually or semi-annually at school, so compliance is relatively easy to achieve.
    • We haven’t actually done the statistical correction (see my reply to gwern above), but it’s unlikely to affect the bottom line. The Cochrane review reports that, “Thirteen trials had evidence of selective reporting and were judged to be at high risk of bias (Goto 2009; Greenberg 1981; Kirwan 2010; Koroma 1996; Nga 2009; Nokes 1992; Olds 1999; Simeon 1995; Solon 2003; Stoltzfus 1997 (Cluster); Stoltzfus 2001; Sur 2005; Willett 1979). The remaining trials did not show evidence of selective reporting.” The studies that did show evidence of selective reporting are not the ones supporting the view that screened deworming improves nutrition.
  5. Thank you for this analysis- I think it’s very interesting that, from a narrowly technical perspective, mass deworming may not be having as much of an impact as has been widely flagged post Miguel et al. But, even in the context of this study, the actual intervention of providing deworming pills was presumably carried out without many gaps in implementation, but this is precisely the stage where most programs can fail while scaling up. This points to a bigger question that needs to be studied more carefully: how does such a program fit within country health systems, how much priority is/should this program be given and if it diverts scarce on the ground capacity from other activities, and, in the end, how will compliance be ensured once the big push from international organizations/charities is reduced?

  6. Alexander – This is just so difficult to answer. However WHO are preparing a response which will be published soon. And of course the veterinary deworming situation where true double blind controlled trials are possible prove incontrovertably that worm infections are detrimental in animals and treatment makes a massive improvement in weight gain. We are after all animals Alan

  7. The problem is not only that SCI may not be “cost-effective,” but that their intervention (mass deworming) may not be effective at all for the outcomes of interest. What outcomes are you interested in? Decreasing the death rate (mortality) of children (as in ?The Life You Can Save?)? Decreasing suffering (morbidity)? Increased education or school attendance? There is NO evidence in the studies cited by GiveWell that any of these outcomes are improved with mass deworming campaigns.
    The evidence cited by GiveWell for effectiveness comes from two Cochrane systematic reviews. First of all, both of them are reviews of studies of the effectiveness of medications used for treatment of infected individuals. None of the studies in the two reviews have evaluated mass treatment, which is quite different from testing and treating individual patients. Even if one were to use that data and assume the outcomes would be similar, there was no clinical improvement shown, i.e. no decrease in mortality or morbidity, no change in ?quality of life.? GiveWell says as much with their statement ??the evidence on their impact on quality of life is thin.? My only disagreement is that the evidence is not just ?thin,? but non-existent. Yet GiveWell then presents an estimate of a “cost per equivalent life saved” for deworming ranging from ~$1700-$3800? stating that they used ?numbers that have clearer and more intuitive meanings, including: ?Years of a given symptom (blindnesss, skin disease, etc.)? averted and ?Cases of a given health condition (blindness, skin disease, etc.) averted?.? But those ?symptoms? and ?cases? were NOT averted at all in the studies cited. So how can one calculate a “cost per equivalent life saved” when no ?equivalent lives? were saved?

  8. Angelo: The cost-effectiveness figures you’re citing are based on the Baird et al. 2011 followup to Miguel and Kremer 2004, which is a quasi-randomized study that finds that deworming received as a child increases income later in life. We’ve written about that study here and the cost-effectiveness figures (inclding posting the spreadsheets used to derive them) here.