The GiveWell Blog

Deworming might have huge impact, but might have close to zero impact

We try to communicate that there are risks involved with all of our top charity recommendations, and that none of our recommendations are a “sure thing.”

Our recommendation of deworming programs (the Schistosomiasis Control Initiative and the Deworm the World Initiative), though, carries particularly significant risk (in the sense of possibly not doing much/any good, rather than in the sense of potentially doing harm). In our 2015 top charities announcement, we wrote:

Most GiveWell staff members would agree that deworming programs are more likely than not to have very little or no impact, but there is some possibility that they have a very large impact. (Our cost-effectiveness model implies that most staff members believe there is at most a 1-2% chance that deworming programs conducted today have similar impacts to those directly implied by the randomized controlled trials on which we rely most heavily, which differed from modern-day deworming programs in a number of important ways.)

The goal of this post is to explain this view and why we still recommend deworming.

Some basics for this post

What is deworming?

Deworming is a program that involves treating people at risk of intestinal parasitic worm infections with parasite-killing drugs. Mass treatment is very inexpensive (in the range of $0.50-$1 per person treated), and because treatment is cheaper than diagnosis and side effects of the drugs are believed to be minor, typically all children in an area where worms are common are treated without being individually tested for infections.

Does it work?

There is strong evidence that administration of the drugs reduces worm loads, but many of the infections appear to be asymptomatic and evidence for short-term health impacts is thin (though a recent meta-analysis that we have not yet fully reviewed reports that deworming led to short-term weight gains). The main evidence we rely on to make the case for deworming comes from a handful of longer term trials that found positive impacts on income or test scores later in life.

For more background on deworming programs see our full report on combination deworming.

Why do we believe it’s more likely than not that deworming programs have little or no impact?

The “1-2% chance” doesn’t mean that we think that there’s a 98-99% chance that deworming programs have no effect at all, but that we think it’s appropriate to use a 1-2% multiplier compared to the impact found in the original trials – this could be thought of as assigning some chance that deworming programs have no impact, and some chance that the impact exists but will be smaller than was measured in those trials. For instance, as we describe below, worm infection rates are much lower in present contexts than they were in the trials.

Where does this view come from?

Our overall recommendation of deworming relies heavily on a randomized controlled trial (RCT) (the type of study we consider to be the “gold standard” in terms of causal attribution) first written about in Miguel and Kremer 2004 and followed by 10-year follow up data reported in Baird et al. 2011, which found very large long-term effects on recipients’ income. We reviewed this study very carefully (see here and here) and we felt that its analysis largely held up to scrutiny.

There’s also some other evidence, including a study that found higher test scores in Ugandan parishes that were dewormed in an earlier RCT, and a high-quality study that is not an RCT but found especially large increases in income in areas in the American South that received deworming campaigns in the early 20th century. However, we consider Baird et al. 2011 to be the most significant result because of its size and the fact that the follow-up found increases in individual income.

While our recommendation relies on the long-term effects, the evidence for short-term effects of deworming on health is thin, so we have little evidence of a mechanism through which deworming programs might bring about long-term impact (though a recent meta-analysis that we have not yet fully reviewed reports that deworming led to short-term weight gains). This raises concerns about whether the long-term impact exists at all, and may suggest that the program is more likely than not to have no significant impact.

Even if there is some long-term impact, we downgrade our expectation of how much impact to expect, due to factors that differ between real-world implementations and the Miguel and Kremer trial. In particular, worm loads were particularly high during the Miguel and Kremer trial in Western Kenya in 1998, in part due to flooding from El Niño, and in part because baseline infection rates are lower in places where SCI and Deworm the World work than in the relevant studies.

Our cost-effectiveness model estimates that the baseline worm infections in the trial we mainly rely on were roughly 4 to 5 times as high as in places where SCI and Deworm the World operate today, and that El Niño further inflated those worm loads during the trial. (These estimates combine data on the prevalence of infections and intensity of infections, and so are especially rough because there is limited data on whether prevalence or intensity of worms is a bigger driver of impact). Further, we don’t know of any evidence that would allow us to disconfirm the possibility that the relationship between worm infection rates and the effectiveness of deworming is nonlinear, and thus that many children in the Miguel and Kremer trial were above a clinically relevant “threshold” of infection that few children treated by our recommended charities are above.

We also downgrade our estimate of the expected value of the impact based on: concerns that the limited number of replications and lack of obvious causal mechanism might mean there is no impact at all, expectation that deworming throughout childhood could have diminishing returns compared to the ~2.4 marginal years of deworming provided in the Miguel and Kremer trial, and the fact that the trial only found a significant income effect on those participants who ended up working in a wage-earning job. See our cost-effectiveness model for more information.

Why do we recommend deworming despite the reasonably high probability that there’s no impact?

Because mass deworming is so cheap, there is a good case for donating to support deworming even when in substantial doubt about the evidence. We estimate the expected value of deworming programs to be as cost-effective as any program we’ve found, even after the substantial adjustments discussed above: our best guess considering those discounts is that it’s still roughly 5-10 times as cost-effective as cash transfers, in expectation. But that expected value arises from combining the possibility of potentially enormous cost-effectiveness with the alternative possibility of little or none.

GiveWell isn’t seeking certainty – we’re seeking outstanding opportunities backed by relatively strong evidence, and deworming meets that standard. For donors interested in trying to do as much good as possible with their donations, we think that deworming is a worthwhile bet.

What could change this recommendation – will more evidence be collected?

To our knowledge, there are currently no large, randomized controlled trials being conducted that are likely to be suitable for long-term follow up to measure impacts on income when the recipients are adults, so we don’t expect to see a high-quality replication of the Miguel and Kremer study in the foreseeable future.

That said, there are some possible sources of additional information:

  • The follow-up data that found increased incomes among recipients in the original Miguel and Kremer study was collected roughly 10 years after the trial was conducted. Our understanding is that 15 year follow-up data has been collected and we expect to receive an initial analysis of it from the researchers this summer.
  • A recent study from Uganda didn’t involve data collection for the purpose of evaluating a randomized controlled trial; rather, the paper identified an old, short-term trial of deworming and an unrelated data set of parish-level test scores collected by a different organization in the same area. Because some of the parishes overlap, it’s possible to compare the test scores from those that were dewormed to those that weren’t. It’s possible that more overlapping data sets will be discovered and so we may see more similar studies in the future.
  • We’ve considered whether to recommend funding for an additional study to replicate Baird et al. 2011: run a new deworming trial that could be followed for a decade to track long term income effects. However, it would take 10+ years to get relevant results, and by that time deworming may be fully funded by the largest global health funders. It would also need to include a very large number of participants to be adequately powered to find plausible effects (since the original trial in Baird et al. 2011 benefited from particularly high infection rates, which likely made it easier to detect an effect), so it would likely be extremely expensive.

For the time being, based on our best guess about the expected cost-effectiveness of the program when all the factors are considered, we continue to recommend deworming programs.

Comments

  • Alex on July 27, 2016 at 8:24 am said:

    So your recommendation for the deworming charity really rests almost entirely on a single piece of research (“the worms at work” paper) – that isn’t a very broad evidence base, is it?

    What about funding a desk-based reanalysis of the primary dataset for this “worms at work” paper, ideally by some scientists working in a separate institution and/or discipline with no vested interests in finding evidence either for or against the original conclusions? Ideally with a pre-analysis plan written+published in advance of sight of the data. That would be a cheap+quick way to see if the findings of “worms at work” hold up to independent scrutiny.

  • Alexander on July 29, 2016 at 11:11 am said:

    Hi Alex – we did an internal re-analysis of the 2012 draft of the Worms at Work paper here, but because the paper was not yet accepted for publication, the authors were (understandably if unfortunately, in my view) unwilling to publicly release the data or code. However, our internal re-analysis missed some problems that the 2015 outside re-analysis of Miguel and Kremer found (see here for context), so I definitely agree that it’s not dispositive. The Worms at Work paper was recently accepted for publication at QJE, and QJE has an open data and code policy, so an outside code review should be feasible in the near future.

  • Great – it sounds like you have a plan.

    You could ask Macartan Humphreys at Columbia – last year, he conducted and wrote up a well-respected reanalysis of the Miguel+Kremer 2004 paper in about 3 weeks flat and posted the findings on his blog (http://www.columbia.edu/~mh2245/w/worms.html). Or you could offer it up through public expressions of interest, in the way that 3ie does, though that process is somewhat slower.

  • Colin Rust on August 16, 2016 at 8:28 pm said:

    I’m pretty surprised that the evidence for short term benefits of deworming programs is so weak. OK, so there is strong evidence that deworming reduces worm loads (go figure!), but even on weight gain (which I would expect to be a no-brainer given we’re talking about parasite in your digestive tract competing with you for your food), there are apparently dueling meta-analyses on whether there is a demonstrated effect.

    Any sense of why there is so little evidence of short term benefits?

    Three kinds of possible explanations come to mind. One possibility is “good worms”: maybe for some subset of people – say with modest worm loads or without comorbidity like malnutrition – the worms are actually a net positive. Another possibility might be subtle side effects of the deworming pills, especially as many of the people treated don’t have worms at all (since the pills are much cheaper than testing whether people are infected, they don’t test the whole population). In other words, everyone treated has the downside of the pill (if any) but only infected people get the upside. A third possibility is that the studies just don’t have enough statistical power: so maybe they don’t show a significant effect, but on the other hand they don’t rule out a large effect size either.

    If it’s thought to be primarily the third issue (the studies haven’t constrained effect size much) then it might still be reasonable to expect short term benefits based on priors.

  • When we are down to a multiplier of 1-2%, the role of the original research in calibrating any estimate of magnitude seems questionable. If it is worth haircutting so immensely, then why even use it as a starting point?

  • Alexander on August 23, 2016 at 4:36 pm said:

    Hi Colin – Good question. I don’t have the same prior that we should obviously expect weight gain effects, for what it’s worth. I think that’s because I don’t find the fact that the worms live in the digestive track to have a straightforward implication for weight gain – if the worms were long-lived (i.e. never exited the body) and more efficient metabolizers than their hosts, I think we’d expect the (human+worm) mass to be lower after deworming. Of course, that’s just a toy model and I don’t remember off the top of my head how long the various worm species tend to live in their human hosts, but I don’t think it’s clear that we should have a prior of meaningful weight effects here.

    But with respect to your question about the explanation, my instinct is that it is mostly #3, which seems like the easiest way to reconcile the fact that studies that only measure people who are known to be infected with worms do find an impact on weight, while those that measure population deworming (according to Cochrane) haven’t found overall effects on weight. This is pretty similar to what I wrote in reply to a similar question on the blog in 2012 for what it’s worth. We still haven’t had a chance to fully engage with the more recent reply to the Cochrane review, but are hoping to do so soon.

  • JC – it’s a good question, and we don’t think it’s unreasonable to think that it’s not worth giving to deworming because there is so much skepticism.

    That said, while we think it’s possible that there isn’t a significant, long-term effect, the collection of data points we have on long-term effects makes us think that it’s plausible enough that an effect exists to be worth considering. The other adjustments — largely adjusting estimated effect size based on differing conditions between the key study and modern implementations — are in some sense separate from the question of whether an effect exists at all. Multiplying everything together to get such a large haircut does make the whole exercise somewhat questionable, but we think that because it still leaves you with such a large effect, and because we think it’s plausible enough that some effect exists, we still think it’s worth doing.

    We think expected value is roughly the right way to handle these sorts of adjustments, because we don’t want to discard the possibility of truly extraordinary outcomes just because our best guess is that the probability is low. Concretely, if someone offered you a $1000 if you drew the ace of spades from a deck of cards (knowable ~2% probability), and said you only had to pay $1 to play, that’s a great deal, even though ~98% of the time you wasted your dollar. Of course, our adjustments to get to the 1-2% range are much less certain than the probability of drawing a given card from an honest deck, and it’s very reasonable to doubt them on that basis, but we think it would be a mistake to discard evidence or interventions just because the implied probability discounts were getting low.

  • Howard White on August 25, 2016 at 2:57 pm said:

    There is a new Campbell review being published next month which addresses many of the perceived shortcomings of the Cochrane review. We hope it wiill be presented at the What Works Global Summit wwgs2016.org. The same team are planning a review with re-analysis of all data at case level (IPD analysis):

  • Howard – thanks for the heads up, we’ll keep an eye out for that review.

  • Just a head’s up that I could not find SCI on Amazon Smile’s list of charities or search engine.

Comments are closed.