The GiveWell Blog

Revisiting the case for developmental effects of deworming

(updated on: ) | by Alexander

This post discusses our detailed examination (including, with help from the authors, reanalyzing raw data) of the Miguel and Kremer 2004 study on deworming (treating people for parasite infections) as a way to raise school attendance, and a followup study (Baird et al. 2012) on the later-in-life impacts.

Our current #3 charity, SCI, focuses on deworming. Deworming is quite cheap (we estimate that SCI spends ~$0.50 per person treated, including all costs) but the benefits are not as obvious and tangible as those of many other health programs because the parasites treated cause few deaths and their effects may be subtle. (So much so that a recent Cochrane review fails to find statistically significant impacts of population deworming on the outcomes that have been most studied). In our view, the case that deworming is a “good buy” depends heavily on the idea of developmental effects: the possibility that deworming children has a subtle, lasting impact on their development, and thus on their ability to be productive and successful throughout life.

The main evidence for this idea comes from (a) Bleakley 2004, a study of the Rockefeller Sanitary Commission’s campaign to eradicate hookworm in the American South in the early 20th century; (b) a series of studies in Kenya, in which school deworming was rolled out on a purposefully arbitrary (randomization-like) basis, and children who received more years of deworming were compared to children who had received fewer. This post focuses on the latter.

We have long had questions about these studies, relating both to the possibility of publication bias and to possible ways in which the setting of the studies was unrepresentative. This year, we decided to ask the study authors for the data and code behind their studies, so that we could run additional analyses to gain more information about the seriousness of our concerns. The authors graciously shared their data and code and helped us to interpret it.

The full details of our reanalysis are available here. A big-picture summary of our concerns and findings follows.

Is data-mining a concern?
The concern

Data-mining is a form of publication bias, in which researchers look at many possible analyses of the data they collect, and present only the analyses most favorable to the conclusions they’re hoping to find. We were concerned about this issue for the series of deworming studies because

  • The first study (Miguel and Kremer 2004) drew a great deal of attention for its positive result (regarding the positive impact of deworming on school attendance), raising the possibility that researchers had the incentive to declare positive results from subsequent studies as well.
  • Different studies used different definitions of “treatment group,” and emphasized different outcomes (for example, the initial study emphasized school attendance; the second emphasized height; the third emphasized earnings).

What we found

After performing our own analysis, we are less concerned about this issue than we were previously. Changing our definition of the “treatment group” didn’t have much impact on the findings, and while the authors did share some outcomes with us that had not been reported in the paper, there were not a lot of these and they didn’t significantly change the picture.

That said, there do remain some reasons to be concerned about this issue. The authors of the most recent follow-up study (the one emphasizing earnings) shared not only data but also their funding proposal and survey questionnaire with us, and we note that

  • The survey was extensive, and included a lot of information that wasn’t included in the data that researchers analyzed. We weren’t surprised to see that this was the case (the follow-up study aimed to collect a rich data set, not just data customized for the questions it was asking), but it still raises the possibility that bias could have crept into the process of transforming raw survey answers into analyzable data.
  • The funding proposal expressed an interest in a wide variety of outcomes, including educational attainment, labor market outcomes (measurable in multiple ways, though the authors stated to us that one of the statistically significant positive effects described in the paper, labor market earnings, is the “canonical” measure for the field of labor economics), cognitive performance, happiness, health measures, and more; it did not clearly declare that any particular one was the primary outcome of interest.We attempted some rudimentary analysis of whether this fact could have facilitated spurious findings, and didn’t find strong reason to think that the findings were spurious. However, we struggled with the question of how a study’s findings should be adjusted for the fact that it had a larger universe of multiple outcomes that it could have chosen to emphasize; normal statistical adjustments for multiple comparisons do not perform well in cases with large numbers of ambiguous outcomes.

We feel that the issues above would have been much easier to resolve if the authors of the studies had preregistered their studies, declaring in advance what their primary and secondary outcomes and analyses of interest were. This is not to say that the authors have done anything unusual or wrong; our understanding is that preregistration was extremely rare (perhaps nonexistent) in economics at the time the study was done. In fact, Ted Miguel, one of the authors of the study, has gone on to co-author one of the first studies we’ve seen in this field that does utilize (and discuss) preregistration. Our point here is just that the practice of preregistration carries substantial credence benefits with us as consumers of research, and would affect our qualitative assessment of these findings.

How well would the studies generalize to other settings?
The area in which the deworming experiment was conducted had unusually high infection rates; in fact, infection rates rose substantially over the course of the study, as shown in Tables II and V of Miguel and Kremer 2004:

Measure Year 1 Prevalence Year 2 Prevalence
Moderate to heavy schistosomiasis infection 7% 18%
Moderate to heavy hookworm infection 15% 22%
Moderate to heavy roundworm infection 16% 24%
Moderate to heavy whipworm infection 10% 17%

The above table likely substantially understates the degree of change, because the second-year figure includes the benefits of treatment externalities experienced by the control group (discussed above). A calculation sent to us by the authors implied that the 18% prevalence of moderate to heavy schistosomiasis infection in the control group in year 2 shown above should be augmented by the 22 percentage point externality effect of treatment to get a genuine counterfactual infection rate of 40% – despite the fact that the initial prevalence was (as shown above) only 7%. This implies that without the program, the area would have seen an extreme rise in prevalence of moderate-to-heavy schistosomiasis infections. A footnote in Miguel and Kremer 2004 attributes this phenomenon to “the extraordinary flooding in 1998 associated with the El Niño weather system, which increased exposure to infected fresh water (note the especially large increases in moderate-to-heavy schistosomiasis infections), created moist conditions favorable for geohelminth larvae, and led to the overflow of latrines, incidentally also creating a major outbreak of fecal-borne cholera” (Pg 174).

Because of this unusual situation, we worry that the results of studies from this place and time may not generalize well to other circumstances in which rates are at lower, more typical levels.

What we found

We did a couple of analyses to see whether the headline effects were sensitive to the prevalence of moderate-to-heavy infections, particularly schistosomiasis infections. As we expected, we did see some reason to believe that deworming had had larger impacts in higher- than in lower-prevalence areas, and that it had had larger impacts for schools with substantial schistosomiasis prevalence. That said, what we found was far from sufficient to completely explain away the studies’ findings. In particular, dividing the schools into those that did and didn’t have substantial schistosomiasis prevalence left us working with fairly small sample sizes, from which it is difficult to conclude anything.

Overall, we are moderately less concerned about this issue than we were before.

Other alternative hypotheses
The findings of the studies discussed here – particularly the follow-up showing substantially improved earnings, resulting from just a couple of rounds of additional deworming treatment in childhood – have always struck us as quite counterintuitive and surprising. As such, we’ve been particularly attentive to alternative explanations of the findings.

One possible alternative explanation is that parents/students may have sought to switch into the schools that “won” early deworming treatments, which could cause the treatment and control groups to differ in ways not picked up by the baseline data measured by the studies. Further discussion with the authors made this concern appear far less likely to us (our understanding is that the deworming program was announced very close to the time when students were registered as “treatment” or “control,” and that school transfers after the start of the program were rare and relatively symmetric between schools).

Another possibility that has become more salient to us in the course of analyzing these studies is that efforts to encourage students to attend school in order to receive treatment might have bled over to later days, increasing attendance in treatment schools over the following years. The particular piece of data that led us to examine this possibility is that within schools, there is no statistically significant difference in attendance rates for treated and untreated students (the effects only appear across schools). (The authors assume that this phenomenon occurred due to the presence of within-school externalities.) In the course of analyzing the studies more closely, we learned that treatment dates were announced at the school in advance in an attempt to boost take-up, and that some efforts were undertaken to boost attendance on drug distribution days.

Finally, we wondered whether the results of these studies might be driven by a few “outlier” schools, but after the analysis we’ve done of the raw data, we are now convinced that this is not an issue.

Bottom line
By sharing their data, code, and other materials with us, the authors of these studies helped us to perform analyses that ultimately gave us more confidence in their results. Several of the aspects of these studies that most worried us (particularly regarding publication bias) turned out not to be important to the conclusions.

We still have substantial reservations about the studies. Preregistration would have been an additional measure that could have increased our confidence and lowered our concerns. In fact, this is the single case we’ve seen in which preregistration would have had the most influence on our conclusions. Had the authors preregistered hours worked or income amongst those employed (the key metrics showing improvement in Baird et al. 2012) as their main outcome of interest prior to collecting follow-up data, we would have far more confidence in the validity of the findings.

Going forward, field replications (carrying out similar deworming programs, and similar analysis to see whether similar results are obtained) would – in our view – greatly improve the robustness of the evidence.

In our view, the vast majority of aid interventions have almost no rigorous evidence behind them. A very small set of interventions – including LLIN distribution – have a broad, impressive evidence base. Deworming is somewhere in between. The studies discussed here are rigorous, have highly encouraging findings, held up to the best scrutiny we could bring to them. At the same time, many questions remain unanswered. This is one of the areas in which an additional long-term study would have the most effect on our views.

Conference call discussing our top charities, Thu. Dec 6th, 7pm Eastern

| by Natalie Crispin

We put a lot of effort into making our research process and reasoning transparent so that anyone can understand and vet the thinking behind our charity recommendations.

Consistent with this, we will be holding a conference call on Thursday, December 6th at 7pm Eastern to discuss our recently updated recommendations. The call is open to anyone who registers via our online form. Staff will take questions by email and answer them over the conference line. If you can’t make this date but would be interested in joining another call at a later date, you can indicate this on the registration form.

If you’re thinking of giving to one of our top charities this year, or you’re just curious about our thinking, we welcome you to join.

We’ve recently held similar discussions with smaller groups of GiveWell supporters. Audio and transcripts from these are available on the conference call page on our website.

Register for the December 6th GiveWell Conference Call

Our top charities for the 2012 giving season

(updated on: ) | by Holden

Over the past year, we’ve continued to follow and investigate our existing top charities, and we’ve also looked for more outstanding giving opportunities. Today we are announcing our updated top charities:

1. Against Malaria Foundation (AMF). AMF focuses on distribution of insecticide-treated nets to prevent malaria; of all the charitable interventions we know of that have clear room for more funding, this one has the strongest evidence of effectiveness and cost-effectiveness. AMF has outstanding self-evaluation and transparency. It first became our #1 charity in late 2011 and has continued to impress us.

2. GiveDirectly. GiveDirectly’s goal: for every $1 it spends, deliver $0.90 directly into the hands of extremely poor people in the developing world, with no strings attached. We first wrote about GiveDirectly in 2011 when it was just getting off the ground, and we now have enough information from its first year to believe that it is succeeding in this goal, and is an outstanding young organization with extremely strong self-evaluation and transparency.

Our definition of “evidence of impact” includes evidence that “wealth is being transferred to low-income people.” We feel that direct cash transfers face an unusually low burden of proof because the link is so tight between cash transfers and giving additional wealth (and choice/empowerment) to recipients. That said, cash transfers have been more heavily studied than any other non-health intervention we know of. The evidence says that they increase short-term consumption, especially of food, and there is suggestive evidence that they may be invested at very high financial rates of return (e.g. ~20% per year). We are not as confident in the investment returns as we are in the evidence for some other interventions, especially bednets, but we believe that the case for increasing consumption on its own terms is strong. The many high-quality studies on cash transfers also provide little support for common concerns about such transfers (such as the idea that they are spent mostly on alcohol).

Different donors will come to different conclusions on the value of this outcome (directly increasing recipients’ wealth) as compared to health interventions, and cash has some salient strengths (putting choice in the hands of recipients) and weaknesses (potential harms if it is poorly spent) relative to health interventions. For our part, we would guess that our other two top charities’ interventions (taken in isolation from the organizations delivering them) do more good on a per-dollar basis, but we are not confident in this, and we see a strong case for supporting GiveDirectly.

3. Schistosomiasis Control Initiative (SCI). SCI focuses on deworming: treating people for parasitic infections. Deworming is extremely inexpensive (~$0.50 per person treated, including all costs), and there is evidence linking it with substantial developmental benefits (people dewormed in childhood may attend school more and earn more later in life); the evidence is not as strong as for insecticide-treated nets but is still far above what we’ve seen for most charitable interventions. We have at times struggled to understand SCI’s activities and impact, and are not as confident in the organization as a whole as in AMF and GiveDirectly, but overall it is an organization with an impressive track record and a strong commitment to transparency, and we consider it an outstanding giving opportunity.

We have published updated reviews of all three charities, as well as audio and a transcript from a conference call we did last week, at which we discussed our preliminary thinking about how to rank these charities with a group of especially involved GiveWell followers. We also give some initial thoughts in this post on the relative strengths and weaknesses of the three charities.

We plan to publish substantial additional content in the coming weeks that will give more context for the thinking behind our rankings. This will include updated reports on insecticide-treated nets, deworming and cash transfers, and blog posts going into more detail on how we see the major considerations (in particular, how the different interventions compare on cost-effectiveness and how we respond to common objections/concerns regarding cash transfers).

This year, more than in past years, our top charities have very different strengths and weaknesses, and we see substantial room for individual donors’ judgment calls in how much to give to each. We encourage donors to read on and review our take on this question. The rest of this post will (a) discuss what we’ve done over the last year that has led us to our current rankings; (b) lay out what we see as the major considerations regarding which of the above three charities to give to.

Our work over the past year
Over the past year, we’ve done the following:

  • Continued to follow AMF and SCI, which were our top two charities as of December 2011. We published the following updates:

    Both made progress and continued to be highly transparent. We had an easier time understanding AMF’s activities (and the impact of GiveWell-sourced donations) than understanding SCI’s activities.

  • Continued to investigate key questions about insecticide-treated nets and deworming.
    • Regarding insecticide-treated nets, we did a major internal vetting of our work and published a post revisiting the cost-effectiveness of this intervention; we also wrote about the possibility of insecticide resistance. Our big-picture stance on this intervention has not changed much – we still consider it an intervention with outstanding evidence of effectiveness and cost-effectiveness – though we have noted some new concerns and caveats.
    • Regarding deworming, we published a post about a new systematic review of the evidence on deworming’s benefits, and we have a post forthcoming revisiting one of the major studies arguing that deworming has developmental benefits (i.e., that it leads to better outcomes later in life). We now find the short-term case for deworming’s health benefits to be weaker than we found it to be in 2011, and we find the long-term case for its developmental benefits to be stronger.
  • Kept tabs on other promising organizations and did a thorough investigation of GiveDirectly. We have considered GiveDirectly promising since its launch, but only recently did we feel that it had enough of a track record to facilitate a deep assessment. In early October of this year, we determined that it was the most promising contender for a top-charity slot (aside from AMF and SCI); we completed a thorough investigation including multiple rounds of reviewing documents and interviewing its CEO; a more thorough investigation of the academic literature on cash transfers (writeup forthcoming); and a multiple-day visit to see GiveDirectly’s operations in Kenya.
  • Looked for other outstanding giving opportunities in the area of global health and nutrition. As discussed earlier this year, we focused on finding ways to deliver the most promising interventions, rather than repeating our earlier approach of scanning a large list of charities. The two most promising interventions we looked into, immunizations and salt iodization, do not appear to have the sort of room for more funding that our top charities do (i.e., we do not see opportunities to translate additional dollars directly into additional people reached). We will be writing more about this in the future.

How much should you give to each charity?
We don’t believe in “diversifying” donations to charity, for the sake of “reducing risk” – we believe in giving in order to maximize the “expected total good accomplished,” which – by default – means finding the best giving opportunity and allocating it 100% of one’s charitable dollars. However, we do see legitimate reasons to divide one’s donation:

  • If you are giving a large enough amount of money, it’s possible to hit diminishing returns by giving it all to one charity. A simple example is that if I were giving $1 billion this year, I wouldn’t give it all to AMF, because that amount would well exceed AMF’s room for more funding.
  • A more subtle version of this idea pertains to learning opportunities. In a sense GiveWell is like a “large donor” with a few million dollars of anticipated money moved. If we direct major funding to more than one charity, we will have improved access to each such charity and will have improved opportunities to track its progress and learn from it. In addition, though we don’t anticipate moving enough money to overwhelm any of the three charities’ room for more funding, there is an argument that each marginal dollar means less to the charity in terms of improving its prominence, ability to experiment and plan, probability of turning out not to be able to scale further, etc.
  • For donors who think of themselves as giving not only to help the charity in question but to help GiveWell, we encourage allocating your dollars in the same way that you would ideally like to see the broader GiveWell community allocate its dollars. If every GiveWell follower follows this principle, we’ll end up with an overall allocation that reflects a weighted average of followers’ opinions of the appropriate allocation. (By contrast, if every GiveWell follower reasons “My personal donation won’t hit diminishing returns, so I’ll just give exclusively to my top choice,” the overall allocation is more likely to end up “distorted.”)

We’ve polled GiveWell staff and core supporters on how they plan to allocate their own funds, and after considering these subjectively, we have settled on a recommended allocation of 70% to AMF, 20% to GiveDirectly, and 10% to SCI. However, we believe there is substantial room for judgment calls and value disagreements here, and we expect to see many individuals deviate significantly from this allocation due to their judgment calls and values. Our rough picture of the judgment calls and value choices that matter most is as follows.

Program Program evidence Program cost-effectiveness Our confidence in the organization Potential for innovation /upside
Against Malaria Foundation Malaria control Very strong Very strong Very high High
Schisto-somiasis Control Initiative Deworming (treating parasitic infections) Fairly strong Highly uncertain, though may be strongest High Low
Give Directly Direct cash transfers to the very poor Strong on short-term consumption; moderate on investment Highly uncertain; may be strong Very high Very high

For donors who place high value on supporting strong overall organizations (in light of the fact that much of the impact of a donation depends on who’s in charge of allocating it), and/or promoting innovation and experimentation, we suggest a relatively higher allocation to GiveDirectly, and lower to SCI.

  • We believe that GiveDirectly has significant “upside”: it is experimenting with a type of intervention that is extremely rare in the nonprofit world, that it is hoping to experiment with many different approaches to this intervention, and that its approach may ultimately evolve and change significantly and may have a major impact on other parts of the nonprofit world.
  • We believe AMF has significant “upside” as well, though in a different way (and probably to a lesser extent than GiveDirectly). Its intervention is one that has a large community around it and a lot of funding behind it. Through its unusual degree of transparency and self-evaluation, AMF may set examples for others and/or produce information/insights that are helpful to the broader malaria control community.
  • While SCI is an outstanding organization, we do not believe it is as strong as the other two on this criterion.

For donors who are particularly “harm-averse” – such that they place significant weight on the “do no harm” principle as opposed to the “maximize expected good accomplished” principle (we subscribe to the latter) – we suggest a relatively lower allocation to GiveDirectly. Cash transfers (particularly as GiveDirectly structures them) have a higher risk of unintended consequences than the other two interventions. We see very little risk of harm for insecticide-treated nets or deworming.

For donors seeking to save rather than improve lives, we suggest a relatively higher allocation to AMF, the only charity of the three whose intervention has been linked with reduced mortality. For donors seeking to improve rather than save lives, we believe AMF should still get the highest allocation overall–the benefits of reducing the burden of malaria likely go well beyond saving lives. (GiveWell staff tend to prefer improving lives over saving them, and our suggested allocation takes this into account.)

For donors who are inclined to be less skeptical than we are of academic evidence and cost-effectiveness analysis, we suggest a relatively higher allocation to SCI, and lower to GiveDirectly. If one takes the evidence on deworming at face value, it indicates extremely impressive long-term benefits. Similar benefits may exist for reducing the burden of malaria, but deworming is substantially cheaper (~$0.50 per person per year, as opposed to ~$1.25) on a per-person basis.

For donors who are more skeptical than we are of academic evidence and cost-effectiveness analysis, and place a higher weight on the reasoning that “people are likely to make better decisions for themselves than we can make for them,” we recommend a relatively higher allocation to GiveDirectly.

We’ll be very interested to see what people end up doing. We encourage donors to post their planned allocations, and reasoning, as comments (or to email us if they’d prefer that their thinking stay private).

7 tips for giving efficiently

(updated on: ) | by Holden

If you’re planning on giving to charity this holiday season, there are a few simple steps you can take that can save a lot of money – allowing you to give more at the same cost to yourself – as well as reduce hassle.

1. Don’t wait until the last minute. Many donors wait until the very end of the calendar year to give. (Over 20% of our 2011 money moved, excluding gifts of $100,000 and more, came through in the last three days of the year alone.) Doing this will make it very difficult to execute some of the steps below (such as giving appreciated stock). And if something unexpected happens (as it often does with large credit card donations), you may have little time to react.

We recommend setting a target date of December 24 or earlier for finalizing your gift (for a week’s cushion).

2. Try to get a tax benefit. Details vary by country and personal situation, but a tax deduction can allow you to give much more to charity at the same cost to yourself. (That said, as discussed below, we believe it is more important to give to the most effective possible charity than to get the maximum tax benefit.)

Our #1 charity, Against Malaria Foundation, offers tax deductibility to citizens of the U.S., the U.K., and Canada. Our current #2 charity is a U.K. charity only, but U.S. donors can get a tax deduction for giving to GiveWell for the support of SCI.

If you’re in another country, you may wish to look into groups in your country that provide the service of (a) taking deductible donations themselves; (b) transferring the funds (minus fees) to charities abroad. While they can charge substantial fees, you may still come out ahead after tax considerations.

3. Avoid the large transaction fees and delays associated with large online donations. When donating via credit card, you will almost always be charged standard credit card processing fees. For donations under $5000, our feeling is that it’s worth paying the fees to avoid the hassle (both for you and the charity) of another donation method. But fees on a $5000 donation will generally exceed $100. In addition, credit card transactions of $5000 and up are often flagged by credit card companies (though a call to the credit card company can generally resolve the situation quickly).

This year, Google Checkout stopped offering fee-free processing, and we were unable to find a viable alternative with lower fees. Therefore, we are recommending that people giving $5000 and up consider writing a check or making a bank transfer. (Details)

4. Give appreciated stock. In the U.S., if you give stock to a charity, neither you nor the charity will have to pay taxes on capital gains (as you would if you sold the stock yourself). If you have stock that you acquired for $1000 (and has a cost basis of $1000) but is now worth $2000, you can give the stock to charity, take a deduction for $2000, and not have to pay capital gains tax on the $1000 of appreciation. This can result in significant savings. (More at Vanguard’s write-up on this topic.)

Taking advantage of this requires having some of your net worth invested in appreciated stock (or other securities), knowing (or being able to obtain) the cost basis of such securities, and arranging to transfer the stock directly to a charity, which is generally a fairly easy process, but varies from broker to broker; waiting until December 31 can be especially problematic if you’re trying to do this.

GiveWell can take direct transfers of stock for the support of our top charities, so let us know if you’re interested in this. A more robust way to smooth the process of giving appreciated securities is to take advantage of a donor advised fund, discussed immediately below.

5. Look into donor-advised funds to make the process smoother and more consistent year-to-year. Donor advised funds allow donors to make a charitable donation (and get a tax deduction) now, while deciding which charity they’d like to support later. The donation goes into a fund that is “advised” by the donor, and the donor may later recommend a grant from the fund to the charity of his/her choice.

We see a couple of advantages to this setup. One advantage is that you can separate your “decision date” (the date on which you decide which charity you’d like to support) from your “transaction date.” That means that if you aren’t ready to decide which charity to support yet, you can still get started on the process of transferring funds and getting a tax deduction for the appropriate year. Another advantage is that if you change the charity you support from year to year, you’re still working with the same partner when it comes to transactions, so the process for e.g. donating stock will not change from year to year. Donor advised funds are often set up to easily accept donated stock, whereas charities may or may not be.

Many large investment companies – Vanguard, Fidelity, Schwab – offer donor advised funds. They generally charge relatively modest management fees.

We do not recommend delaying this year’s donation while you look into donor advised funds. Donor advised funds can make giving more convenient over the long term, year-to-year, but you can get all the advantages of efficient giving this year without one. We also maintain our own donor-advised fund for donors interested in supporting our recommended charities.

6. Consider the political environment. If you believe that your tax rate is likely to be higher in 2013 than in 2012, you may wish to give at the very beginning of 2013 rather than the very end of 2012.

We’ll likely be counting donations made in the first week or so of 2013 toward our 2012 money moved (we’ve made similar adjustments in the past).

7. Choose your charity wisely. Saving money on taxes and transaction fees can be significant, approaching or exceeding a 50% increase in the amount you’re able to give. However, we believe that your choice of charity is a much larger factor in how much good your giving accomplishes.

Our charity recommendations make it possible to support outstanding, thoroughly vetted organizations – which we’ve investigated by reviewing academic evidence, interviewing staff, analyzing internal documents, conducting site visits, assessing funding needs, and more – without needing to do your own research. We publicly publish the full details of our process and analysis, so you can spot-check whatever part of our work and reasoning you’d like to.

Final notes.

  • If you support our recommended charities (on the basis of our recommendation) but you don’t give through our website, please use the donation report form to let us know about your gift; doing so helps GiveWell track our impact.
  • We believe that even when dealing with a relatively complicated gift (for example, a gift of stock), it’s possible to give quite quickly and with only minor hassle. The much more difficult challenge is choosing a charity, and we’ve tried to make that easy as well. So we hope you’ll give this season, even if you’re just starting to think about it now; if you’d like more advice or help, don’t hesitate to contact us.

GiveWell’s annual charity recommendations refresh

| by Elie

As we do every year, we’re planning to release updated charity recommendations by December 1st. We’re currently weighing what we’ve learned about the organizations that could receive our top ratings. Currently, the contenders for top spots are Against Malaria Foundation (AMF) (our current #1 charity), Schistosomiasis Control Initiative (SCI) (our current #2 charity), and GiveDirectly (which we have written about before, but haven’t previously recommended due to its newness; now that it has more of a track record we are taking a closer look).

As part of this work, we’ve recently released updates on AMF (latest update here) and SCI (latest update here). I also recently returned from a trip to Kenya to see GiveDirectly’s work in the field, and we are working on an updated review of GiveDirectly that incorporates all we’ve learned recently.

Insecticide resistance and malaria control

(updated on: ) | by Holden

A recent Scientific American post discusses a study that may indicate an emergence of “behavioral resistance” to insecticide-treated nets. Mosquitoes in Benin were found to shift their “mean catching time” (roughly speaking, the time of day when they were most likely to be active) from 2-3am to 5am; this may indicate changing behavior to attack humans closer to the daytime (when they’re more likely to be away from their insecticide-treated nets) as opposed to the middle of the night.

Along with a study released last year showing increased insecticide resistance following a net distribution campaign, this study highlights a major concern for supporters of insecticide-treated nets (ITNs): ITNs may become less effective as mosquitoes adapt to them – whether by evolving resistance to insecticide or changing their behavior. We are concerned about this issue, seeking to learn more about it, and even considering recommending funding research on it. That said, we think the balance of considerations still points to ITN distribution as an outstanding intervention.

Below, we lay out what we know regarding key questions around ITN resistance (defined broadly as “any ways in which populations of mosquitoes adapt to the presence of ITNs in order to make them less effective”). We then give some general considerations in favor of not weighing this factor too heavily against ITN distribution for the time being.

We address the following questions:

  • What is known about the current state of resistance? A fair amount of data has been collected on some forms of resistance; less has been collected on other forms. The resistance data that is available points to widespread (but not universal) resistance in western Africa, little resistance in southern Africa, and very little data for central Africa. More
  • Is the prevalence of resistance increasing and is this caused by ITN distributions? The consensus view among malaria scholars seems to be that prevalence of resistance has been increasing, but we have not been able to find evidence for this view that clearly distinguishes “increasing prevalence of resistance” from “increasing efforts to monitor resistance and thus increasing reports of resistance.” If prevalence of resistance has been increasing, it may be due to multiple factors, including but not limited to ITN distributions. More
  • How does resistance affect the effectiveness of ITN distributions? Unfortunately there is little data available to assess this issue; it isn’t standard to collect data on malaria cases as part of an ITN distribution, and it seems that cases in which one can pinpoint the timing of both an ITN distribution and subsequent trends in malaria cases are relatively rare.It appears that there has only been one confirmed case of malaria control (in this case, insecticide spraying) failing to control a mosquito population due to resistance. There are several cases of ITNs’ apparently remaining effective despite widespread resistance, and many plausible explanations for why this could be the case. However, it appears that relatively little is known about this issue; it appears that more study could shed significantly more light on the matter. More
  • What can aid groups do in order to manage resistance? Various techniques for mixing/rotating different insecticides may be used to manage resistance in areas where it is problematic; unfortunately, however, there is currently only one class of insecticide that can be used on ITNs. So in areas where resistance is confirmed to be an issue, other control methods can be brought in, but in areas where little is known and only ITNs are used, mixing/rotating insecticides is not an option.Despite this, the World Health Organization recommends the continued use of ITNs even in places where resistance is confirmed; they have several ways of being effective and are expected to retain partial or full effectiveness.

    Longer-term, work is underway developing new technologies, including ITNs that use different insecticides, to deal with this issue. More

We see resistance as a major concern; a reason that it’s important to continue collecting data on the real-world impact of LLINs (as Against Malaria Foundation is attempting to do); and an area of research that may represent a good giving opportunity itself.

That said, our positive view on LLINs remains in place. There is strong evidence that LLINs reduce malaria and save lives and only preliminary/suggestive/mixed evidence that insecticide resistance may reduce their impact. In addition, it appears to us that the malaria control community has been devoting at least some attention and investigation to this issue for a long time, has developed a reasonable knowledge base (if one that has plenty of room to grow), and still recommends the use of LLINs regardless of the resistance situation.

Indeed, the fact that we’re discussing this issue at all speaks to the extraordinarily and unusually strong evidence base (and supply of data) behind ITN distribution. For most aid interventions that donors can fund, the set of “things that could go wrong” is large and broad, and we have little evidence to address most of them, but when looking at LLIN distribution, the salient concerns are few and specific enough that the malaria control community is able to put substantial resources into specifically investigating them.

Note that

  • This post does not give citations, but the technical version of this writeup does.
  • Against Malaria Foundation, our #1 charity, has made a post of its own on this topic.
  • Throughout this writeup, we primarily rely on two review papers (World Health Organization and Ranson 2011) whose claims we have not vetted (by, for example, examining the studies they cite); for the most part the claims cited here are fairly straightforward and we have relatively little cause for skepticism about them.

What is known about the current state of resistance?
“Resistance” can be used to refer to genetic properties of mosquitoes, to refer to behavioral properties of mosquitoes, or to refer directly to failures of mosquito control. In the context of malaria control, we’ve generally seen it used to refer to the first two; as discussed below, there is relatively little evidence of control failure due to these factors, so “resistance” is usually used to discuss specific risk factors for control failure.

Ranson (2011) provides a map summarizing the available information about resistance, using publications from 2000-2010. In this context, “resistance” is defined by the percentage of mosquitoes that die after a set period of exposure to insecticide under laboratory conditions; anything under 80% is considered “resistant” and anything under 97% is considered “possibly resistant.” This measure is intended for sensitive, early detection and does not necessarily indicate ineffectiveness of control measures.

Resistance appears common (though far from universal) in the area of western African running from Cote d’Ivoire through Cameroon; it is less common in the countries west of that and in southeast Africa; data from other parts of Africa is thin or nonexistent.

Another map in the same document lists what is known about the spread of genes known to be associated with resistance; detecting genes is believed to be an even more sensitive and earlier way to detect threats of resistance. If these genes were fully responsible for the observed resistance in laboratories, that would be good news, since resistance associated with these genes appears to us to have been most heavily studied and (as discussed below) appears to be associated with relatively lower risks of control failure. However, not all of the resistance observed in the map above can be attributed to these genes, and metabolic resistance (a different type) has been implicated in 10 studies in 8 countries.

The above covers only target-site and metabolic resistance, which make insecticides less effective in killing mosquitoes; by the nature of the laboratory test, it does not address behavioral resistance (mosquitoes’ changing behavior in order to avoid nets). There is currently relatively little evidence regarding behavioral resistance and no clear consensus about its importance. (Though we would encourage readers to take recent reports of possible behavioral resistance with a grain of salt. Mosquitoes’ altering their behavior could reflect many things and doesn’t necessarily imply adaptations to nets; behavior adaptations such as seeking to feed during the daytime may make mosquitoes easier to detect and thus more vulnerable; and most importantly, we would always caution against putting too much weight on a single study, particularly a recent one).

Is the level of resistance increasing? Is this caused by the use of ITNs?
It has been difficult for us to reliably distinguish the increase in reports of resistance (caused by increased efforts to assess resistance) from an increase in actual prevalence of resistance; reports on resistance often do not draw clear distinctions between the two. We also have been unable to find a comprehensive review regarding the question of what factors appear to have contributed to the development of resistance. Ranson (2011) states that there is reason to believe that ITNs contribute to the development of resistance, but resistance may also be brought about through the use of agricultural pesticides and other factors:

the primary source of this selection pressure [for resistant mosquitoes] is still unclear although evidence is accumulating to incriminate the agricultural use of insecticides in selection for insecticide resistance in malaria vectors. For example, seasonal fluctuations in results from WHO bioassays in major agricultural regions of Sudan and Cameroon reflect the timing of application of insecticides to the key crops in the region [45, 54]. In addition several reports have documented a higher incidence of insecticide resistance in areas of intensive agriculture than found in comparable non-agricultural sites [58, 59, 128]. There are also examples where use of pyrethroids for malaria control appears to be the primary factor in selecting for resistance [62, 129, 130]. Other sources of selection pressure including the use of consumer products such as coils or aerosols, or selection due to other toxins permeating mosquito breeding sites may also be contributing to the resistance phenotype [25, 131].

What is the impact of resistance on ITN distribution and other control measures?
It appears to us that

  • There is relatively little data available to assess this issue. It isn’t standard to collect data on malaria cases as part of an ITN distribution, and it seems that cases in which one can pinpoint the timing of both an ITN distribution and subsequent trends in malaria cases are relatively rare. We’ll be writing more about this general issue in the future; our impression is that what data there is implies that malaria has been in decline and (less strongly) that ITNs have been effective, in general, but the case is far from comprehensive.
  • There aren’t many cases in which a study found a failure of malaria control as a result of resistance. The clearest-cut case, according to both of the sources we’ve relied on most for this discussion, was around 2000 on the border between South Africa and Mozambique. Ranson 2011 cites another, less clear-cut, case from the island of Bioko on the West African coast, and both sources cite additional cause for concern from studies using experimental huts (though Ranson 2011 notes that the evidence from these studies is mixed).
  • There are multiple studies implying that ITNs can still be impactful even in areas where resistance is confirmed. However, these studies may be of limited use since they are focused only on a particular kind of resistance.
  • The reviews we’ve focused on express the belief that resistance is likely to become a problem eventually and that addressing it should be considered an urgent priority.
  • Our impression is that resistance was also a factor in earlier malaria control efforts, but may not have been as rigorously studied; the sources we’ve reviewed do not refer to these earlier efforts.

There are multiple reasons that ITNs may retain effectiveness even against mosquitoes that are “resistant” (in the sense of demonstrating low mortality rates in laboratory settings). In addition to the fact that ITNs provide a physical barrier, the insecticide may also repel mosquitoes (and cause them to seek out other targets) even when it does not kill or fully disable them. In addition, it’s possible that mosquitoes are still killed by the insecticide (despite reduced susceptibility) when they have enough contact with it; that insecticide may inhibit them in other ways that stops them from transmitting malaria; that resistant mosquitoes are less fit overall or less prone to transmitting malaria; or that mosquitoes that are resistant to insecticides at young ages may become less resistant as they age (and that older mosquitoes are more relevant to malaria transmission.) We have come across one study that argues for the last of these phenomena.

What can aid actors do about this issue?
One approach to managing resistance is to periodically change the type of insecticide being used, so as to reduce selective pressure for resistance to particular insecticides. (This may reverse the spread of resistance, if resistant mosquitoes are less fit in other ways.) However, there is only one class of insecticide (pyrethroids) approved for use on LLINs, so in areas where LLINs are the main form of malaria control, options are relatively limited (though in cases where resistance is confirmed to be having a major effect, other methods may be brought in specifically in order to get the benefits of using multiple insecticides). The World Health Organization recommends that in all circumstances – even where resistance is confirmed and is interfering with control efforts – LLINs should continue to be promoted.

Hopefully, there will be LLINs with different insecticides that are usable in the future; research has been done toward this goal and has made preliminary progress.

Closing notes
We’re very concerned about insecticide resistance. It appears that relatively little is known about the extent, causes, and control implications of resistance; that resistance could substantially effect (or even negate) the effectiveness of malaria control; and that more research has great potential to improve the impact of the substantial amounts of money spent on LLIN distribution (and other forms of mosquito control). This investigation has highlighted research on insecticide resistance as a potentially outstanding giving opportunity itself, and we hope to look into it more.

That said, this issue doesn’t change our bottom line that LLIN distribution is a highly cost-effective intervention or that Against Malaria Foundation is our #1-rated charity. There is strong evidence that LLINs reduce malaria and save lives and only preliminary/ suggestive/ mixed evidence that insecticide resistance may reduce their impact. Importantly, it appears to us that the malaria control community has been devoting at least some attention and investigation to this issue for a long time, has developed a reasonable knowledge base (if one that has plenty of room to grow), and still recommends the use of LLINs regardless of the resistance situation (as noted in the previous section). We have previously been impressed with the thoughtfulness and data behind malaria scholars’ answers to our concerns over the question of why universal coverage is now being pursued (as opposed to targeting children under five), the question of whether ITNs delay the development of immunity and thus merely delay malaria deaths, and the question of whether ITNs are used by recipients (though we’ve had more trouble when it comes to information about how long ITNs remain in use in the field), and we are inclined to put reasonably high weight on recommendations such as this.

Finally, we’d like to note that

  • We see it as very important to monitor and report changes in the malaria burden following ITN distributions, as Against Malaria Foundation is attempting to do. This isn’t a substitute for studying resistance specifically, which may help the malaria control community anticipate problems before they arise. But it seems important as a way to know when resistance – or something else – has become a big enough problem to interfere with the effectiveness of ITNs in the field.
  • We think the fact that we’re discussing this issue at all speaks to the extraordinarily and unusually strong evidence base (and supply of data) behind ITN distribution. For most aid interventions that donors can fund – water infrastructure, agriculture interventions, etc. – the set of “things that could go wrong” is large and broad, and we have little evidence to address most of them. By contrast, when looking at LLIN distribution, we see strong basic evidence for impact and reasonably good (if not conclusive) answers to the most obvious concerns about how real-world effectiveness compares to the efficacy seen in studies. The remaining concerns are few and specific enough that the malaria control community is able to put substantial resources into specifically investigating them.