The GiveWell Blog

Obstacles to “giving as consumption”

(updated on: ) | by Holden

GiveWell’s traditional work (the work behind our current top charities) and our work on GiveWell Labs reflect two very different visions of giving.

  • The first, giving as consumption, sees giving as analogous to making a purchase. For every $X one spends, one gets some desirable outcome (such as a life saved), and the goal is to find giving opportunities that can deliver these outcomes approximately linearly and with good value-for-money.
  • The second, giving as investment, sees giving as analogous to investing in a company. Risk is known to be high, and outcomes hard to foresee in detail (particularly for earlier-stage investments). Rather than asking “What will each $X buy?” one tends to focus more on questions like “Does this organization have a good team and model?” and “Is this organization positioned to have a huge upside, even if one can’t say in advance just what this would look like?”

These don’t represent strict categories – any giving opportunity can be described in the language of either – but they reflect different ways of thinking about giving, and tend toward different ways of evaluating (and describing) the pros and cons of different giving opportunities.

Over the years, we’ve come to believe that “giving as consumption” is an apt framework only for a tiny minority of giving opportunities today – particularly if one seeks strong evidence and cost-effectiveness – and “giving as investment” is likely to be the more broadly useful framework for evaluating giving opportunities. Recent developments with our top charities (see recent posts on AMF and SCI) have moved us further in this direction.

Our audience has expressed (and we have felt) confusion over how hard it seems to be to find proven, cost-effective giving opportunities along the lines of “$X saves a life.” Many of the major funders we talk to – who tend to prefer the “giving as investment” model – have expressed views along the lines of “It’s easy to spend some money, buy some bednets and save some lives; we’re being more ambitious.” Yet we believe that proven, cost-effective “giving as consumption” is a very hard thing to find, and we think we can explain why.

In brief:

  • There are a relatively small number of programs that are strongly evidence-backed, in terms of their ability to improve lives. Evidence about the impact of a charitable intervention on life outcomes is inherently difficult and expensive to collect; it faces a large number of inherent challenges in terms of reliability and generalizability. More
  • Of evidence-backed programs, many are delivered widely via governments and the global aid community. That leaves quite a small number of interventions that both (a) are strongly evidence-backed (b) have room for more funding. In many cases, these are interventions for which the strongest evidence is relatively recent, such as cash transfers and deworming. More
  • Charities often aren’t structured to focus on directly delivering a small number of programs. Charities that do deliver evidence-backed interventions often present a further challenge to the “giving as consumption” model for a couple of reasons: (a) they often seek to promote their programs to governments (in addition to, or instead of, delivering the programs themselves), and (b) they often work on a broad variety of activities (making it difficult to understand the marginal impact of one’s donation). More
  • “Rollout” is rarely straightforward. Because collecting rigorous evidence is expensive, difficult and slow to bear results (along the lines of our first bullet point), there’s often great uncertainty in how a program is adapting – and how relevant its evidence base remains – in new contexts and changing times. More than one of our recommended charities has struggled to reach a new level of scale, and with all our priority programs there are major questions about how closely the context in which they were studied matches the context in which they’re executed today. More
  • Transparency and accountability present a further challenge. The current culture around charities is one in which nuance, substance and criticism are extraordinarily rare. Even if a charity has relatively strong evidence of impact, it may hesitate to share the technical details and subject itself to the sort of scrutiny – and honest discussion of strengths, weaknesses, successes and shortcomings – that we seek to provide. Doing so is also a major time investment, so the incentive to do so has to be sufficient. As our money moved has grown, we have had more success in getting engagement from charities on this front. More

Below, we expand on each of these obstacles to finding good “giving as consumption” opportunities, and discuss how (if at all) one might work to overcome them. We believe that GiveWell has played a role in improving the quality of “giving as consumption” opportunities, and will continue to do so. At the same time, we think that the “giving as consumption” framework will likely always apply to only a small fraction of giving opportunities, and broadening our horizons (via GiveWell Labs) is essential.

In the meantime, we note that GiveDirectly is the only giving opportunity we can name today that seems to strongly and closely fit the full “giving as consumption” model: focus on an evidence-backed program, using funds to directly implement the program, and producing and publicly sharing strong monitoring and evaluation data from its implementation. This doesn’t mean that GiveDirectly is the best of our top charities: as discussed in our announcement, the others have major potential advantages. But it does mean that our traditional criteria – “proven, cost-effective, scalable” – no longer describe our full set of top charities. Accordingly, we have modified our criteria page and our process page so that they accurately describe what distinguishes our top charities from others – and list what we see as the pros and cons of supporting them. (More details below.)

Obstacle 1: a relatively small number of programs are evidence-backed
We’ve searched extensively over the years for proven, cost-effective programs that charities can deliver. At this point, we believe it’s likely that we’ve identified all or nearly all such programs.

We believe the problems of selection bias and publication bias are serious ones. As such, we’ve become skeptical of much of the research that is published, and we’ve developed principles for how to identify strong evidence. (See Our Principles for Assessing Evidence, How We Evaluate a Study, Surveying the Research on a Topic).

With these principles in mind, strong studies are relatively rare, especially in areas outside of health. A strong study often involves using randomization to create a “treatment group” (receiving a program) and a “control group” (not receiving the program, and thus serving as a basis for comparison); following both the treatment and control group over a long period of time; and seeking objective measures of whether the program improved lives. Such evaluation can be very expensive and take a long time to execute, and thus it can be important to put a great deal of care and planning into it so as not to waste time and resources. Such evaluation generally manifests as academic literature, and we’ve never encountered a case in which a charity continually builds randomized (or other rigorous) evaluation into a substantial portion of its day-to-day programming.

Health programs are generally easier to study than other programs, for a couple of reasons. Appropriate measures to evaluate health programs can be relatively straightforward – death rates, prevalence of various markers and symptoms of disease, etc. – and it often appears that results can be generalized fairly well. By contrast, examining the long-run impact of e.g. an education or job training program on quality of life can involve questionable measures, very long time horizons, and substantial questions over whether the results in the study will generalize to new contexts.

We discuss the proven cost-effective programs we’ve found, and our process for finding them, at Priority Programs.

Prospects for progress on this obstacle. Developing new health technologies and approaches could lead to new priority programs. In addition, there are cases of existing health-based programs that we feel could use stronger evidence than they have (these include deworming and clean water interventions).

Conducting more rigorous studies of non-health programs – as Innovations for Poverty Action and others do – could lead to strong evidence bases for non-health programs. However, creating a strong evidence base for a non-health program appears to be extremely difficult, and we’ve seen it happen only in one case to date (the relatively, though not extremely, strong evidence base for cash transfers). Part of the reason for this may be the inherent difficulty, high expense and long time horizon of assessing non-health impacts in a generalizable way; there may be some more complex contributing factors as well (it seems to us that many of the people conducting these studies are economists, who are often more interested in exploring a general principle and/or demonstrating a new idea than in conducting the second, third, fourth, etc. study on a program that has already been evaluated once).

We’re hopeful that by creating and demonstrating demand for evidence-backed programs, GiveWell can improve incentives for people to carry out more such studies.

Obstacle 2: many evidence-backed programs are delivered widely via governments and the global aid community
We wrote at length about this issue early this year. At the time, one of the major exceptions we saw was that of bednets, but even on that front the funding picture appears to be improving dramatically. Total global health aid is in the range of $30 billion per year and much of the “lowest-hanging fruit” is already picked.

This dynamic is easy to underestimate, because charities may advertise themselves as delivering certain interventions (such as vaccines). However, donations toward this end may ultimately be fungible with other donations. There’s a big difference between “having your donation allocated to paying for vaccine delivery” and “having your donation cause more vaccines to be delivered.”

It’s a good thing that governments and multilateral organizations play a major role in rolling out proven, cost-effective programs. However, it also means that the opportunities for donors to “buy” more of such programs are less common than they may appear.

Deworming and cash transfers are both programs whose strongest evidence is relatively recent. Because of this, they may currently be in a “sweet spot” between becoming widely accepted as good uses of funds and becoming widely rolled out – creating opportunities for private donors to fill gaps before larger funders do, and in some cases to leverage larger funders via advocacy and technical assistance.

Obstacle 3: charities often aren’t structured to focus on directly delivering a small number of programs
The Deworm the World Initiative (DtWI) is an example of trying to take advantage of the kind of “sweet spot” described in the previous section. It focuses on advocacy and technical assistance in order to encourage, and assist, governments to execute large-scale deworming programs. Schistosomiasis Control Initiative (SCI) appears to have a mix of activities: in some cases it uses funds to carry out programs itself, and in others it uses them to fill gaps, do advocacy, and create pilot programs in the hopes of influencing governments and other funders. (For example, see our most recent update on SCI.)

These are very logical activities for a charity – arguably more logical (if they can successfully influence larger funders) than direct delivery. However, these activities pose serious challenges from an evaluation perspective: whereas GiveDirectly can show documentation on programs it has directly carried out, it’s much harder to say just what donations to DtWI have accomplished (and will accomplish).

Another structural issue is that many charities work on a broad variety of activities. We discuss some such charities in a 2011 post. When dealing with such charities, it can be extremely challenging to trace the impact of additional donations.

Prospects for progress on this obstacle. We don’t believe it’s necessarily a bad thing for charities to focus on harder-to-evaluate activities such as technical assistance and advocacy, and accordingly, we’ve modified our criteria so that they no longer require organization-specific evidence of effectiveness. Regarding large broad charities working on a variety of activities, we believe that it may become easier to trace the impact of marginal dollars as the dollar amounts become larger (at a high enough level of funding, one can fund an entire project that wouldn’t have been on the agenda otherwise).

We see hope for progress on being able to evaluate sub-programs of charities that work on multiple activities. Our recommendation of DtWI represents the first time that we’ve recommended an initiative within a larger organization; given the amounts of money moved we anticipate this year, we are comfortable with this and anticipate being able to assess the impact of GiveWell-directed funds on deworming activities relatively well.

Obstacle 4: “rollout” is rarely straightforward
The context in which a program was studied is often very different from the context in which it is now being carried out:

In addition, a charity can hit many unanticipated obstacles as it expands from one area to another:

We generally seek as much monitoring and evaluation as we can from the organizations we review, so that we have a basis for comparing their programs as they are executed today with the programs that studies have demonstrated the effectiveness of. However, as discussed above, we have never seen a case in which highly rigorous evaluation is integrated into a charity’s general activities; the data that is available generally has major limitations (many of which aren’t immediately apparent – for example, see our recent post about SCI’s evidence of impact).

Prospects for progress on this obstacle. Changing contexts will always be a real challenge, but we are optimistic that we will see more, and better-designed, monitoring and evaluation over time to act as a diagnostic. We feel that Against Malaria Foundation has dramatically improved its approach to monitoring and evaluation, adding measures to assess whether the insecticide-treated nets it distributes remain in use and are effective over time. GiveDirectly, which is the recommended charity with the strongest overall monitoring and evaluation from our perspective, has only been in existence since 2011. In both cases, we feel that GiveWell’s existence provided encouragement – and funding – for strong monitoring and evaluation, though we can’t make a firm statement about the impact we’ve had (it’s possible that in both cases the same outcome would have occurred if GiveWell had not existed). If we continue to demonstrate strong and growing money moved, and continue to be public about what we see as the strengths and weaknesses of the evidence collected by recommended charities, we expect to see more improvement over time.

Obstacle 5: transparency and accountability
In our experience, it’s almost never possible to understand the case for a charity’s impact based on publicly available materials. Monitoring and evaluation reports are generally internal (and require assistance to interpret); the sort of information necessary to understand room for more funding often requires extensive back-and-forth to obtain. Thus, if we seek to thoroughly understand a charity’s work, it’s generally necessary that the charity voluntarily participate in our extensive evaluation process.

This has been a significant issue throughout our existence. We’ve tried to make participation in our process as attractive to charities as possible, which has generally included sending them previews of content we’d like to publish about them, and the discussions we’ve had around preview content have consistently given us the impression that charities are uncomfortable (more so than other organizations) being discussed publicly in anything but glowing terms. In addition, it has often been surprisingly difficult to get charities to share substantive information even when they have it. We’ve often been connected to fundraising-oriented staff who have sent promotional materials in response to our substantive requests. (For some flavor of this, see the list of grant application responses from our 2007 process; the page is out of date and has problematic formatting but still contains the relevant materials.)

At this time, we believe that the idea of public, thorough evaluations of charities – discussing weaknesses as well as strengths, shortcomings as well as successes – remains a very unusual (and, to many charities, discomfiting) idea. It is a major factor in why there are so few GiveWell-recommended charities.

With that said, there has been real progress.

  • We have seen dramatic improvement over the years from SCI in terms of sending substantive information to help us understand its activities.
  • This year (accompanied by a change in leadership), DtWI changed from an organization that we knew very little about to an organization that participated eagerly and thoroughly in our process, leading to a recommendation.
  • GiveDirectly has been highly transparent since its inception in 2011, when it allowed us to publish a pre-analysis plan for a study on its work. We believe that GiveWell may have played a role in causing GiveDirectly to share more information publicly than it would have otherwise (although its founders have shown what we perceive as a genuine openness to and enthusiasm about extreme transparency).
  • We have also recently seen interest from other charities that previously declined to participate in our process, and we plan to follow these up in the coming year.

Prospects for progress on this obstacle. We are highly optimistic that if we can continue to demonstrate strong and rising money moved, we will see more and more interest from charities in participating in our process and being thoroughly and publicly evaluated.

With that said, we also recognize that participation in our process is a major ask, and this year we have begun providing a list of charities that focuses on evidence-backed programs, regardless of whether they participate in our process, so that donors can support these if they prefer.

Bottom line

There are many reasons it’s hard to find a proven, cost-effective giving opportunity, along the lines of “Every $X I spend accomplishes Y.” We have seen – and perhaps played a role in – substantial progress on some fronts, particularly with regard to (a) improved monitoring and evaluation and (b) improved willingness of charities to share substantive information and be thoroughly, publicly evaluated and followed up. We expect more progress, and we believe that our offerings for donors interested in “giving as consumption” will improve. GiveWell’s money moved provides an incentive for evidence-backed charities to share information, and an incentive for younger charities to build evidence and transparency in from the beginning. We are also contemplating other measures we can take (aside from our charity recommendations) to contribute to progress, such as providing seed grants to help potential future top charities get started.

With that said, some of the obstacles to “giving as consumption” are fundamental, and we now believe that this framework will – for the foreseeable future – apply to only a very small subset of giving opportunities. GiveWell Labs is exploring giving opportunities using a very different framework.

In the meantime, we find ourselves with very little in the way of giving opportunities that we can call “proven,” even though we do have giving opportunities that we consider outstanding (partly due to more “investment” type factors, which we discuss at our top charities announcement).

Accordingly, we have modified our criteria page and our process page so that they accurately describe what distinguishes our top charities from others – and list what we see as the pros and cons of supporting them. In brief, we have removed language implying that charities are “proven” in the sense of having demonstrated their own impact, or “scalable” in the sense that we can precisely predict and quantify the impact of additional donations. Instead, we emphasize that our top charities (a) focus on evidence-backed programs serving the global poor; (b) have been thoroughly vetted, such that even if we haven’t pinned down the answer to each of our key questions, we’ve gathered as much information as we can and can make informed statements about our top charities’ strengths and weaknesses. For those who are concerned that openness to our process is not a desirable quality in a charity, we also provide a more extensive list of charities focused on proven cost-effective programs.

We also now prominently discuss the pros and cons of giving to our top charities:

We think the principal advantages of our current top charities are that:

  • They represent the best opportunities we’re aware of to help low-income people with relatively high confidence and relatively short time horizons. If you’re looking to give this year and you don’t know where to start, we’d strongly recommend supporting our top charities.
  • Due to the emphasis on thorough vetting, transparency, and following up, our top charities represent excellent learning opportunities, and we feel that one of the most desirable outcomes of giving is learning more that will inform later giving. Supporting our top charities helps GiveWell demonstrate impact and improves our ability to learn, and we are dedicated to sharing what we learn publicly.

Some counter-considerations:

  • There is an argument for saving money rather than giving, and giving at the point where better information on top giving opportunities is available. We do expect to make substantial progress on GiveWell Labs over the next few years.
  • If you have access to other giving opportunities that you understand well, have a great deal of context on and have high confidence in — whether these consist of supporting an established organization or helping a newer one get off the ground — it may make more sense to take advantage of your unusual position and “fund what others won’t,” since GiveWell’s research is available to (and influences) large numbers of people.

Good Ventures matching gift to GiveDirectly and grants to top charities

(updated on: ) | by Holden

Good Ventures has announced:

  • Grants to our top charities: $2 million to GiveDirectly, $1.5 million to Deworm the World Initiative, $750,000 to Schistosomiasis Control Initiative.
  • A match, up to $5 million total and $100,000 per donor, on donations made to GiveDirectly from today through January 31, 2014.

Good Ventures spells out its reasoning here.

We very much appreciate Good Ventures’ generous support of our top charities.

These decisions were made by Good Ventures and not GiveWell, but we were part of the discussions leading up to them, and thought we ought to briefly summarize what recommendations we’ve made and thoughts we’ve shared on this front.

On the grants, we advised Good Ventures to do as it did and agree with the reasoning stated in its announcement. We believe that Good Ventures will have better giving opportunities available to it in the future than today, and we also see negative aspects to “crowding out” the donations of individuals, so we did not advise it to close top charities’ full funding gaps, but we did advise it to help ensure that the minimum targets are hit.

On the match, we raised the possibility of matching donations to GiveDirectly. We stated that we saw positive value in encouraging other individuals to give to a top charity, and that we believed money given to any top charity – particularly GiveDirectly, which seems to have the most absorptive capacity and which most staff are favoring for their personal donations this year – is money well spent. Ultimately, Elie and I recommended against the match (feeling that these funds would be better spent on a future giving opportunity), while stating that we believed there were strong arguments both ways and that we respected and supported Good Ventures’ decision. Alexander recommended in favor of the match.

We’re aware of the tension between supporting a match and our previous writing on matches. We believe that a maximally rational approach to giving should not be influenced by most donation matching. However, Good Ventures’ match is targeted at influencing a broad range of donors, many of whom may not fully align with our view and may be positively influenced by the match offer. And we also believe it is not misleading in the way that many matches are: the purpose of the match is to bring in more donors, and there is a legitimate possibility that the full match will not be taken advantage of (in which case those funds simply won’t be donated to GiveDirectly). If the full amount of the match is reached, we believe Good Ventures will be giving more in total to our top charities this year than they would have been had a match not been a possibility.

Good Ventures is more positive on GiveDirectly than on our other top charities, to a greater degree than most GiveWell staff. We’re still planning for individual staff members to discuss their planned personal allocations in the next couple weeks.

Rigorous study of GiveDirectly’s cash transfers

(updated on: ) | by Holden

Carrying out a highly rigorous study is difficult, expensive, and can take a long time. It’s very rare to see a charity that has carried out a highly rigorous study of its own work – one that can isolate the impact of its program from all the other factors affecting its clients. As such, we generally recommended charities running programs that have independently been rigorously studied, and this approach leads to major challenges in generalizing.

  • Against Malaria Foundation distributes bednets, which have been found to save lives and reduce malaria in multiple high-quality studies – but all of these studies were carried out over 10 years ago, when insecticide resistance was less of a concern and general health was likely worse (leading to possibly elevated impacts for nets), and the programs studied were different in some important ways from a typical modern net distribution.
  • Schistosomiasis Control Initiative and Deworm the World Initiative carry out deworming, a program whose case for effectiveness rests on two studies: one on a hookworm eradication campaign in the American South in the early 20th century, and another from a randomized controlled trial in Kenya during El Nino (in which El Nino led to greatly elevated levels of worm infection, and thus likely greatly elevated impact of treatment).
  • Previously, our case for cash transfers rested on a variety of studies looking at programs that were structured differently – in important ways – from GiveDirectly’s program. In particular, they often involved conditional cash transfers (transfers given in exchange for e.g. sending children to school and engaging in other donor-desired behavior) and generally involved smaller amounts given over longer periods over time.

This year, for the first time, we’re aware of a charity that can point to a highly rigorous and recent study of its own work. The charity is GiveDirectly and the study is Haushofer and Shapiro 2013, a randomized controlled trial (with a pre-analysis plan to boot) reporting:

  • Increased consumption across a variety of categories, including food, education, business expenses and durable assets, but not alcohol and tobacco.
  • Improved standing on a food security index (which was one of the topmost outcomes of interest according to the pre-analysis plan) and an index of psychological well-being.
  • An (unelaborated) claim in a recently updated report on the study that spending on iron roofs produced a return on investment of 23%.

There are, as with any study, major limitations.

  • The majority of recipients received smaller transfers than GiveDirectly’s standard program provides (though some received roughly the same size transfer).
  • Much of the data is based on self-reports (though this is also true of other studies, such as those supporting deworming, that look at income and consumption).
  • The study only covers the area GiveDirectly was working in at the time; as GiveDirectly expands, it will encounter substantially different environments.
  • The study is relatively short-term, and we hope there will be follow-ups (we know there are some plans on this front).

More details at our full discussion of the Haushofer and Shapiro 2013

The Economist has also discussed this study.

Overall, this study represents a substantial update on GiveDirectly, and puts it in something of a unique position relative to other charities we’re aware of.

It seems to us that much of our audience overestimates the evidence base for deworming and underestimates the evidence base for cash transfers. In our view, GiveDirectly has – all things considered – the most strongly evidence-backed intervention of our current three top charities (though we’d still say the case for bednets is stronger).

GiveWell’s top charities for giving season 2013

(updated on: ) | by Holden

Our top charities for this giving season are GiveDirectly, Schistosomiasis Control Initiative (SCI), and the Deworm the World Initiative (DtWI) (led by Evidence Action). The first two have been recommended previously. The third is a new addition, and we encourage readers to see our full review.

We’ve had extensive internal discussions about how to rank these three giving opportunities against each other. These discussions have been particularly challenging this year because:

  • The three have extremely different strengths and weaknesses, in our view. Much of the decision between them comes down to judgment calls about how to weigh factors such as (a) estimated cost-effectiveness of a charity’s activities (SCI appears strongest here, in isolation, though some staff discount this observation because of the low robustness of our estimates); (b) potential leverage (DtWI has a potential advantage here, as it focuses on influencing government spending rather than covering all costs with its own funds); (c) a direct and robust case for impact (we believe GiveDirectly is strongest on this front); (d) “upside,” in terms of potential for the charity’s growth to have far-reaching effects beyond the direct impact of its services. (We think all three have some “upside” but see GiveDirectly as particularly promising in terms of its potential to improve the aid community’s understanding of, and interest in, cash transfer programs.)
  • The amounts of money given on the strength of our recommendations could be substantial, and so the question of “what do further donations accomplish?” can have complex answers: for example, the first million dollars to a charity can be quite different in their expected impact from the second million. We lay out what we know about each charity’s funding gap below.

For each charity, we have set out a “minimum target” that we very much hope to see it raise. The targets have no precision to them, but roughly speaking, we see more value in donations up to the minimum target for each charity than in donations beyond the minimum target for any of them. (In other words, if two of the charities had hit their minimum target and the third had not, we would prefer additional donations to go to the third charity until it hit its minimum target.) We believe that each charity will hit its minimum target. We would also be happy to see each charity raise funds beyond its minimum target, and expect that this will happen to some degree.

The minimum targets consider both the nature of the funding gap and the goal of providing positive incentives via rewarding charities that exemplify the values GiveWell is seeking to promote, such as transparency, strong monitoring and evaluation, and full participation in GiveWell’s evaluation process. We believe that all three top charities are outstanding in these respects and thus hope to move substantial funds to each, though we also feel there are some differences on this front that contribute to our differing minimum targets.

When it comes to donations beyond the minimum target, we haven’t reached a clear conclusion on which charities should be prioritized, and feel that donors will decide – reasonably accurately – which charities are the best fit for their own worldview and values with the guidance we provide. Accordingly, we have decided not to provide numerical rankings this year.

Please note that there will be some additional content coming out within the next 1-2 weeks:

  • Good Ventures is planning to announce its own plans re: grants to top charities. We expect this announcement to come this week and expect that it may influence donors’ choices.
  • We are planning to write about the evolution of our criteria for recommending charities, which are being modified from what they were in previous years. This content will not be directly relevant to the choice between charities (this post attempts to summarize all relevant considerations).
  • We are also planning to have individual staff members (who feel comfortable doing so) announce their plans and their reasoning for their own personal giving. In general, the most popular choice internally is GiveDirectly.

The rest of this post lays out what we see as the key considerations regarding the funding gap, strengths and weaknesses of each recommended charity.

GiveDirectly
Funding gap

We provide the details of GiveDirectly’s funding gap in our review. In brief, GiveDirectly has told us that, to facilitate its planning, it aims to raise $40 million, the amount it budgets for 2014 and 2015. It hopes to spend $14 million in 2014 and $26 million in 2015.

As of the writing of this blog post, GiveDirectly has allocated or committed over $5 million to cash transfer campaigns (the vast majority of these funds in the past year), utilizing most of the funds it has received to date from donors.

Our understanding is that the majority of additional donations will go directly to cash transfers, which themselves are part of GiveDirectly’s (a) experimentation with its operational model as it expands and (b) experiments into the impacts of different models of cash transfer programs.

We hope that GiveDirectly is able to close a sizable portion of its funding gap, and have set a “minimum target” of $2.5 million, about half of what it has spent in the last year. We would be relatively comfortable with its receiving up to $20 million, though as it received more funds we would begin to prefer that those funds be allocated elsewhere.

Details on GiveDirectly’s room for more funding

Strengths and weaknesses

  • Program impact and cost-effectiveness. Cash transfers are relatively strongly evidence-backed, and are in the range of our other priority programs’ cost-effectiveness, but our best guess is that cash transfers are 2-3x less cost-effective than the deworming programs carried out by SCI. Details at our updated cost-effectiveness spreadsheet. Our estimates are subject to substantial uncertainty.
  • Monitoring and evaluation. All the programs implemented by our top charities are evidence-backed, but GiveDirectly is the only one that has a randomized controlled trial (finding strong results) of its own implementation of its program. Thus, GiveDirectly has been evaluated in a more rigorous way than our other top charities. We also feel that we have seen stronger evidence of successful execution of its program than for our other top charities.
  • Directness and robustness of the case for impact. We believe there is a highly direct and robust connection between GiveDirectly’s receiving funds and those funds reaching very poor individuals. We believe GiveDirectly to be the strongest of the recommended charities on this front.
  • Transparency and participation in our process. GiveDirectly has been exceptionally open with us, enabling us to understand the full details of its work.
  • Scale. Were GiveDirectly to receive the full level of support it is seeking, this would result in a significant increase in scale. We view this both as a risk and an opportunity. As of the end of 2013, GiveDirectly was distributing cash at a rate of approximately $3.6 million per year; it hopes to reach close to $15 million by the end of 2014. GiveDirectly has scaled up successfully in the past; it had been distributing cash at a rate of $360,000 per year at the end of 2012. For donors who believe GiveDirectly can effectively scale up, this represents an opportunity to provide early-stage funding to an organization that may eventually grow to be much larger. For donors who are skeptical of GiveDirectly’s ability to scale, providing such funds may appear unreasonably risky (though only after the first $5-10 million in support; the first $5-10 million would be consistent with the size GiveDirectly has been to date). Full GiveDirectly review

Schistosomiasis Control Initiative (SCI)
Funding gap

SCI has told us that it hopes to raise at least $4 million for work it plans to undertake in 2014 and believes it could effectively absorb up to $10 million (though some of this spending would spread into 2015). This work would mostly consist of direct implementation of deworming.

Since November 2011, when we first recommended SCI, SCI has received approximately $4.6 million in unrestricted funds. It has spent $2.9 million, committed an additional $1.2 million and approximately $1 million remains unallocated, including approximately $500,000 it had in reserve prior to our recommendation (more).

We have set a “minimum target” of $1 million for SCI, which is consistent with the levels of GiveWell-attributable funding it has received in the past. We would be relatively comfortable with its receiving up to $10 million, though as it received more funds we would begin to prefer that those funds be allocated elsewhere.

Details of SCI’s room for more funding

Strengths and weaknesses

  • Program impact and cost-effectiveness. Our best guess is that SCI’s deworming program is more cost-effective than cash transfers by a factor of 2-3x and 1-2x as cost-effective as DtWI’s deworming program. Details at our updated cost-effectiveness spreadsheet. Our estimates are subject to substantial uncertainty.
  • Monitoring and evaluation. We recently became aware of major limitations to the studies we had previously been relying on as evidence of SCI’s impact. We will be reviewing further evidence that SCI sent us in response; we had not previously focused on collecting such evidence because of our impression that the studies we were using were sufficient. The program SCI works on (deworming) is relatively strongly evidence-backed.
  • Directness and robustness of the case for impact. We believe that the connection between SCI’s receiving funds and those funds’ reaching very poor individuals is more direct than DtWI’s but less direct than GiveDirectly’s. SCI has spent (and plans to spend) a sizable amount of its budget directly carrying out deworming programs, but it also spends funds on a variety of activities related to deworming (e.g., advocacy, operational research).
  • Transparency and participation in our process. From 2009 (when we first made contact with SCI) through mid-2013, we struggled to communicate well with SCI. Since August 2013, our communication with has SCI has improved substantially. We still do not feel that we have the same level of visibility into SCI’s operations as we do into those of our other top charities, and this is a major factor in the relatively lower “minimum target” we have set for SCI as opposed to other top charities, though we are optimistic that this will change in the next year.
  • Scale. Were SCI to receive $5 million, it would represent an increase in the amount of funding it processes. SCI has spent (or committed) approximately $2 million in unrestricted funds in each of the past two years and is currently primarily funded by a 5-year, ~$15 million grant from DFID. It is currently under consideration for a 4-year, $15-18 million grant from DFID. In sum, SCI has been operating at a level of approximately $6 million/year and may soon be operating at a level of approximately $10 million/year (due to the DFID grant). An additional $5-10 million would increase its scale further.

Full SCI review

Deworm the World Initiative, led by Evidence Action (DtWI)
Funding gap

DtWI has told us that it would like to receive $2-3 million to give it the flexibility to pursue opportunities that are not directly funded by other donors. It has also told us that Evidence Action, its parent organization, has a funding gap over the next 2 years of $5-7 million.

Because our analysis has focused on DtWI, we have focused on its funding gap.

We would like to see DtWI close much or all of its gap so that it can have an adequate supply of unrestricted funding, and we have set a minimum target of $2 million. We see significantly less value in funds beyond that point, and particularly past the $3 million mark.

Details of DtWI’s room for more funding

Strengths and weaknesses

  • Program impact and cost-effectiveness. In isolation, our best guess is that DtWI’s deworming program is likely less cost effective than SCI’s (about 50-100% as cost effective) and GiveWell staff differ about its cost-effectiveness relative to cash transfers (1.5-3x as cost effective). This is because worm infections appear less prevalent and (particularly) less intense where DtWI works than where SCI works. However, DtWI focuses on influencing government spending rather than covering all costs with its own funds, and we think it is therefore appropriate to give it some degree of credit for “leverage” – making your dollars go further than they otherwise would. Estimating the degree of “leverage” is highly subjective; our best guess is that it should add a multiplier of about 3 to the cost-effectiveness, making DtWI more than competitive with other top charities on this front. Details at our updated cost-effectiveness spreadsheet.
  • Monitoring and evaluation. We believe DtWI’s self-evaluation to have been less rigorous than that of GiveDirectly, though we are hopeful that it will improve going forward. (As noted above, we are currently revisiting the quality of SCI’s self-evaluation.)
  • Directness and robustness of the case for impact. DtWI is a technical assistance/advocacy organization, and therefore the chain from donation to person helped is the least direct of any of our top charities. This is the flip side of DtWI’s potential “leverage” – because it focuses on influencing and assisting other players rather than on direct implementation, its value-added per dollar spent is quite plausibly higher but also harder to pin down.
  • Quality of communication. From 2009 (when we first made contact) through early 2013, we struggled to communicate well with DtWI. Since early 2013 (which coincided with DtWI changing its leadership), our ability to communicate with DtWI improved substantially. We now feel that we have as much visibility into DtWI’s operations as we do any of our other top charities.
  • Scale. Our understanding is that $2-3 million in unrestricted funds will provide DtWI with flexibility that will be helpful as it grows. Because DtWI requests $2-3 million, we would expect that any funds it receives above $2-3 million would likely effectively support Evidence Action more broadly (e.g., its Dispensers for Safe Water program, which we have not examined although we have published a report on water quality interventions, and potentially other initiatives that it might launch).
  • Fungibility. DtWI is an initiative led by its parent organization, Evidence Action. There is some possibility that funds given due to this recommendation effectively support Evidence Action and its Dispensers for Safe Water program (or future initiatives) rather than DtWI. If donors are particularly concerned about this, they should carefully consider their support of DtWI.

Full DtWI review

A note on Against Malaria Foundation (AMF)
As discussed in this blog post, we are not recommending AMF to donors this year due to lack of short-term room for more funding. AMF currently has access to approximately $12 million that it has yet to commit to funding distributions.

We believe that it is not unlikely that AMF will reach an agreement to use the bulk of its funds in the next 6 months, and therefore would not be surprised if donors choose to continue supporting it.

We will continue to follow AMF’s progress and reconsider its recommendation status if/when it finalizes a distribution.

Full AMF review

Rethinking SCI’s evidence of impact

(updated on: ) | by Holden

We’ve recently encountered some new (to us) information about studies we have relied on in our evaluation of the Schistosomiasis Control Initiative (SCI), which is one of our top charities (and will remain one after our upcoming refresh). We won’t be able to fully investigate all the implications of this information by the end of giving season, but at this point we provisionally hold that:

  • We are less confident in SCI’s track record of reducing worm infection prevalence than previously. In assessing SCI’s track record, we have relied on a set of studies showing dramatic drops in worm infection prevalence and intensity coinciding with SCI-run deworming programs; we now believe that most of these studies do not show what we had thought they showed. Specifically, it appears that most of the studies we’ve relied on involved treating children for worms as part of the study (i.e., children who tested positive for infection were treated by research personnel), so improvements measured over time could be attributable to the activities of the people carrying out the study rather than necessarily to the coverage of the national control program. We do not believe this represents an error or purposeful deception – the lead author on the study straightforwardly confirmed this issue to us and defended it based on ethical considerations. However, we believe that the published version of the study was (perhaps unintentionally) not clear on this point.
  • We plan to review other evidence of SCI programs’ effectiveness; we have not focused on collecting such evidence because of our impression that the studies we were using were sufficient. It is now a high priority for us to solicit and review such evidence. On November 28 (in response to a draft of this post sent Nov. 11), SCI sent us reports for Burundi and Niger that, on preliminary review, look strong and encouraging to us, though we have not been cleared to share the reports publicly and have not yet done a thorough examination. We plan to review these further and to solicit other monitoring and evaluation data.
  • We plan to continue recommending SCI through this year’s giving season. We feel that SCI represents an outstanding giving opportunity, one of the best we are aware of for donors seeking to support evidence-backed programs serving the global poor. More

New information about studies of national deworming programs
In our 2012 review of SCI, we emphasized a set of panel studies in which children at multiple sites were tested for infection (schistosomiasis and in some cases other conditions) before the start of an SCI-led control program, then followed up and tested again one and two years later. The studies show a fairly consistent picture of dramatic drops in disease prevalence. We acknowledged multiple limitations to these studies, but felt that they gave relatively strong suggestive evidence that the SCI-led control programs had had their intended effects.

We recently discovered – via an investigation that started when Sally Murray and Rob Wiblin of Giving What We Can alerted us to the paper, Does mass drug administration for the integrated treatment of neglected tropical diseases really work?: assessing evidence for the control of schistosomiasis and soil-transmitted helminths in Uganda – that in one of these studies (Kabatereine et al. 2007), when children tested positive for infection in the course of the study, they were then directly treated. We confirmed this point over email with Dr. Narcis Kabatereine, the lead author of the study. In the discussion that followed, SCI stated:

The data in Uganda in this study was collected as part of the initial studies supported by the Bill and Melinda Gates Foundation to evaluate the feasibility and impact of mass drug administration for SCH. Due to the nature of the study, which involved drawing of blood and ultrasonography, ethical considerations demanded that treatment had to be offered by investigators. Findings from these and additional studies confirmed the link between SCH infections and morbidity. Data from these studies also provided very valuable information on other aspects of SCH control. Ongoing monitoring and evaluation protocols in Uganda and all other SCI assisted countries were able to use a different methodology and did not require the treatment of all positive cases. (Children in sentinel sites are tested approx. 1 month prior to MDA so that any positive cases will be treated as part of the MDA. This ensures that the sentinel sites accurately reflects the impact of the programme as a whole but also does not leave known positive children untreated, which would be unethical.) Therefore from 2006 onwards studies assessing the impact of SCH mass treatment did not include the treatment of identified positives and therefore were not subject to the theoretical bias that may have been inherent in the original protocols.

On November 28 (in response to a draft of this post sent Nov. 11), SCI sent us reports for Burundi and Niger following the post-2006 methodology that, on preliminary review, look strong and encouraging to us, though we have not been cleared to share the reports publicly and have not yet done a thorough examination. We plan to review these further and to solicit other monitoring and evaluation data.

We feel that a study using the original methodology (not the post-2006 methodology) – treating children who test positive for infection – could easily find the observed declines in schistosomiasis, even if there were no national control program running at all, because such declines could be caused by the practice of treating children for infection as part of the study (and then following up with the same children in later years). Therefore, we believe that the results of such a study cannot be taken as a reflection of the quality or success of a national control program such as those led by SCI.

We were previously unaware that Kabatereine et al. 2007 had involved treating children as part of the study. Dr. Kabatereine pointed out to us in an email that the study did intend to communicate this, pointing to the quote that opens the “Findings” section: “We enrolled 4351 children from 37 schools, of which 2815 (64.7%) were traced and treated at one year follow-up and 1871 (43.0%) at two year follow- up.” Dr. Kabatereine also stated that phrases such as “Anthelmintic treatment delivered as part of a national helminth control programme” imply “Anthelmintic treatment delivered as it is done in a national helminth control programme” (e.g., same drugs, same dosage and frequency).

We now see that the phrase “traced and treated” in the above quote should have raised a flag for us. However, this phrase could have referred to treatment by the national control program (“and treated” seems to refer only to followup rounds, not to the first round of measurement), and it appears in what we see as an unexpected place in the paper: in the opening of the “Findings” section, where the emphasis appears to be on how many children were examined and followed up in each round and what the results were. As far as we can tell, this issue is not noted in any of the sections on methodology or ethics. In addition, the introduction and conclusion of the paper do not mention this issue and strongly seem to us to imply that the intent of the paper was to examine the impact of the national control program, as opposed to the impact of treatment delivered along the same protocol as the national control program. We are inclined, therefore, to chalk the fact that we did not notice this issue up to a lack of clarity in the paper, rather than to a shortcoming on our part. We recognize that others may disagree. For reference, a link to the paper in question is available here.

The upshot is that we no longer see studies such as this one as demonstrating the effectiveness of a national control program as implemented.

We plan to review other evidence of SCI programs’ effectiveness; we have not focused on collecting such evidence because of our impression that the studies we were using were sufficient. It is now a high priority for us to solicit and review such evidence. On November 28 (in response to a draft of this post sent Nov. 11), SCI sent us reports for Burundi and Niger that, on preliminary review, look strong and encouraging to us, though we have not been cleared to share the reports publicly and have not yet done a thorough examination. We plan to review these further and to solicit other monitoring and evaluation data.

Continuing to recommend SCI
This represents a potentially substantial update regarding the evidence of SCI’s track record; we will need to reassess the case for its track record thoroughly. The fact that we weren’t aware of crucial information about how at least one study was conducted reflects concerns we have earlier noted (here and here) about communication difficulties with SCI.

However, we plan to continue recommending SCI through this giving season, because:

  • We would still guess, given all the information we have – including the new reports we have seen as well as our impression that school-based deworming programs are among the more straightforward interventions (the latter is based partly on site visits and partly on several years of ongoing conversations with deworming-focused charities) – that the programs led by SCI are relatively effective.
  • Our criteria for top charities are shifting to put more emphasis on potential leverage (which we feel we have made progress in understanding) and less emphasis on organization-specific monitoring and evaluation. We will be writing about this more in a future post. By our modified criteria, SCI remains outstanding.

We have substantial work to do in rethinking the evidence regarding SCI’s track record. With that said, we continue to feel that SCI represents an outstanding giving opportunity, one of the best we are aware of for donors seeking to support evidence-backed programs serving the global poor.

Change in Against Malaria Foundation recommendation status (room-for-more-funding-related)

(updated on: ) | by Holden

Note: AMF has reviewed a draft of this post. We believe AMF’s willingness to let individual donors follow along with it – both the successes and the struggles – is highly unusual in the charity world, and an extremely important and positive quality.

Since naming the Against Malaria Foundation (AMF) as our #1 charity in late 2011, we have tracked $10.6 million in donations made to it as a result of our recommendation. In that time, AMF has held the funds while attempting to negotiate a net distribution to spend them on. It has not yet finalized a distribution large enough to spend the bulk of these funds (though it has funded a smaller-scale (~$1 million) distribution in Malawi).

We have been following its negotiations, and this post discusses why AMF has been unable to finalize a sufficiently large distribution. At this time, we plan not to recommend more donations to AMF until and unless it commits the bulk of its current funds to net distributions. Note that this decision does not reflect a negative view of AMF, but rather reflects room for more funding related issues. We note that AMF does not share our view on a lack of room for more funding. AMF’s take is available here. 

In the course of investigating and discussing why AMF was unable to finalize a distribution, we have noted some of what we believe to be mistakes on AMF’s part (more below). We are unsure of how serious these mistakes are, and they are not the driver of the change in recommendation status. We stand by our earlier decision to recommend AMF as our #1 charity, and if and when AMF commits the bulk of its current funds, we will revisit its status and likely recommend it again.

AMF’s response: We disagree with significant parts of GiveWell’s analysis. It is straightforward to find people who want and need nets. We could distribute all the nets we can buy if we were prepared to tolerate high rates of theft. We feel a strong duty to our donors to wait until we can use the funds without significant net losses. This means working carefully with distribution partners who share this goal.

Until recently, it looked as though a finalized distribution might be imminent. Since AMF’s prospects for a distribution in Sierra Leone fell through in September 2013, both we and AMF have been intensifying our efforts to diagnose potential problems, and so much of the content in this post reflects information that is new to both us and AMF.

The rest of this post discusses the following:

  • Why has AMF been unable to finalize a sufficiently large distribution? AMF’s past distributions have been relatively small in scale, compared to the types of distributions it is looking at now (the need to increase scale is driven by the increase in funding that GiveWell’s top recommendation has brought about). To find a sufficiently large distribution requires negotiating with the national malaria control programs of countries in sub-Saharan Africa, which we perceive to have some discretion in which funders they work with, and which we perceive to be choosing funders based on a variety of factors including size and reporting requirements. It also requires coordinating with other major funders. AMF has substantial reporting requirements, yet in some cases is able to fund only a relatively small piece of a given country’s distribution. As such, there may be fundamental reason for other parties to the distribution to prefer not to work with AMF if they can obtain nets from another source. We believe that this is the most important and fundamental explanation. We also believe that other factors AMF controls, and which it should have handled differently, may have played a role (see two points down). More
  • Does this indicate a lack of global funding gap for insecticide-treated nets? We don’t believe that the struggles to date were a function of the lack of a global funding gap. Over the time period in question, our analysis – based on population-at-risk as compared to number of nets distributed – continued to show a sizable gap; several of AMF’s negotiations proceeded to advanced stages; and our understanding from the conversations we’ve had is that AMF’s negotiations proceeded as far as they did because of an acute desire for more nets. Going forward, we believe that the global funding gap may be smaller, which may steepen AMF’s challenges in terms of negotiating distributions. More
  • Has AMF made mistakes? What could it have done better? In 2012, both we and AMF agree that AMF did not investigate enough possibilities at once, and we believe that it improved on this front in 2013. This year, the picture is less clear, as AMF has seemed very close on more than one occasion to having a distribution finalized.
    • AMF states – and we would guess – that the most important factors have been AMF’s relatively small size (compared to other net funders) and its relatively strong reporting requirements (which we broadly support). Broadly speaking, we would not characterize either of these fundamental factors as mistakes, with a caveat (see next bullet point)
    • While some aspects of AMF’s strong reporting requirements are an important part of its value added, other aspects of its requirements may have been excessive. More details below.
    • We believe that AMF’s preference for funding nets (as opposed to other costs such as shipping and evaluation) may have played a role. We see some justification for this preference but also believe that we are – at least partly – in persistent disagreement with AMF on the matter.
    • Finally, we believe that AMF’s communication style has been problematic and that it would benefit from investing more time in building relationships in the malaria control community. The level of reporting sought by AMF makes it likely to meet resistance from organizations with different standards, creating a challenge that we think AMF should have handled better (and is taking steps to handle better in the future).

    More

  • How is AMF responding to the distribution challenges? We believe AMF is taking the most appropriate action under the circumstances, which is to solicit a greater degree of involvement from some of the highly experienced people on its Malaria Advisory Group (MAG). We believe that these people will bring an outside perspective, credibility, and experience with the malaria community, and we are optimistic that they will be able to diagnose and respond to AMF’s obstacles. In addition, at AMF’s CEO’s encouragement, some of AMF’s trustees – particularly Peter Sherratt and Richard Lane, who he feels have relevant experience of working with governments and in development – are planning to become more involved in investigating and negotiating potential distributions. This may further help to address problems related to communication style. More
  • What will happen to the money donated to AMF if AMF can’t finalize a distribution? AMF remains committed to spending its funds on one or more distributions, and if necessary will distribute nets in multiple smaller distributions over a longer period of time. We asked AMF what its plans would be if this approach too proved infeasible (which we consider to be unlikely but possible). AMF responded that the probability of this is too low to merit planning for this given the continuing prevalence of malaria and a significant global net gap. More
  • What is GiveWell’s current view of AMF? Why is GiveWell planning to withdraw its recommendation? We feel AMF has made mistakes and remain in disagreement on some points (more below), but we continue to feel overall that AMF is an outstanding organization in terms of aiming to bring a unique level of accountability to one of the most proven, cost-effective interventions that exists. We are aware that this emphasis on accountability inherently involves challenges when dealing with partners who do not share the same vision of appropriate data collection and reporting. We are still hopeful that AMF will negotiate one or more suitable distributions, and we believe that continuing to hold the funds it has is important to maintaining its negotiating position. More. However, given the significant amount of time that has passed, it now seems to us that there is a substantial risk that AMF will not accomplish this in the near future, and that having more funds to commit would steepen rather than lessen its challenges. (It’s possible that AMF’s challenges would be lessened if it were to receive a very large amount of money – in the range of $30-40 million, which would allow it to fund an entire country’s distribution on its own – but we believe that donations in the range of our projected money moved will increase the number of distributions AMF needs to find while still not making a big enough difference to its size to improve its negotiating position substantially.) Therefore, we believe it is most prudent to donate elsewhere until and unless AMF commits the bulk of the funds it currently holds, at which point we will revisit its ranking.
  • Did GiveWell err in its previous recommendation of AMF? What has GiveWell learned from AMF’s struggles? We feel that our prior recommendation of AMF was appropriate, and that the struggles we’ve seen have come mostly from previously recognized (though underestimated) risks. However, we feel that our investigations and communications could have been better, and we have updated our views on a number of fronts. Specifically, (a) we should think harder and communicate more clearly about cases in which our recommendation could bring about a change in the scale of a charity, and the potential difficulties associated with this; (b) we should put more effort into ensuring clear communication between the donors coming to our website and the charities receiving funds from them, in terms of donor preferences about how their funds are to be spent; (c) we should put more effort into interviewing people and organizations who have worked with the organizations we’re investigating, as part of our broader efforts to evaluate people and organizations holistically; (d) we need to be highly cognizant – and to effectively communicate – that as of today, there don’t appear to be any “sure bets” with quantifiable outcomes in charity, especially when it comes to directing relatively large amounts of money. More

This post is partly based on conversations we had over the last few weeks in order to investigate the key factors in AMF’s struggles to negotiate distributions. Some of these conversations remain confidential, but we have posted notes from Don de Savigny of AMF’s Malaria Advisory Group.

Why has AMF been unable to finalize a sufficiently large distribution?
Prior to becoming GiveWell’s top-rated charity, AMF funded a number of relatively small distributions carried out by nonprofit organizations. Its largest distribution was the one that was in progress when we first upgraded its ranking to #1: a 270,000-net distribution in Malawi with Concern Universal as a partner, which we wrote about in our March 2012 and September 2012 updates. Since September 2012, AMF has had a total of approximately $9 million in revenues, of which we believe approximately $7 million can be attributed to GiveWell’s recommendation, and the amount it now holds could pay for about 4 million nets – substantially more than it has distributed in the past. Accordingly, AMF has sought to partner directly with a government on a large-scale net distribution, and has not yet finalized such a distribution.

We have discussed the details of negotiations with AMF, but the extent to which this information can be made public is limited. AMF’s future distribution tracker shows the state of possible distributions as of June 2013, and we have discussed some additional detail in previous updates (September 2012, February 2013).

Generally, it seems to us that a given government is, at any given point in time, deciding between multiple options for providing nets for a distribution (including forgoing enough support to do a complete distribution, and instead leaving some areas uncovered). When deciding whether to work with a funder, the following factors all may come into play:

  • The size of the funder. President’s Malaria Initiative (PMI) and The Global Fund to Fight AIDS, Tuberculosis and Malaria (GFATM) are particularly large funders that can potentially fund the bulk, or entirety, of a country’s net needs, and so working with them is likely to be particularly worthwhile. While AMF’s level of funding is too large to continue focusing exclusively on distributions of the size it has funded three times in Malawi (several hundred thousand nets per distribution), it is only large enough to fund the bulk of the nets a country needs when dealing with countries with gaps of ~4 million nets or fewer.
  • The funder’s requirements in terms of data collection and reporting. A major part of AMF’s appeal, according to our criteria, is its unusual interest in publicly sharing as much data as possible, particularly regarding successful delivery and usage of nets and any changes in malaria case rates pre- and post-distribution. However, AMF’s requests for such data may lessen its appeal for major potential partners for a variety of reasons – in particular, (a) costs and administrative hassles involved with collecting and releasing the data; (b) potentially limited capacity for data collection; (c) potential embarrassment from making the data public (in case the data does not give a favorable picture of how the distribution went).
  • The funder’s other requirements and requests about how the distribution is to be executed. In particular, AMF has offered to pay for nets but has generally not offered up front to pay for non-net costs – including shipping, distribution and data collection – which is something we discuss more below.
  • Overall degree of comfort and familiarity with the funder. AMF is new to partnerships of this nature, which may reduce the degree of comfort. In addition, AMF may have caused some interpersonal tension, discussed more below.

We aren’t able to disentangle these factors with confidence and say which have been the major ones in AMF’s difficulty finalizing a distribution.

AMF believes that the primary factor is its concern over theft and its reluctance to provide nets without strong measures in place to ensure that they will arrive safely and be distributed to the right people. It believes its insistence on transparency and detailed accountability has caused it to encounter significant resistance from some others in the aid community who have less focus on this area. We are not as confident that we can identify the primary factor, though we do believe that AMF’s emphasis on transparency and accountability (which we broadly support) creates inherent challenges.

Does this indicate a lack of global funding gap for insecticide-treated nets?
In the past, we have analyzed population-at-risk data and distribution data to estimate the global net gap, concluding that it is quite large. We are currently in the process of finalizing our refresh of this analysis, and we believe that we will again project a gap, although the fact that nets have gotten cheaper over time – as well as increased funding from other sources – means the estimated gap is likely to be substantially smaller than previous estimates.

We believe that the total number of nets available over the past couple of years has been insufficient to cover the full population at risk. In addition, the partial progress AMF has made in arranging distributions (see notes from Don de Savigny for more) does not, to us, imply a world in which nets are easily available. AMF has encountered multiple countries in which the available funding did not seem sufficient to cover the full planned distribution, and in which AMF was requested to fill the gap.

Going forward, we believe that the global funding gap may be smaller (more), which may steepen AMF’s challenges in terms of negotiating distributions, though it appears that some degree of funding gap will remain.

Has AMF made mistakes? What could it have done better?
As stated above, it’s difficult to disentangle the factors that have led to large-scale distributions’ not going forward so far. In some cases we feel that these factors are unavoidable (AMF’s size) and/or reflect AMF’s strengths from our perspective (its interest in collecting and publishing data on distributions). We also wish to note that AMF has completed many smaller-scale distributions (up to several hundred thousand nets) in the past, and we feel that its struggles can fairly be attributed to the increase in size that has come with our recommendation. With that said, there are some cases in which we feel that AMF has made mistakes, and/or in which we continue to disagree with AMF. Much of the thinking here has been driven by recent conversations with people who have worked with AMF; some of these conversations remain off the record, but we have posted notes from Don de Savigny of AMF’s Malaria Advisory Group.

Excessive requirements

Broadly, we support AMF in its attempts to secure strong and public reporting on its distributions, even though doing so may make it more difficult for such distributions to be negotiated.

With that said, it appears to us that AMF’s requirements should have been rethought and adjusted more than they were, at the point when AMF made the transition from partnering with nonprofits to partnering directly with governments. Don de Savigny of AMF’s Malaria Advisory Group was among the people who noted this concern. We believe that AMF now plans to adjust its approach.

A couple of specific notes:

  • While we have previously been enthusiastic about AMF’s requests for malaria case rate data, we now believe that the costs of requesting this data may outweigh the benefits. Our view has changed because (a) we have come to believe that this data is highly unreliable (more at our conversation notes from August 15th); (b) we have been told that some countries simply do not have the capacity to collect this data, at least without funding to support their health-system infrastructure more broadly. AMF does believe that if possible an attempt should be made to track the impact of nets in this way, but also recognizes the case for flexibility on this point.
  • We still believe that other data AMF seeks (regarding delivery, condition and usage of nets) is valuable and important, and its raising questions around such data may already have had some impact on discussions in the malaria control community (more at our conversation notes from Don de Savigny of AMF’s Malaria Advisory Group).

Preference for funding only nets

AMF prominently states that donations from the public are used exclusively for buying nets. It believes that it must make good on this promise.

Last year, we became concerned that AMF’s preference for funding only net purchases (and not other costs associated with distributions, such as shipping, monitoring and evaluation) might be an obstacle to distributions’ moving forward. We spoke with Rob Mather about the matter, and got the impression in our conversation that (a) this had not been a major obstacle to moving forward; (b) if it became such an obstacle, AMF would raise money specifically for this purpose. Good Ventures’s 2013 donation was specifically marked as available for non-net costs; Rob was reluctant to mark other donations in this way and stated that he did not see a need, and we didn’t perceive it as a major issue, so we didn’t pursue the matter further. We also note that AMF has paid for non-net costs in its Malawi distributions, so it is clear to us that its reluctance to fund such costs is not absolute.

This year we have encountered one case of AMF’s offering to fund additional nets, with another funder’s then arranging to reallocate their funding from nets to non-net costs, specifically net use monitoring costs. (AMF did not feel able to offer funding for non-net costs when it was not funding nets.) This has highlighted to us that we previously underestimated the degree of AMF’s reluctance to fund substantial non-net costs, and with it, the potential consequences.

AMF has stated strongly to us that it does not believe its not offering to fund non-net costs is a major factor in agreements’ not being finalized. We remain unsure of whether AMF’s reluctance to fund non-net costs is a major factor (or any factor) in its struggles to finalize a distribution, but it seems to us that this is a strong possibility.

Having spoken further with AMF, we’ve been given two arguments in favor of avoiding paying for non-net costs:

  1. AMF has no in-country presence to effectively monitor non-net costs. It is easier to track and audit the use of funds (in a way that e.g. avoids bribery) in the case of net purchases than in the case of other costs (particularly costs of distribution and evaluation). AMF has the technical capacity for the former, but not for the latter, for which it believes in-country presence or a highly accountable in-country partner is vital. AMF seeks to ensure that major (i.e., government) funders with substantial in-country staff take responsibility for tracking and auditing non-net expenses, and believes that the most practical way to do this is to ensure that major funders allocate their funds to non-net costs. This issue may also have legal implications: AMF has received professional audit guidance that paying for substantial non-net costs could incur legal requirements to audit the funds that would go beyond AMF’s technical capacity. We are unsure of how significant an issue legal requirements represent, and plan to look into the matter further; overall, we believe this could potentially be a strong argument for behaving as AMF has, but remain uncertain. Note that this argument pertains to tracking expenditures and ensuring appropriate use of funds, which is different from evaluating e.g. usage and impact of nets (an area in which we feel that AMF has higher standards than government funders).
  2. Many of AMF’s donors prefer that their funds be restricted in this way, and because of the commitments given on AMF’s website, AMF has an obligation to assume by default that this is what its donors prefer. (AMF also notes that a) it believes it has paid for non-net costs when not doing so would hold up a distribution and the other pieces are in place; b) it has approached larger donors, and is prepared to approach other large donors, to ask whether specific donations could be allocated to non-net costs if such a need arose.) In response, we have communicated strongly to AMF that we do not believe the bulk of GiveWell-attributable donations should be restricted in this way, and we have offered to survey donors we have contact information for if it would help resolve the matter.

At this point, we believe there may be good justification for AMF’s reluctance to fund non-net costs, but we also believe that at least part of the reluctance comes from AMF’s perception of donors’ preferences – and thus from a failure on our part to ensure that donors coming from our website communicate their preferences to AMF. We believe that this is at least partially a matter of persistent disagreement: AMF sees value (when communicating to the broader public) in the simplicity and tangibility of “100% buys nets,” while we believe this is not valuable where GiveWell-directed donors are concerned.

Problematic communication style

AMF has pushed for what we see as new levels of information and public accountability, and this creates natural challenges and calls for building relationships in the development community. To this point, we believe that AMF has handled these challenges suboptimally, in the following respects:

  • AMF has done relatively little in-person interaction with potential partners, as opposed to communication by email and phone.
  • AMF has had a very direct communication style, and may have failed to pick up on (and encourage) communications from people whose style may be less direct. We feel that this dynamic may have led to AMF’s failing to detect potential problems as quickly as it could have – in particular, we were told that AMF should not have been as surprised as it was by the Sierra Leone distribution’s falling through. The sources from which we’ve drawn this view unfortunately remain confidential.

We recognize that the same quality that has made AMF particularly easy for us to communicate with (a highly direct communication style) may be a drawback in its relations with some potential partners.

We believe that AMF is taking appropriate steps (more below) to improve on this front.

The upshot of the above three factors is that AMF may have been perceived as overly demanding by potential partners – coming in as an outsider, asserting substantial requirements without offering to directly reimburse the costs of its requests, and appearing less open to negotiation than it was in reality.

With that said, we doubt that the above factors have been the primary explanation for AMF’s struggles to finalize a distribution. We see its small size and emphasis on strong and transparent data sharing as fundamental obstacles, and we’re cognizant that it’s been willing to negotiate on other factors. We believe that the above issues have largely been a function of transitioning from one sort of partner (nonprofits on smaller-scale distributions) to another (government partners on large-scale distributions), and as discussed below, we believe that AMF broadly is taking appropriate steps to improve on these matters.

How is AMF responding to its difficulties?
We believe that Rob Mather, AMF’s Founder and CEO, is at least as eager to see a distribution finalized as we are, and is highly concerned about the delays.

When he was unable to finalize a distribution last year, he committed to building a bigger pipeline of potential partners (details), and we believe he followed through well on that goal, as he explored a significantly larger number of options in 2013.

This year, the main step Rob is taking in reaction to the delays is to solicit greater involvement from his Malaria Advisory Group, specifically Don de Savigny, Bob Snow, and Abdisalan Mohamed Noor. The Malaria Advisory Group normally provides input on AMF’s distributions (examples), but is not employed or paid by AMF, and the individuals in question are well-credentialed malaria scholars (see bios for Don de Savigny, Bob Snow and Abdisalan Mohammed Noor).

We believe this is the appropriate action to take. These individuals are likely to command respect both from Rob and from the malaria community at large, and they will hopefully consider both AMF’s interest in strong reporting and arguments from the malaria community about what level of reporting is excessive, without having strong incentives to put more weight on one or the other. Therefore, their findings are likely to have the best combination of neutrality and respect from both parties that seems feasible in this situation.

Rob has also encouraged several of AMF’s trustees – particularly Peter Sherratt and Richard Lane, who he feels have relevant experience of working with governments and in development – to become more involved in investigating and negotiating potential distributions.

Finally, we have provided written feedback to AMF and its advisors and trustees, and will be sending them this post as well.

Ultimately, we expect AMF to reach reasonable resolutions on its reporting requirements and on the issue of how best to communicate them. We are less sure that we will end up in sync with AMF on the question of non-net costs, but we believe there are legitimate arguments for its perspective on this matter and will respect whatever it ultimately decides on this front, while making clear that AMF should not consider donor preferences to be a major argument for restricting its funds (at least concerning donors going on GiveWell’s recommendation).

What will happen to the money donated to AMF if AMF can’t finalize a distribution?
We have encouraged unrestricted donations to AMF, so in most cases we feel it is ultimately up to AMF how to use the funds. (AMF believes it is bound by its public commitments to use the funds to buy nets unless specifically agreed otherwise.) Our impression is that AMF will:

  • Commit the funds to a country-wide distribution that meets high standards of data collection and sharing.
  • Distribute nets in multiple smaller distributions over a longer period of time.

In either case, we feel that the substantial delay in committing funds represents a substantial negative development, though we also feel that dollars spent well with a delay (as we believe they will be in either case) are better than dollars spent poorly.

We asked AMF what its plans would be if this approach too proved infeasible (which we consider to be unlikely but possible). AMF responded that the probability of this is too low to merit planning for this given the continuing prevalence of malaria and a significant global net gap.

What is GiveWell’s current view of AMF? Why is GiveWell planning to withdraw its recommendation?
We continue to feel overall that AMF is an outstanding organization in that it is aiming to bring a unique level of accountability to one of the most proven, cost-effective interventions that exists.

However, a sizable risk has emerged that AMF will not be able to find enough distributions to commit the funds it has, and increasing that amount would steepen this challenge (though perhaps not if AMF raised such significant new funding that it became comparable in size to other major funders in relevant countries). Therefore, we are planning to recommend that people looking to give now donate elsewhere, until and unless AMF demonstrates an ability to put together large enough distributions to commit the funding it has now. We note that AMF does not share our view on a lack of room for more funding. AMF’s take is available here. 

We feel that AMF has made some mistakes, noted above. We are somewhat concerned that AMF is perceived as an “outsider” and – through some combination of communication style and perceived reluctance to fund non-net costs – has been making a poor impression on potential distribution partners. We believe that AMF is taking appropriate steps to diagnose and address these issues, despite some potential disagreements, particularly on the question of non-net costs. We remain unsure of the relative importance of AMF’s mistakes and the fundamental challenge of asking for unusual levels of transparency and accountability (and note that AMF feels the latter is the primary factor).

We wouldn’t guarantee that AMF would immediately become recommended again upon committing the bulk of its funds. The findings that come out of its Malaria Advisory Group’s increased involvement, and the exact circumstances under which a distribution is finalized, would affect our view. However, if and when AMF does commit the bulk of its funds, we will reconsider its ranking and will likely recommend it again.

Did GiveWell err in its previous recommendation of AMF? What has GiveWell learned from AMF’s struggles?
We’ve been reflecting on this episode and trying to draw lessons from it. While we still have some reflecting to do, at this point some lessons we believe we can take away are:

  • We should think harder and communicate more clearly about cases in which our recommendation could bring about a change in the scale of a charity. We were always aware that our recommendation would substantially increase the funds available to AMF, but initially we didn’t recognize that this would fundamentally change the character of the distributions it was partnering on (see our 2011 review). By our 2012 review, it had become clear that ability to finalize a distribution was a major risk factor, but on balance we decided it was a relatively minor one. We think it’s a good thing to substantially increase the funding of outstanding charities, challenging them to reach a new level of scale. Such an endeavor has potential “upside” of turning a limited organization into one with more absorptive capacity, as long as we don’t multiply the funds available by too high a number in too short a time. In the case of AMF, we feel that the increase in funds was appropriate – enough to bring it to a new level of scale, but not much more than that. However, it needs to be strongly noted, both internally and externally, that such a “step up” involves major risk. We don’t feel that we were clear on the nature of the risk here internally or that we communicated enough about it externally. In the future, when we are recommending that a charity be allowed to substantially increase its scale, we will make more effort to note the major risks that this will entail – and this will apply to this year’s top charities as well.
  • We need to ensure clear communication between the donors coming to our website and the charities receiving funds from them, in terms of donor preferences. This is the second time that a recommended charity has expressed to us that it assumes – by default – that its donors wish their funds to be restricted in certain ways. (The other case involves GiveDirectly; in that case we had clear communication up front and ensured that our donate page clearly distinguishes between “flexible funds” and “typical funds,” with the latter being more restricted). We strongly prefer to avoid restrictions on donations, and we don’t want our recommended charities to feel bound by donor preference for anything other than optimal positive impact.
  • We should look for ways to deepen and improve our evaluations of people. We’ve previously noted that we place some weight on our evaluations of the people running an organization, and we’ve also expressed reservations about doing so (“we’re wary of putting too much weight on informal, intangible impressions that are likely to end up being ‘tests of how similar charity representatives are to us’ rather than ‘tests of how well-suited charity representatives are to do their work.’ It seems easy for evaluations of people to end up being driven by charisma, by similarity to the evaluator, and/or by characteristics that don’t translate well to nonprofit work.”) We still believe that Rob is an outstanding individual, and we don’t want to place too much weight on complaints about his communication style when these could easily be motivated by frustration with other aspects of AMF, particularly its data disclosure requests. We also believe that the concerns on this front are being addressed appropriately. That said, the complaints have highlighted the point (which we previously noted) that our evaluations of people may overweight their similarity to us, particularly in communication style, and that our evaluations can miss substantial ways in which qualities that make a charity good at communicating with us can be unhelpful (or even drawbacks) for its ability to do its work. We want to find other methods of evaluation that can enrich our understanding of this aspect of an organization, such as the idea in the next point.
  • We should put more effort into interviewing other organizations and individuals who have worked with our recommended (or potential recommended) charities. We feel that some important considerations have been raised by our discussions with others in the malaria control community about their interactions with AMF. In general, an organization’s reputation among those it works with is important. This may also be a consideration in favor of highly credentialed organization leaders (who are less likely to come across as outsiders); this is something that we haven’t previously placed much weight on.
  • In general, we need to be highly cognizant – and to effectively communicate – that as of today, there don’t appear to be any “sure bets” with quantifiable outcomes in charity, especially when it comes to directing relatively large amounts of money. Many of those who write about GiveWell or use it to donate seem to be under the impression that our top charities represent “sure thing” options allowing a donor to save or improve a known number of lives per dollar given. Yet every year brings new observations about the complexity of giving and about the caveats to seemingly straightforward value propositions. We’ve written before (in 2011) that there is no such thing as a “sure thing” in giving, but we think this point merits further discussion, and we plan to discuss it more in a future post.

With all of that said, we don’t regret our prior recommendation of AMF and believe it was appropriate. AMF had a substantial track record of conducting multiple distributions, including one large distribution, with a level of public disclosure and tracking that no other net distribution charity (and few other charities of any kind) can match. Over 2 years, we have directed enough funding to it to multiply its funds available by several-fold, in the hopes that it would be able to operate at a larger scale; given our other options for recommended charities and the upside if this happened successfully, this was a risk worth taking. It still may be a risk that turns out well, as AMF continues to take steps to improve and to overcome its challenges; it was a risk that has led to substantial learning on our part, which has only been possible because of AMF’s strong transparency and direct communications; and even if AMF is not ultimately able to find appropriate distributions, the result will be delayed (not wasted) donations, as discussed above. We continue to strive to improve, but we stand by our earlier decision to recommend AMF as our #1 charity, and there is a strong possibility that we will do so again in the future.

Finally, we’d like to acknowledge AMF’s willingness to be open and transparent throughout this process. It has reviewed a draft of this post, and we believe its willingness to let individual donors follow along with it – both the successes and the struggles – is highly unusual in the charity world, and will prove enormously beneficial in the long run.