The GiveWell Blog

Change in Against Malaria Foundation recommendation status (room-for-more-funding-related)

Note: AMF has reviewed a draft of this post. We believe AMF’s willingness to let individual donors follow along with it – both the successes and the struggles – is highly unusual in the charity world, and an extremely important and positive quality.

Since naming the Against Malaria Foundation (AMF) as our #1 charity in late 2011, we have tracked $10.6 million in donations made to it as a result of our recommendation. In that time, AMF has held the funds while attempting to negotiate a net distribution to spend them on. It has not yet finalized a distribution large enough to spend the bulk of these funds (though it has funded a smaller-scale (~$1 million) distribution in Malawi).

We have been following its negotiations, and this post discusses why AMF has been unable to finalize a sufficiently large distribution. At this time, we plan not to recommend more donations to AMF until and unless it commits the bulk of its current funds to net distributions. Note that this decision does not reflect a negative view of AMF, but rather reflects room for more funding related issues. We note that AMF does not share our view on a lack of room for more funding. AMF’s take is available here. 

In the course of investigating and discussing why AMF was unable to finalize a distribution, we have noted some of what we believe to be mistakes on AMF’s part (more below). We are unsure of how serious these mistakes are, and they are not the driver of the change in recommendation status. We stand by our earlier decision to recommend AMF as our #1 charity, and if and when AMF commits the bulk of its current funds, we will revisit its status and likely recommend it again.

AMF’s response: We disagree with significant parts of GiveWell’s analysis. It is straightforward to find people who want and need nets. We could distribute all the nets we can buy if we were prepared to tolerate high rates of theft. We feel a strong duty to our donors to wait until we can use the funds without significant net losses. This means working carefully with distribution partners who share this goal.

Until recently, it looked as though a finalized distribution might be imminent. Since AMF’s prospects for a distribution in Sierra Leone fell through in September 2013, both we and AMF have been intensifying our efforts to diagnose potential problems, and so much of the content in this post reflects information that is new to both us and AMF.

The rest of this post discusses the following:

  • Why has AMF been unable to finalize a sufficiently large distribution? AMF’s past distributions have been relatively small in scale, compared to the types of distributions it is looking at now (the need to increase scale is driven by the increase in funding that GiveWell’s top recommendation has brought about). To find a sufficiently large distribution requires negotiating with the national malaria control programs of countries in sub-Saharan Africa, which we perceive to have some discretion in which funders they work with, and which we perceive to be choosing funders based on a variety of factors including size and reporting requirements. It also requires coordinating with other major funders. AMF has substantial reporting requirements, yet in some cases is able to fund only a relatively small piece of a given country’s distribution. As such, there may be fundamental reason for other parties to the distribution to prefer not to work with AMF if they can obtain nets from another source. We believe that this is the most important and fundamental explanation. We also believe that other factors AMF controls, and which it should have handled differently, may have played a role (see two points down). More
  • Does this indicate a lack of global funding gap for insecticide-treated nets? We don’t believe that the struggles to date were a function of the lack of a global funding gap. Over the time period in question, our analysis – based on population-at-risk as compared to number of nets distributed – continued to show a sizable gap; several of AMF’s negotiations proceeded to advanced stages; and our understanding from the conversations we’ve had is that AMF’s negotiations proceeded as far as they did because of an acute desire for more nets. Going forward, we believe that the global funding gap may be smaller, which may steepen AMF’s challenges in terms of negotiating distributions. More
  • Has AMF made mistakes? What could it have done better? In 2012, both we and AMF agree that AMF did not investigate enough possibilities at once, and we believe that it improved on this front in 2013. This year, the picture is less clear, as AMF has seemed very close on more than one occasion to having a distribution finalized.
    • AMF states – and we would guess – that the most important factors have been AMF’s relatively small size (compared to other net funders) and its relatively strong reporting requirements (which we broadly support). Broadly speaking, we would not characterize either of these fundamental factors as mistakes, with a caveat (see next bullet point)
    • While some aspects of AMF’s strong reporting requirements are an important part of its value added, other aspects of its requirements may have been excessive. More details below.
    • We believe that AMF’s preference for funding nets (as opposed to other costs such as shipping and evaluation) may have played a role. We see some justification for this preference but also believe that we are – at least partly – in persistent disagreement with AMF on the matter.
    • Finally, we believe that AMF’s communication style has been problematic and that it would benefit from investing more time in building relationships in the malaria control community. The level of reporting sought by AMF makes it likely to meet resistance from organizations with different standards, creating a challenge that we think AMF should have handled better (and is taking steps to handle better in the future).

    More

  • How is AMF responding to the distribution challenges? We believe AMF is taking the most appropriate action under the circumstances, which is to solicit a greater degree of involvement from some of the highly experienced people on its Malaria Advisory Group (MAG). We believe that these people will bring an outside perspective, credibility, and experience with the malaria community, and we are optimistic that they will be able to diagnose and respond to AMF’s obstacles. In addition, at AMF’s CEO’s encouragement, some of AMF’s trustees – particularly Peter Sherratt and Richard Lane, who he feels have relevant experience of working with governments and in development – are planning to become more involved in investigating and negotiating potential distributions. This may further help to address problems related to communication style. More
  • What will happen to the money donated to AMF if AMF can’t finalize a distribution? AMF remains committed to spending its funds on one or more distributions, and if necessary will distribute nets in multiple smaller distributions over a longer period of time. We asked AMF what its plans would be if this approach too proved infeasible (which we consider to be unlikely but possible). AMF responded that the probability of this is too low to merit planning for this given the continuing prevalence of malaria and a significant global net gap. More
  • What is GiveWell’s current view of AMF? Why is GiveWell planning to withdraw its recommendation? We feel AMF has made mistakes and remain in disagreement on some points (more below), but we continue to feel overall that AMF is an outstanding organization in terms of aiming to bring a unique level of accountability to one of the most proven, cost-effective interventions that exists. We are aware that this emphasis on accountability inherently involves challenges when dealing with partners who do not share the same vision of appropriate data collection and reporting. We are still hopeful that AMF will negotiate one or more suitable distributions, and we believe that continuing to hold the funds it has is important to maintaining its negotiating position. More. However, given the significant amount of time that has passed, it now seems to us that there is a substantial risk that AMF will not accomplish this in the near future, and that having more funds to commit would steepen rather than lessen its challenges. (It’s possible that AMF’s challenges would be lessened if it were to receive a very large amount of money – in the range of $30-40 million, which would allow it to fund an entire country’s distribution on its own – but we believe that donations in the range of our projected money moved will increase the number of distributions AMF needs to find while still not making a big enough difference to its size to improve its negotiating position substantially.) Therefore, we believe it is most prudent to donate elsewhere until and unless AMF commits the bulk of the funds it currently holds, at which point we will revisit its ranking.
  • Did GiveWell err in its previous recommendation of AMF? What has GiveWell learned from AMF’s struggles? We feel that our prior recommendation of AMF was appropriate, and that the struggles we’ve seen have come mostly from previously recognized (though underestimated) risks. However, we feel that our investigations and communications could have been better, and we have updated our views on a number of fronts. Specifically, (a) we should think harder and communicate more clearly about cases in which our recommendation could bring about a change in the scale of a charity, and the potential difficulties associated with this; (b) we should put more effort into ensuring clear communication between the donors coming to our website and the charities receiving funds from them, in terms of donor preferences about how their funds are to be spent; (c) we should put more effort into interviewing people and organizations who have worked with the organizations we’re investigating, as part of our broader efforts to evaluate people and organizations holistically; (d) we need to be highly cognizant – and to effectively communicate – that as of today, there don’t appear to be any “sure bets” with quantifiable outcomes in charity, especially when it comes to directing relatively large amounts of money. More

This post is partly based on conversations we had over the last few weeks in order to investigate the key factors in AMF’s struggles to negotiate distributions. Some of these conversations remain confidential, but we have posted notes from Don de Savigny of AMF’s Malaria Advisory Group.

Why has AMF been unable to finalize a sufficiently large distribution?
Prior to becoming GiveWell’s top-rated charity, AMF funded a number of relatively small distributions carried out by nonprofit organizations. Its largest distribution was the one that was in progress when we first upgraded its ranking to #1: a 270,000-net distribution in Malawi with Concern Universal as a partner, which we wrote about in our March 2012 and September 2012 updates. Since September 2012, AMF has had a total of approximately $9 million in revenues, of which we believe approximately $7 million can be attributed to GiveWell’s recommendation, and the amount it now holds could pay for about 4 million nets – substantially more than it has distributed in the past. Accordingly, AMF has sought to partner directly with a government on a large-scale net distribution, and has not yet finalized such a distribution.

We have discussed the details of negotiations with AMF, but the extent to which this information can be made public is limited. AMF’s future distribution tracker shows the state of possible distributions as of June 2013, and we have discussed some additional detail in previous updates (September 2012, February 2013).

Generally, it seems to us that a given government is, at any given point in time, deciding between multiple options for providing nets for a distribution (including forgoing enough support to do a complete distribution, and instead leaving some areas uncovered). When deciding whether to work with a funder, the following factors all may come into play:

  • The size of the funder. President’s Malaria Initiative (PMI) and The Global Fund to Fight AIDS, Tuberculosis and Malaria (GFATM) are particularly large funders that can potentially fund the bulk, or entirety, of a country’s net needs, and so working with them is likely to be particularly worthwhile. While AMF’s level of funding is too large to continue focusing exclusively on distributions of the size it has funded three times in Malawi (several hundred thousand nets per distribution), it is only large enough to fund the bulk of the nets a country needs when dealing with countries with gaps of ~4 million nets or fewer.
  • The funder’s requirements in terms of data collection and reporting. A major part of AMF’s appeal, according to our criteria, is its unusual interest in publicly sharing as much data as possible, particularly regarding successful delivery and usage of nets and any changes in malaria case rates pre- and post-distribution. However, AMF’s requests for such data may lessen its appeal for major potential partners for a variety of reasons – in particular, (a) costs and administrative hassles involved with collecting and releasing the data; (b) potentially limited capacity for data collection; (c) potential embarrassment from making the data public (in case the data does not give a favorable picture of how the distribution went).
  • The funder’s other requirements and requests about how the distribution is to be executed. In particular, AMF has offered to pay for nets but has generally not offered up front to pay for non-net costs – including shipping, distribution and data collection – which is something we discuss more below.
  • Overall degree of comfort and familiarity with the funder. AMF is new to partnerships of this nature, which may reduce the degree of comfort. In addition, AMF may have caused some interpersonal tension, discussed more below.

We aren’t able to disentangle these factors with confidence and say which have been the major ones in AMF’s difficulty finalizing a distribution.

AMF believes that the primary factor is its concern over theft and its reluctance to provide nets without strong measures in place to ensure that they will arrive safely and be distributed to the right people. It believes its insistence on transparency and detailed accountability has caused it to encounter significant resistance from some others in the aid community who have less focus on this area. We are not as confident that we can identify the primary factor, though we do believe that AMF’s emphasis on transparency and accountability (which we broadly support) creates inherent challenges.

Does this indicate a lack of global funding gap for insecticide-treated nets?
In the past, we have analyzed population-at-risk data and distribution data to estimate the global net gap, concluding that it is quite large. We are currently in the process of finalizing our refresh of this analysis, and we believe that we will again project a gap, although the fact that nets have gotten cheaper over time – as well as increased funding from other sources – means the estimated gap is likely to be substantially smaller than previous estimates.

We believe that the total number of nets available over the past couple of years has been insufficient to cover the full population at risk. In addition, the partial progress AMF has made in arranging distributions (see notes from Don de Savigny for more) does not, to us, imply a world in which nets are easily available. AMF has encountered multiple countries in which the available funding did not seem sufficient to cover the full planned distribution, and in which AMF was requested to fill the gap.

Going forward, we believe that the global funding gap may be smaller (more), which may steepen AMF’s challenges in terms of negotiating distributions, though it appears that some degree of funding gap will remain.

Has AMF made mistakes? What could it have done better?
As stated above, it’s difficult to disentangle the factors that have led to large-scale distributions’ not going forward so far. In some cases we feel that these factors are unavoidable (AMF’s size) and/or reflect AMF’s strengths from our perspective (its interest in collecting and publishing data on distributions). We also wish to note that AMF has completed many smaller-scale distributions (up to several hundred thousand nets) in the past, and we feel that its struggles can fairly be attributed to the increase in size that has come with our recommendation. With that said, there are some cases in which we feel that AMF has made mistakes, and/or in which we continue to disagree with AMF. Much of the thinking here has been driven by recent conversations with people who have worked with AMF; some of these conversations remain off the record, but we have posted notes from Don de Savigny of AMF’s Malaria Advisory Group.

Excessive requirements

Broadly, we support AMF in its attempts to secure strong and public reporting on its distributions, even though doing so may make it more difficult for such distributions to be negotiated.

With that said, it appears to us that AMF’s requirements should have been rethought and adjusted more than they were, at the point when AMF made the transition from partnering with nonprofits to partnering directly with governments. Don de Savigny of AMF’s Malaria Advisory Group was among the people who noted this concern. We believe that AMF now plans to adjust its approach.

A couple of specific notes:

  • While we have previously been enthusiastic about AMF’s requests for malaria case rate data, we now believe that the costs of requesting this data may outweigh the benefits. Our view has changed because (a) we have come to believe that this data is highly unreliable (more at our conversation notes from August 15th); (b) we have been told that some countries simply do not have the capacity to collect this data, at least without funding to support their health-system infrastructure more broadly. AMF does believe that if possible an attempt should be made to track the impact of nets in this way, but also recognizes the case for flexibility on this point.
  • We still believe that other data AMF seeks (regarding delivery, condition and usage of nets) is valuable and important, and its raising questions around such data may already have had some impact on discussions in the malaria control community (more at our conversation notes from Don de Savigny of AMF’s Malaria Advisory Group).

Preference for funding only nets

AMF prominently states that donations from the public are used exclusively for buying nets. It believes that it must make good on this promise.

Last year, we became concerned that AMF’s preference for funding only net purchases (and not other costs associated with distributions, such as shipping, monitoring and evaluation) might be an obstacle to distributions’ moving forward. We spoke with Rob Mather about the matter, and got the impression in our conversation that (a) this had not been a major obstacle to moving forward; (b) if it became such an obstacle, AMF would raise money specifically for this purpose. Good Ventures’s 2013 donation was specifically marked as available for non-net costs; Rob was reluctant to mark other donations in this way and stated that he did not see a need, and we didn’t perceive it as a major issue, so we didn’t pursue the matter further. We also note that AMF has paid for non-net costs in its Malawi distributions, so it is clear to us that its reluctance to fund such costs is not absolute.

This year we have encountered one case of AMF’s offering to fund additional nets, with another funder’s then arranging to reallocate their funding from nets to non-net costs, specifically net use monitoring costs. (AMF did not feel able to offer funding for non-net costs when it was not funding nets.) This has highlighted to us that we previously underestimated the degree of AMF’s reluctance to fund substantial non-net costs, and with it, the potential consequences.

AMF has stated strongly to us that it does not believe its not offering to fund non-net costs is a major factor in agreements’ not being finalized. We remain unsure of whether AMF’s reluctance to fund non-net costs is a major factor (or any factor) in its struggles to finalize a distribution, but it seems to us that this is a strong possibility.

Having spoken further with AMF, we’ve been given two arguments in favor of avoiding paying for non-net costs:

  1. AMF has no in-country presence to effectively monitor non-net costs. It is easier to track and audit the use of funds (in a way that e.g. avoids bribery) in the case of net purchases than in the case of other costs (particularly costs of distribution and evaluation). AMF has the technical capacity for the former, but not for the latter, for which it believes in-country presence or a highly accountable in-country partner is vital. AMF seeks to ensure that major (i.e., government) funders with substantial in-country staff take responsibility for tracking and auditing non-net expenses, and believes that the most practical way to do this is to ensure that major funders allocate their funds to non-net costs. This issue may also have legal implications: AMF has received professional audit guidance that paying for substantial non-net costs could incur legal requirements to audit the funds that would go beyond AMF’s technical capacity. We are unsure of how significant an issue legal requirements represent, and plan to look into the matter further; overall, we believe this could potentially be a strong argument for behaving as AMF has, but remain uncertain. Note that this argument pertains to tracking expenditures and ensuring appropriate use of funds, which is different from evaluating e.g. usage and impact of nets (an area in which we feel that AMF has higher standards than government funders).
  2. Many of AMF’s donors prefer that their funds be restricted in this way, and because of the commitments given on AMF’s website, AMF has an obligation to assume by default that this is what its donors prefer. (AMF also notes that a) it believes it has paid for non-net costs when not doing so would hold up a distribution and the other pieces are in place; b) it has approached larger donors, and is prepared to approach other large donors, to ask whether specific donations could be allocated to non-net costs if such a need arose.) In response, we have communicated strongly to AMF that we do not believe the bulk of GiveWell-attributable donations should be restricted in this way, and we have offered to survey donors we have contact information for if it would help resolve the matter.

At this point, we believe there may be good justification for AMF’s reluctance to fund non-net costs, but we also believe that at least part of the reluctance comes from AMF’s perception of donors’ preferences – and thus from a failure on our part to ensure that donors coming from our website communicate their preferences to AMF. We believe that this is at least partially a matter of persistent disagreement: AMF sees value (when communicating to the broader public) in the simplicity and tangibility of “100% buys nets,” while we believe this is not valuable where GiveWell-directed donors are concerned.

Problematic communication style

AMF has pushed for what we see as new levels of information and public accountability, and this creates natural challenges and calls for building relationships in the development community. To this point, we believe that AMF has handled these challenges suboptimally, in the following respects:

  • AMF has done relatively little in-person interaction with potential partners, as opposed to communication by email and phone.
  • AMF has had a very direct communication style, and may have failed to pick up on (and encourage) communications from people whose style may be less direct. We feel that this dynamic may have led to AMF’s failing to detect potential problems as quickly as it could have – in particular, we were told that AMF should not have been as surprised as it was by the Sierra Leone distribution’s falling through. The sources from which we’ve drawn this view unfortunately remain confidential.

We recognize that the same quality that has made AMF particularly easy for us to communicate with (a highly direct communication style) may be a drawback in its relations with some potential partners.

We believe that AMF is taking appropriate steps (more below) to improve on this front.

The upshot of the above three factors is that AMF may have been perceived as overly demanding by potential partners – coming in as an outsider, asserting substantial requirements without offering to directly reimburse the costs of its requests, and appearing less open to negotiation than it was in reality.

With that said, we doubt that the above factors have been the primary explanation for AMF’s struggles to finalize a distribution. We see its small size and emphasis on strong and transparent data sharing as fundamental obstacles, and we’re cognizant that it’s been willing to negotiate on other factors. We believe that the above issues have largely been a function of transitioning from one sort of partner (nonprofits on smaller-scale distributions) to another (government partners on large-scale distributions), and as discussed below, we believe that AMF broadly is taking appropriate steps to improve on these matters.

How is AMF responding to its difficulties?
We believe that Rob Mather, AMF’s Founder and CEO, is at least as eager to see a distribution finalized as we are, and is highly concerned about the delays.

When he was unable to finalize a distribution last year, he committed to building a bigger pipeline of potential partners (details), and we believe he followed through well on that goal, as he explored a significantly larger number of options in 2013.

This year, the main step Rob is taking in reaction to the delays is to solicit greater involvement from his Malaria Advisory Group, specifically Don de Savigny, Bob Snow, and Abdisalan Mohamed Noor. The Malaria Advisory Group normally provides input on AMF’s distributions (examples), but is not employed or paid by AMF, and the individuals in question are well-credentialed malaria scholars (see bios for Don de Savigny, Bob Snow and Abdisalan Mohammed Noor).

We believe this is the appropriate action to take. These individuals are likely to command respect both from Rob and from the malaria community at large, and they will hopefully consider both AMF’s interest in strong reporting and arguments from the malaria community about what level of reporting is excessive, without having strong incentives to put more weight on one or the other. Therefore, their findings are likely to have the best combination of neutrality and respect from both parties that seems feasible in this situation.

Rob has also encouraged several of AMF’s trustees – particularly Peter Sherratt and Richard Lane, who he feels have relevant experience of working with governments and in development – to become more involved in investigating and negotiating potential distributions.

Finally, we have provided written feedback to AMF and its advisors and trustees, and will be sending them this post as well.

Ultimately, we expect AMF to reach reasonable resolutions on its reporting requirements and on the issue of how best to communicate them. We are less sure that we will end up in sync with AMF on the question of non-net costs, but we believe there are legitimate arguments for its perspective on this matter and will respect whatever it ultimately decides on this front, while making clear that AMF should not consider donor preferences to be a major argument for restricting its funds (at least concerning donors going on GiveWell’s recommendation).

What will happen to the money donated to AMF if AMF can’t finalize a distribution?
We have encouraged unrestricted donations to AMF, so in most cases we feel it is ultimately up to AMF how to use the funds. (AMF believes it is bound by its public commitments to use the funds to buy nets unless specifically agreed otherwise.) Our impression is that AMF will:

  • Commit the funds to a country-wide distribution that meets high standards of data collection and sharing.
  • Distribute nets in multiple smaller distributions over a longer period of time.

In either case, we feel that the substantial delay in committing funds represents a substantial negative development, though we also feel that dollars spent well with a delay (as we believe they will be in either case) are better than dollars spent poorly.

We asked AMF what its plans would be if this approach too proved infeasible (which we consider to be unlikely but possible). AMF responded that the probability of this is too low to merit planning for this given the continuing prevalence of malaria and a significant global net gap.

What is GiveWell’s current view of AMF? Why is GiveWell planning to withdraw its recommendation?
We continue to feel overall that AMF is an outstanding organization in that it is aiming to bring a unique level of accountability to one of the most proven, cost-effective interventions that exists.

However, a sizable risk has emerged that AMF will not be able to find enough distributions to commit the funds it has, and increasing that amount would steepen this challenge (though perhaps not if AMF raised such significant new funding that it became comparable in size to other major funders in relevant countries). Therefore, we are planning to recommend that people looking to give now donate elsewhere, until and unless AMF demonstrates an ability to put together large enough distributions to commit the funding it has now. We note that AMF does not share our view on a lack of room for more funding. AMF’s take is available here. 

We feel that AMF has made some mistakes, noted above. We are somewhat concerned that AMF is perceived as an “outsider” and – through some combination of communication style and perceived reluctance to fund non-net costs – has been making a poor impression on potential distribution partners. We believe that AMF is taking appropriate steps to diagnose and address these issues, despite some potential disagreements, particularly on the question of non-net costs. We remain unsure of the relative importance of AMF’s mistakes and the fundamental challenge of asking for unusual levels of transparency and accountability (and note that AMF feels the latter is the primary factor).

We wouldn’t guarantee that AMF would immediately become recommended again upon committing the bulk of its funds. The findings that come out of its Malaria Advisory Group’s increased involvement, and the exact circumstances under which a distribution is finalized, would affect our view. However, if and when AMF does commit the bulk of its funds, we will reconsider its ranking and will likely recommend it again.

Did GiveWell err in its previous recommendation of AMF? What has GiveWell learned from AMF’s struggles?
We’ve been reflecting on this episode and trying to draw lessons from it. While we still have some reflecting to do, at this point some lessons we believe we can take away are:

  • We should think harder and communicate more clearly about cases in which our recommendation could bring about a change in the scale of a charity. We were always aware that our recommendation would substantially increase the funds available to AMF, but initially we didn’t recognize that this would fundamentally change the character of the distributions it was partnering on (see our 2011 review). By our 2012 review, it had become clear that ability to finalize a distribution was a major risk factor, but on balance we decided it was a relatively minor one. We think it’s a good thing to substantially increase the funding of outstanding charities, challenging them to reach a new level of scale. Such an endeavor has potential “upside” of turning a limited organization into one with more absorptive capacity, as long as we don’t multiply the funds available by too high a number in too short a time. In the case of AMF, we feel that the increase in funds was appropriate – enough to bring it to a new level of scale, but not much more than that. However, it needs to be strongly noted, both internally and externally, that such a “step up” involves major risk. We don’t feel that we were clear on the nature of the risk here internally or that we communicated enough about it externally. In the future, when we are recommending that a charity be allowed to substantially increase its scale, we will make more effort to note the major risks that this will entail – and this will apply to this year’s top charities as well.
  • We need to ensure clear communication between the donors coming to our website and the charities receiving funds from them, in terms of donor preferences. This is the second time that a recommended charity has expressed to us that it assumes – by default – that its donors wish their funds to be restricted in certain ways. (The other case involves GiveDirectly; in that case we had clear communication up front and ensured that our donate page clearly distinguishes between “flexible funds” and “typical funds,” with the latter being more restricted). We strongly prefer to avoid restrictions on donations, and we don’t want our recommended charities to feel bound by donor preference for anything other than optimal positive impact.
  • We should look for ways to deepen and improve our evaluations of people. We’ve previously noted that we place some weight on our evaluations of the people running an organization, and we’ve also expressed reservations about doing so (“we’re wary of putting too much weight on informal, intangible impressions that are likely to end up being ‘tests of how similar charity representatives are to us’ rather than ‘tests of how well-suited charity representatives are to do their work.’ It seems easy for evaluations of people to end up being driven by charisma, by similarity to the evaluator, and/or by characteristics that don’t translate well to nonprofit work.”) We still believe that Rob is an outstanding individual, and we don’t want to place too much weight on complaints about his communication style when these could easily be motivated by frustration with other aspects of AMF, particularly its data disclosure requests. We also believe that the concerns on this front are being addressed appropriately. That said, the complaints have highlighted the point (which we previously noted) that our evaluations of people may overweight their similarity to us, particularly in communication style, and that our evaluations can miss substantial ways in which qualities that make a charity good at communicating with us can be unhelpful (or even drawbacks) for its ability to do its work. We want to find other methods of evaluation that can enrich our understanding of this aspect of an organization, such as the idea in the next point.
  • We should put more effort into interviewing other organizations and individuals who have worked with our recommended (or potential recommended) charities. We feel that some important considerations have been raised by our discussions with others in the malaria control community about their interactions with AMF. In general, an organization’s reputation among those it works with is important. This may also be a consideration in favor of highly credentialed organization leaders (who are less likely to come across as outsiders); this is something that we haven’t previously placed much weight on.
  • In general, we need to be highly cognizant – and to effectively communicate – that as of today, there don’t appear to be any “sure bets” with quantifiable outcomes in charity, especially when it comes to directing relatively large amounts of money. Many of those who write about GiveWell or use it to donate seem to be under the impression that our top charities represent “sure thing” options allowing a donor to save or improve a known number of lives per dollar given. Yet every year brings new observations about the complexity of giving and about the caveats to seemingly straightforward value propositions. We’ve written before (in 2011) that there is no such thing as a “sure thing” in giving, but we think this point merits further discussion, and we plan to discuss it more in a future post.

With all of that said, we don’t regret our prior recommendation of AMF and believe it was appropriate. AMF had a substantial track record of conducting multiple distributions, including one large distribution, with a level of public disclosure and tracking that no other net distribution charity (and few other charities of any kind) can match. Over 2 years, we have directed enough funding to it to multiply its funds available by several-fold, in the hopes that it would be able to operate at a larger scale; given our other options for recommended charities and the upside if this happened successfully, this was a risk worth taking. It still may be a risk that turns out well, as AMF continues to take steps to improve and to overcome its challenges; it was a risk that has led to substantial learning on our part, which has only been possible because of AMF’s strong transparency and direct communications; and even if AMF is not ultimately able to find appropriate distributions, the result will be delayed (not wasted) donations, as discussed above. We continue to strive to improve, but we stand by our earlier decision to recommend AMF as our #1 charity, and there is a strong possibility that we will do so again in the future.

Finally, we’d like to acknowledge AMF’s willingness to be open and transparent throughout this process. It has reviewed a draft of this post, and we believe its willingness to let individual donors follow along with it – both the successes and the struggles – is highly unusual in the charity world, and will prove enormously beneficial in the long run.

Comments

  • Jonah Sinick on November 26, 2013 at 4:44 pm said:

    Interesting post.

    The discussion under the heading “Did GiveWell err in its previous recommendation of AMF?” is ambiguous with respect to whether:

    (i) You believe that given the information that was available at the time, recommending AMF was the right thing to do, but that knowing what we know now, marginal donations to GiveDirectly were more cost-effective than marginal donations to AMF.

    (ii) You believe that relative to our current state of knowledge, marginal donations to AMF were more cost-effective than marginal donations to GiveDirectly.

    Which of these is your position?

  • I have been concerned in the past by your explicit consideration of the quality of the people who lead your top charities. In addition to the problems you have cited, it seems to me humans are well-documented to be very poor judges of other humans. It seemed to me your acknowledgement of this consideration as a factor in your judgements risked biasing you in favor of giving the consideration more weight than was warranted.

    The approach of speaking with other organizations about their interactions with a given charity seems to me a very desirable improvement of method. At the very least, it should given a relatively accurate view of how an individual and organization is perceived, which certainly contributes to its effectiveness. I appreciate these developments and find them encouraging for your project overall (though obviously not for AMF specifically.)

  • Luke Muehlhauser on November 27, 2013 at 11:48 am said:

    It’s too bad that AMF’s admirable and somewhat costly transparency is part of the cause for their current lack of RFMF (assuming Holden is right; I await AMF’s reply). It’ll be interesting to learn more in the future about the domains in which this tends to happen, and the domains in which it doesn’t.

    Once again I’m glad these issues are being discussed in public, so that the entire philanthropic community can learn.

    I hope GiveWell will one day be able to present what they’ve learned thus far at the annual retreat that presents tutorials for the Giving Pledge people on how to give more effectively. (See the recent 60 minutes on this.)

  • Jonah, I think the answer is different at different margins. Getting AMF to the next level of scale remains, I think, a risk worth having taken and probably a better use of funds than if those funds had gone to GiveDirectly, but it’s hard to say exactly how many dollars were needed to accomplish that end. Getting GiveDirectly to a comparable level of scale would also involve risks. I still think a sizable amount of the funds AMF is holding are doing more good being held by AMF than they would have if donated to GiveDirectly, but I don’t have a stable view on the matter; the focus of the statement was intended to be “given the information that was available at the time.”

  • I suspect there’s a lesson here for GiveWell, too.

    Has GiveWell encountered any difficulties due to its direct, intellectually honest style? In my personal experience, people are generally not interested in the truth if it conflicts with their self-interest or notions of self-worth; they are more than willing to use political levers to attack or undermine threats to their power or egos.

  • Rob Mather on November 28, 2013 at 11:26 am said:

    We defend GiveWell’s right to conduct the analysis they do and the way they do it. We have always said this, irrespective of AMF’s individual ranking, and we remain staunch supporters of their approach.

    We disagree with some of the analysis GiveWell has done and the conclusions reached, and we have been open and frank with them about this. This particularly applies to their take on non-net costs and room for more funding. Our experiences tell us potential theft is a major issue with large net distributions and we haven’t quite been able to convince GiveWell of our approach to this. We have written more about our approach here (http://www.AgainstMalaria.com/Update28Nov13). GiveWell have every right to place a different emphasis on this than we do.

    That said, we have certainly made mistakes, and no doubt will continue to do so, from which we learn to do things better.

    AMF’s focus is on reaching agreements for net distributions with the funds we have in-hand. Some of these discussions are advancing well and would use the entire funds currently held by AMF.

    Over the last four years AMF’s ability to fight malaria has benefitted enormously from contact with, and the reviews of, GiveWell. Beyond this, the more detailed critical review and direct conversations with GiveWell staff have helped AMF to improve as an organisation.

    We wish to continue to be evaluated, commented upon publicly and be held to account by GiveWell and others, particularly our donors. We will of course continue to be open with requests from GiveWell staff for information.

  • I was recently concerned about the impact of AMF, because when AMF donate nets, it’s possible they simply save another charity from having to give the nets- the ‘net effect’ (sorry!) thus being whatever the *second*, displaced, charity then does with the money it’s just saved. But if AMF struggle to find organisations who’ll accept their nets, it suggests orgs may only use AMF nets if they *really have to*, and hence AMFs distributions *are* likely to be to people who really couldn’t find nets elsewhere! AMF’s ‘net effect’ (I really can’t avoid that pun) might look better after the doubts about room to expand! Provided, of course, they do in fact find some (reluctant?!) recipients.

    How confident was GiveWell that AMF only/mostly gave nets out where other donors were not forthcoming? I imagine this must be very hard to assess.

  • Sam, we have encountered some hostility due to our style, but we have also been working on our tone over time (it has long been acknowledge on our Mistakes page) and feel that we’ve been encountering less hostility over time as well.

  • Rob, thanks for the response. We have added a link to your update in the main post. We agree that AMF has been extremely open with us.

    To be clear, we see the risk of theft as a potential issue (along with risks around whether nets stay in use and stay in good condition post-distribution) and broadly support AMF’s attempts to raise the bar for data collection and reporting around these issues.

  • John Maxwell IV on December 2, 2013 at 2:31 am said:

    How bad is it if the nets get stolen, really? That’s one way for them to be “distributed”. Either the thief uses the net themselves or they sell it on the open market (the net effect of lots of thieves doing this is that the market is flooded with black-market nets, driving the price down and making them more affordable). The big downside is the transfer in wealth from Africans who need nets to Africans who are unethical enough to steal nets.

  • Peter Hurford on December 2, 2013 at 3:59 pm said:

    One aspect I’m concerned with is that it isn’t just AMF that lacks room for more funding, but all GiveWell top charities, due to the influence of Good Ventures. I noticed that the “minimum targets” and, for that matter, even the “maximum targets” are all within the total endowment size of Good Ventures. Yet, while we haven’t seen Good Ventures’s allocations yet for this year, I imagine they won’t fully absorb these targets.

    I understand that many foundations don’t want to disperse their endowments all at once, but Good Ventures isn’t an ordinary foundation. Do they just not want to take the advice you’ve outlined in ~ (~), or have things changed since then?

    Why isn’t Good Ventures fully funding these opportunities? If they’re not, why should the logic be different for individual donors? Does Good Ventures scale open them to opportunities that individual donors don’t have? If so, why shouldn’t individual donors give to Good Ventures instead?

    PS: Let this not be taken as a critique of Good Ventures. What they’re doing is wonderful and it’s amazing to have an “effective altruist” foundation, let alone one with so much money.

  • Alexander on December 3, 2013 at 2:42 pm said:

    Sally: In general, we do not believe there is a reliable way to assess whether particular gaps that AMF fills would have been filled in their absence. However, the way we’ve dealt with this consideration in the past is to focus on the magnitude of the global net gap, on the premise that even if AMF is offsetting other net donors, the existence of an overall gap suggests that other net donors are likely to redirect their nets to another place, instead of doing some other activity with their funds, but we do not know the extent to which that is actually the case.

    John: We don’t know for sure, but we would guess that nets that are stolen are used less effectively than nets that are distributed as intended. Our impression is that few individuals purchase nets to use appropriately, so nets that are stolen may end up being used for less valuable uses (e.g. fishing net or decoration or bridal veil). Also, donated nets are typically targeted at poor communities; stolen nets may end up purchased by wealthier households that would purchase nets directly from the manufacturers otherwise (which could be double harmful because community coverage may play an important role in the effectiveness of nets).

    Peter: You can read about how Good Ventures decided how to allocate their funds this year here. We don’t believe it would be advisable for them to give more than they are, both because we believe it would offset individual donors and because we expect to have better giving opportunities for them (and for individual donors) in the future. We’re planning to write more about the tradeoff between giving now and saving this year in the coming weeks, but we think it would be defensible for individual donors to choose to save this year.

  • Ian Turner on December 6, 2013 at 7:01 am said:

    In your 2012 blog post on evaluating people, you wrote “A major hesitation is that we?re wary of putting too much weight on informal, intangible impressions that are likely to end up being ‘tests of how similar charity representatives are to us’ rather than ‘tests of how well-suited charity representatives are to do their work.'”

    You wrote that a year after your first recommendation of AMF. In light of the current outcome, does that make you feel more comfortable with evaluating people (since you may be biased to organizations like GiveWell anyway); or does it make you want to revisit the way you evaluate people in the first place? Do you expect to revisit that 10/2012 blog post?

  • Ian Turner on December 7, 2013 at 11:29 am said:

    Regarding stolen nets, it also seems possible to me that nets which are stolen wholesale could be resold to other nonprofits, essentially converting the theft into cash.

  • Holden on December 9, 2013 at 3:50 pm said:

    Ian, I’m not sure I follow. The 10/2012 blog post is still an accurate reflection of the main factors that go into our current evaluations of people, though as this post states, we’re thinking about how we can add more to the picture.

  • Ian Turner on December 11, 2013 at 7:57 am said:

    Hi Holden,

    I guess my question is, if one major problem with focusing on people is that you may end up biased towards people like you, which may cause problems; and if it turns out that you end up with that bias anyway (by focusing on transparency, for example), does that make you feel more comfortable with evaluating people? (Or less, or no difference?)

  • Ian, we think the bias caused by requiring transparency is on its way to fading over time as our influence grows, and it isn’t identical with the biases we might have in evaluating people (though there is some overlap). Overall I’d say this episode has made us slightly more cautious about evaluating people.

  • Andrew Menan on December 20, 2013 at 11:21 am said:

    I’m dissapointed because I was wanting Givewell and Against Malaria to contact me about this as I raised over $600 in January 2013 and no nets have been disbursed. Despite both organizations thoroughness- a very simple, monthly or quarterly email sent out to givewell supporters and Against Malaria Supporters- I think would serve our community better than this Blog. I know both organizations are doing great, hard work. Rather than blame, please remember to just inform your donors about what’s going on.

  • Peter Hurford on December 20, 2013 at 11:53 am said:

    @Andrew: GiveWell recognizes this as a mistake. See “February to September 2013: Infrequent updates on our top-ranked charity” under the Mistakes section for the explanation and plan for fixing the problem going forward.

  • Andrew Menan on December 20, 2013 at 12:30 pm said:

    Thank you for your response Peter. My suggestion is that the heading could read “February to September 2013: Infrequent updates on our top ranked charity Against Malaria” so that a reader may recognize it easier.

  • GERALD NOVAK on December 26, 2013 at 12:34 am said:

    Regarding expenses vs. revenue: Can I have a report showing the details of these numbers?

    Also, what is revenue? If I donate X dollars to a charity, is that considered revenue? (I sure hope not!)

    What is the source of your revenue? Do you conduct fund raising drives to cover operational expenses? What are your administrative & overhead expenses? (This should all be covered by the report requested above/)

    Thanks for your help.

  • Ian Turner on December 28, 2013 at 9:08 pm said:

    Gerald, if you’re looking for financial information on the Against Malaria Foundation, you might get better traction asking the Foundation itself directly.

Comments are closed.