The GiveWell Blog

More errors in widely-cited figures: The case of mothers2mothers

(updated on: ) | by Holden

Note: mothers2mothers has provided a response to this post that can be viewed below.

Summary: mothers2mothers, a well-respected group that focuses on HIV programs in Africa, published figures on its website that we have recently come to believe are erroneous. We feel this finding is important not because of what it says about mothers2mothers, but because of what it says about the wider community that has been funding, awarding, and citing mothers2mothers’s figures.

mothers2mothers (m2m) is a group focused on prevention of mother-to-child transmission of HIV (PMTCT). It has as many awards – and major funders – as any nonprofit (its size) that we’ve seen.

Its published figures suggest that it serves a huge number of women – specifically, that it accounts for around 20% of all women on PMTCT in sub-Saharan Africa. Yet when we performed a simple check on its figures, we saw major anomalies:

  • In some countries, m2m’s stated number of women served exceeds the national total for women on PMTCT, from public UNITAIDS data. (For example, in Swaziland m2m reports ~25,000 mothers served; but UNITAIDS reports a total of ~10,000 Swaziland women on PMTCT for the same year.)
  • In other countries, m2m’s stated number of women served does not exceed the national total, but it is enough to account for 50-100% of it. However, looking at the trends in national data, one does not see an increase after m2m’s entry into the country. (Charts below.)

After corresponding with mothers2mothers, we believe that the anomalies we’ve seen are chiefly explained by flaws in mothers2mothers’s data.

mothers2mothers has told us it is now considering adding a disclaimer to its website. It has also provided a response, which is included below.

We feel this finding is important not because of what it says about mothers2mothers, but because of what it says about the wider community that has been funding, awarding, and citing mothers2mothers’s figures.

There is a lot more to the value of a nonprofit than the quality of its data, and there are a lot of questions that a good investigation ought to ask besides whether the numbers add up. We certainly don’t think the anomalies we’ve found show that m2m isn’t doing great work, or that its support and awards are undeserved (and we are still considering the possibility of further investigating m2m as a potential GiveWell-recommended group). Still, seeing this sort of problem from an organization that gets as much attention as m2m seems significant. It’s another piece of evidence that the philanthropic world – including many of the largest and best-resourced funders – is not asking all of the critical questions that it could be asking.

The implausible implications of mothers2mothers’ figures, and how we came across them

We’ve long found mothers2mothers to be potentially promising because of its focus on antiretrovirals for the prevention of mother-to-child transmission (PMTCT) of HIV, which we consider a priority intervention. We examined mothers2mothers in 2009, and concluded that the evidence of effectiveness it was pointing to didn’t meet our standards (details). In April of 2011, our interest was rekindled when a funder of mothers2mothers told us that mothers2mothers treats a substantial percentage of all the women needing PMTCT worldwide.

Our immediate reaction was: “If that’s true, we ought to be able to see the case for mothers2mothers’s impact in country-level data – perhaps via major improvements in country-level data following mothers2mothers’s entry into a country – and a convincing impact at that high a level would be impressive and compelling.” So we decided to collect the relevant data and see what sort of picture it presented.

Philanthropy’s success stories

(updated on: ) | by Holden

[Added August 27, 2014: GiveWell Labs is now known as the Open Philanthropy Project.]

The first item on the agenda for GiveWell Labs is to get the broadest possible view of philanthropy: its history (what’s gone well, what hasn’t) and its current state (what’s being funded and what isn’t).

On the history front, I’ve found very little of interest. Most books on philanthropy are how-to’s rather than histories, and most of the histories don’t appear to focus on listing specific cases where foundations had (or failed to have) impact. One exception is the Casebook for The Foundation: A Great American Secret, which lists and discusses “100 of the highest-achieving foundation initiatives” since 1900.

I thoroughly examined this volume, and collected some basic notes into a spreadsheet. My reflections follow. In brief, I felt there were some very strong cases here, particularly in the area of medical research, and I was surprised by philanthropy’s history of being active in shaping various graduate education programs. At the same time, I thought the Casebook’s history had important shortcomings – in particular, not putting successes in context along with failures – and I see a lot of room for improvement in the amount of information available about the history of philanthropic successes and failures.

Note that I am not well-informed about most of the cases discussed here and have relied on the Casebook for my information. The notes that follow are only loose impressions and are not backed by the sort of evidence that we usually seek (even for blog posts).

Philanthropy has some extremely impressive accomplishments. Among other things, foundations have been (in my view) reasonably credited for leading the way on building schools and hospitals in the rural Southern U.S., piloting the shoulder line on U.S. roads, successfully advocating for federal legislation in areas including health care for the homeless and nuclear deproliferation, the research that led to the Green Revolution, and many major advances in medical research (including the first combination drug therapy for AIDS and the development of the pap smear). There are many other projects that sound like they may have been impactful, but which the Casebook doesn’t give enough context on for me to have a strong view.

The most impressive cases (in my view) are mostly the earlier ones. Though the Casebook focuses on more recent philanthropy (78 of its 100 cases are post-1950), 9 of the 14 cases I found most impressive are pre-1950 (and a 10th is from 1952).

A possible explanation is that the space of doing good has become more crowded over time. For example, note that

Update: Against Malaria Foundation’s costs

(updated on: ) | by Stephanie Wykstra

We learned last week that we underestimated the total cost of the Against Malaria Foundation’s insecticide-treated net distribution.

A nonprofit interested in reducing costs for its own model read our AMF review and contacted AMF and Concern Universal (AMF’s distribution partner in Malawi). The nonprofit representative noticed that Concern’s costs seemed lower than he expected, and he hoped that he could find a way to reduce his own organization’s costs.

Concern Universal’s response to the nonprofit representative’s request listed several categories of costs that we excluded in our cost estimate. We will release a full report about these costs once we have all the details and have put together a revised estimate.

In our view, it is a huge benefit of our transparency that people outside GiveWell are able to scrutinize the material we publish on charities and ask questions that lead us to catch errors. This allows us to improve our charity reviews as well as our overall process.

Update: A few people have asked us, “How likely is this to change AMF’s #1-ranking?” It almost definitely will not change based on this. Our back of the envelope estimate of the costs puts them at less than $1 per net, which would not change our bottom line.

GiveWell’s plan for 2012: Specifics of research

(updated on: ) | by Holden

[Added August 27, 2014: GiveWell Labs is now known as the Open Philanthropy Project.]

We previously laid out our high-level priorities for 2012. The top two priorities are “make significant progress on GiveWell Labs” and “find more outstanding giving opportunities under the same basic framework as our existing recommendations.” This post elaborates on our plans for these two priorities.

A note on relative priorities: our current top charities have significant room for more funding, so it would not be catastrophic (though it would be highly undesirable) to end 2012 without new top charities. Because of this, we view GiveWell Labs as slightly more crucial for 2012. However, we plan substantial work on both and anticipate that the quality of our standard research will continue to improve significantly.

GiveWell Labs

We believe that GiveWell Labs is very important for our long-term impact; it represents a substantial new opportunity to both find great giving opportunities and expand our potential target audience (more).

However, at this time GiveWell Labs is still in the very early stages. (We announced it in September, but a few weeks later put our entire focus on finding top charities in time for 2011’s holiday season.) The stage it’s at is somewhat comparable to the stage GiveWell was at in August of 2007; and like the GiveWell of 2007, we will probably go through a lot of experimentation, go down some significant dead ends, and possibly miss some deadlines and change our vision of what we’re trying to accomplish. So we don’t want to commit to highly concrete or definite goals at this time.

That said, here’s our current working framework.

The building blocks of GiveWell Labs

As we try to find the best giving opportunities, we believe it will be helpful to work separately on the questions of what the most promising sectors (general areas of philanthropy, such as “climate change mitigation” or “tuberculosis control” are) vs. what the most promising projects are within a sector. We’re thinking of GiveWell Labs as being divided into the following categories:

  1. Completely open-ended, sector-agnostic investigation (example: examining data on foundation grants to get a sense for what today’s foundations work on).
  2. Basic research to determine how promising a sector is (example: investigating climate change in a low-depth way, focusing on determining what strategies are open to philanthropists and whether their cost-effectiveness could be competitive with other sectors).
  3. In-depth work getting a deep understanding of a particular sector (example: trying to gather as many relevant ideas/conversations as possible for tuberculosis control).
  4. Researching a particular project, or kind of project, to determine whether to recommend it.

There is some justification for doing the 4 steps sequentially: #1 helps one choose the right sectors to research (#2), which helps one choose the right sectors to focus on (#3), which helps one choose the best projects to recommend (#4). However, there is also some justification for working on multiple tracks in parallel: learning more about specific projects and specific sectors will probably inform the way we go about deciding between sectors, and there are some sectors we already know well enough to consider them high-priority. In addition, we don’t ever expect to have final or rigid choices of the most promising sectors, and will always be open to particularly promising projects from any sector.

GiveWell’s plan for 2012: Top-level priorities

(updated on: ) | by Holden

[Added August 27, 2014: GiveWell Labs is now known as the Open Philanthropy Project.]

In previous posts, we discussed the progress we’ve made, where we stand, and how we can improve in core areas. This post focuses on the latter, and lays out our top-level strategic choices for the next year.

The big picture
Broadly, we see the key aspects of GiveWell – the areas in which we can improve – as

  1. Research expansion: finding additional outstanding giving opportunities.
  2. Research maintenance and systemization: keeping our research up to date, while allocating as much responsibility as possible to junior staff. This includes regular updates on charities that we have directed significant funding to.
  3. Research vetting: checking the quality of our research and providing evidence for this quality.
  4. Outreach: working to increase awareness of GiveWell, traffic to our site, conversion of website traffic into donors and followers, etc.
  5. Fundraising/operating: maintaining the organization.

These are broadly similar to the areas for improvement we’ve listed in the past. And as in the past, we feel that the first two of these – finding more outstanding giving opportunities and staying up to date on those we’ve found – are at the core of our work and remain our top priorities. The basic reasoning:

  • In 2011, as in 2010, we experienced substantial growth despite not making outreach a major priority for the year. We believe this is because producing quality research – while taking “low-hanging fruit” on the outreach side – leads to strong growth in referral links, organic search performance, and word-of-mouth. From surveying our largest donors, we believe that it is common for such donors to read our reports relatively carefully.
  • We believe that our chances for growing over the long term are highly dependent on our providing as much “room for money moved” as we can in the most outstanding giving opportunities possible, while doing as much due diligence as possible to maintain a strong reputation.
  • Letting up on the quality of our research would be a major risk. A highly dedicated effort to outreach would not clearly have greater returns (just in terms of attracting attention) than continuing to improve the quality of our research.

We are planning some work on #3-#5 above, but believe that we can perform strongly in both areas without making them major priorities for the year.

Our priorities and goals
#1: make significant progress on GiveWell Labs, the new arm of our research process that will be open to any giving opportunity, no matter what form and what sector. A future post will discuss the specifics of our plans and hoped-for progress on GiveWell Labs. This initiative represents a substantial new opportunity to both find great giving opportunities and expand our potential target audience (more).

#2: Find more top charities under the same basic framework as our existing recommendations. A future post will elaborate on our plans for this.

Our current top charities have significant room for more funding, so it would not be catastrophic (though it would be highly undesirable) to end 2012 without new top charities. Because of this, we view GiveWell Labs as slightly more crucial for 2012. However, we plan substantial work on both and anticipate that the quality of our standard research will continue to improve significantly.

#3: Expand our team. We are currently recruiting Research Analysts; we are also thinking about whether hiring for more specialized roles may more efficiently increase our capacity. We hope to have at least eight full-time employees by the end of 2012.

Other goals include:

  • Regular updates on the charities we have moved the most funding to.
  • Putting some time into more deeply investigating research questions that are particularly important to us, such as the risks of population growth and the benefits of deworming. These sorts of investigations are along the lines of the 2011 investigation that ended with our finding major errors in cost-effectiveness estimates published by the World Health Organization, and rethinking how we use these sorts of figures.
  • Updating our research on disaster relief.
  • Raising any funding needed to finance the expansion of our team.
  • Revisiting our process for having our research subjected to formal external review.
  • Improving our process for tracking and processing donations (more).
  • Low-hanging fruit on the “outreach” front:
    • Improving our website to reflect some specific feedback we’ve recently gotten.
    • Further conference calls and community events to discuss our research (rather than simply writing about it).
    • Any other opportunities we see to make reasonable potential gains on the “outreach” front without excessive investment on our part.

Conference call to discuss GiveWell’s annual review and plan

(updated on: ) | by Elie

We’re still in the midst of publishing our annual review and plan to the blog (4 posts down, 2 to go).

We’re planning to hold a conference call on Thursday, March 1 at 8pm EST to answer any questions people have about our progress to date and plans for the coming year..

We plan to post a recording and transcript from the call, so if you can’t make it but have questions, please submit them via email or blog comment.

If you’d like to attend this call, please register your interest using this signup form.