The GiveWell Blog

Guest post from John Beale at VillageReach

(updated on: ) | by Guest

This is a guest post from John Beale, VillageReach’s Director, Strategic Development & Group Lead, Social Business Group.

VillageReach has been reviewed by GiveWell since mid-2009, and was recommended as its #1 recommendation for two years, until November 2011. In providing this updated review of our work in Mozambique, we see a need to explain the context for what we do.

Two key principles define the organization:

  1. our mission is to save lives and improve health by increasing access to quality healthcare for remote, underserved communities; and
  2. an emphasis on measurement, results and transparency in reporting its plans, challenges, failures and results.

The organization was established in 2000, and for years directed all of its efforts at a demonstration project in Mozambique. During the past three years, the scope and scale of VillageReach has increased dramatically: we are engaged in numerous new projects supported by our donors, foundations, USAID and other international development organizations. All of this additional work to improve health systems was borne out of the initial experience we gained in working on the initial Mozambique demonstration project.

Our interest in transparency is common to many of our donors and core to GiveWell’s mission. In our case, we see the need for new approaches to improving health systems in low-income countries that can yield greater impact at lower cost. Our focus is to strengthen existing health systems through improvements in health system capacity (supply chain, management and personnel training), information technology (to improve the quality of data reported from the field) and the creation of social businesses (that create shared infrastructure to improve transportation, energy supply and communications for rural health facilities and surrounding communities).

We focus on the last mile of the health system, where a lack of human resource capacity and infrastructure can limit the ability of the system to serve its communities. Instead of looking top-down, we see greater improvements being made possible by looking bottom-up. To highlight the benefits of this approach, it is clear to us that we must be objective in publicly documenting both the challenges and successes the approach records, and to draw attention to the need to allocate more global health resources to improving access to healthcare.

There are naturally risks in trying new approaches in search of significant rewards. We accept that there are risks and that we will not succeed all the time; but, we believe that through taking chances, sometimes making mistakes, learning and adjusting, we will achieve results for communities whose basic need for healthcare have been left unmet for too long. We believe many of our supporters share our vision.

For the Mozambique expansion, we tested the sustainability of the system with a new approach under which local governmental health authorities assume responsibility to operate and fund the distribution system. What we found, is that when government funding is erratic, the vaccines are not distributed. As a result, the availability and quality of healthcare becomes erratic, and ultimately the communities we seek to serve suffer. We evaluated the program, saw that there were too many months when vaccines were not getting distributed adequately, and quickly decided to intervene to achieve our primary goal. Because improving health outcomes is a higher priority for us than sustainability, we have agreed – at the cost of about $25,000 per province per year – to step in to fill gaps in government funding when necessary to ensure the distributions occur on a regular basis as we develop new approaches that can smooth the government’s funding streams. This sort of adaptation is important to achieving results, and since this change we’ve seen the vaccine distributions happening every month.

It’s also in our best interests to expose the challenges in what we are attempting to do as much as the successes. We’re looking for system change in global health: our effort is to lead by example and document the results. This system change perspective hasn’t been covered by GiveWell because the focus is more purely on transparency and measureable success.

VillageReach’s view is there is a lot of innovation in global health but insufficient effort to ensure the innovations reach the underserved. We’re engaged in improving health systems in order to save lives, but the broader goal is to see governments and other organizations doing this type of work because the need extends well beyond VillageReach’s modest resources. Some contributors clearly prefer to support only organizations they deem to be successful, but many of our supporters are interested in our approach because we’re trying to achieve something that’s difficult and unpredictable, but still worthy and representing a needed change.

We will continue to work to reach the underserved, documenting what worked, as well as what didn’t.

John Beale
Director, Strategic Development &
Group Lead, Social Business Group

Update on the Against Malaria Foundation: Our current #1-ranked charity

(updated on: ) | by Elie

Highlights from our update of AMF follow. For those who want more information, please see our full AMF update.

Background

Historically, AMF had distributed bednets in the following way: on-the-ground organizations applied to AMF for nets; after AMF reviewed and approved proposals, it would ship nets to the charity; finally, AMF would check in to see whether the nets had been distributed and were being used.

In early 2011, AMF changed its model. It received significant funding, which allowed it to proactively identify opportunities for net distributions rather than reacting to charities’ requests. In mid-2011, it decided to provide about 250,000 nets (at a cost of a little over $1 million to AMF) to the Ntcheu district in Malawi as part of a national net distribution.

Note: In addition to the costs incurred by AMF, Concern Universal, AMF’s distribution partner in Malawi also incurred costs. We estimated these costs in our review, but are currently working to update our estimate of Concern’s costs because we believe that we underestimated Concern Universal’s costs in our original AMF review. We don’t believe our estimate of total costs per net will change significantly.

The Malawi distribution

In October 2011, three GiveWell staff members visited the Ntcheu district while pre-distribution activities were taking place (such as surveying local households to determine who needed nets and how many each household needed). Concern Universal, the organization distributing AMF-provided nets in Ntcheu, started the distribution in mid-December and now (as of mid-March) Concern Universal has reported that it had distributed 242,745 nets to households in the district, out of the 251,720 nets provided by AMF. For its final distribution location in late March/early April, Concern Universal requires another 16,574 nets, which AMF is sending. (More information in our full, detailed update.)

Concern has posted weekly reports on its progress on AMF’s website, which not only share data on weekly nets distributed but also share problems identified during distributions. Concern reports highlight a number of instances of attempted theft and fraud by the health workers (HSAs) Concern has employed to distribute nets. In one case, Concern believes that an HSA fabricated the existence of two villages in order to steal the nets. Neither we nor AMF are surprised by attempted fraud, and we are glad that Concern’s process has identified such instances.

We cannot be sure whether Concern has identified all instances of malfeasance. That said, we believe that Concern’s process for identifying attempted fraud (along with AMF’s oversight of Concern and our monitoring of the entire process) is robust and therefore, we would be aware of significant problems. (More information in our full, detailed update.)

AMF’s 2012 plans

  • AMF currently has access to approximately $3.5 million in received (or committed and soon to be received) funding. Approximately $2.3 million of this comes from GiveWell-directed donors.
  • Malawi is currently in the midst of a national net distribution, which has been delayed. (AMF-funded distributions are not delayed; the rest of the national distribution is.) Rob Mather told us that Malawi estimates it needs 5.8 million nets, of which it has access to 5.2 million (4.7 million from the Global Fund and approximately 500,000 from the President’s Malaria Initiative). It therefore has a gap of 600,000 nets, and AMF is in discussions with the NMCP to see if AMF will provide these nets.
  • Rob Mather has told us that he is not yet ready to provide these nets. He (a) has not yet seen sufficient data to convince him that there is, indeed, a gap of 600,000 nets, and (b) he is not yet convinced that AMF funding, as opposed to funding from other partners (e.g., Global Fund), is needed. In the event that Malawi does provide this information, AMF would provide additional nets for the Malawi distribution. AMF has therefore reserved $2.5 million of its funding while it assesses the need for nets in Malawi.
  • AMF will likely reach a decision about Malawi in either the next few weeks – if the decision is not to provide further nets for Malawi – or in the next 3-4 months if it looks like funding further nets is possible. Three to four months is the timeframe required to collect data from the National Malaria Control Program and from the field to determine the net gap. Coordination with the Global Fund and President’s Malaria Initiative is also required. In the event of a positive outcome, Mr. Mather expects that the nets would be distributed before the rainy season in November. In the event that AMF decides not to contribute further to the Malawi effort, it believes that there are strong needs for nets exist in campaigns in Ghana, Mali, Togo and several other countries.
  • One of AMF’s broad goals in its interactions with the Global Fund and the National Malaria Control Program in Malawi is improving the way that net distributions are conducted. Mr. Mather believes the AMF/Concern Universal approach in the Ntcheu district distribution follows best practice for pre-distribution data collection, distribution oversight, and post-distribution monitoring (described in our AMF review). As AMF’s influence increases (due to increased funding ability), it hopes to influence other players involved in bednet distribution to adopt the Ntcheu distribution as a model.

Expected future revenue

AMF told us that it has no current commitments for significant future revenue, but is in discussions with donors who could potentially make 7-figure donations. We are currently comfortable with AMF receiving up to ~$15 million over the next year. This figure would be about half of what would be needed to close the net gaps for Malawi, Mali and Ghana (according to our most conservative estimates published in our coverage analysis), and would constitute a significant “step up” for AMF’s size – a somewhat risky but overall strong opportunity for donors. We would have to revisit the question of AMF’s room for more funding if it were to raise more than $15 million.

KIPP Houston update

(updated on: ) | by Elie

In 2011, we recommended KIPP Houston as our top recommendation for donors interested in giving to a United States-focused charity. Our recommendation was based on KIPP’s strong track record and KIPP Houston’s funding gap (caused by education funding cuts passed by the Texas state legislature). We recently checked in with KIPP Houston to update our view of its situation.

Our hypothesis was that KIPP Houston needed more funds than it could raise, and our check-in tried to test that hypothesis by seeing whether cuts had been made over the last 12 months.

One note: KIPP Houston is a “standout organization,” not one of our two top charities. We have not put nearly the effort into understanding KIPP Houston as with our highest rated charities. We continue to put the vast majority of our effort into finding and understanding charities that will receive our highest ratings.

In July 2011, KIPP Houston sent us a list of possible changes it might implement in its 2011-12 school-year to balance its budget. (See our blog post from July 2011.) Collectively, these cuts totaled $4.4 million. In addition, it aimed to save an additional $400,000 by identifying “operational inefficiencies” and to increase revenues by increasing its fundraising goal by $1.7 million and utilizing investment income of $1.2 million for operational purposes instead of capital investment.

In February 2012, KIPP Houston sent us a summary of the changes it did implement for its 2011-12 school year:

  • Increased fundraising: KIPP Houston has met its goal of increasing fundraising by $1.7 million for the fiscal year ending June 30, 2012, relative to their original plan. Note that after achieving this goal earlier than anticipated, KIPP’s fundraising team switched to fundraising for the 2012-13 school year, so the document they sent us shows that it exactly met its 2011-12 fundraising goals.
  • KIPP Houston saved approximately $5.7 million via spending cuts.
    • $3.7 million of this came via reductions in “KIPP Unique” expenses. Our understanding is that the primary “KIPP unique” changes were: (1) elimination of field lessons, such as trips to Washington, D.C.; (2) reduction of school hours (though KIPP still maintains a higher classroom hour total than the local districts); (3) changes to Saturday school; and (4) summer stipend reduction due to reducing the extended year calendar.
    • $930,000 came from reductions in administrative expenses; $400,000 came from reducing employee compensation (freezing salaries and increasing employee contributions to health plans); $370,000 came from “campus savings” (such as eliminating receptionist positions at schools that could get by without them); and $340,000 came from employing teaching aides as opposed to teaching fellows (our limited understanding is that teaching aides have lower qualifications and experience compared to teaching fellows).

It is not impossible to argue that these cuts should/would have been made regardless of the funding situation. (For example, the cuts to employee compensation may have been driven by the ongoing recession and the state funding reduction causing a reduction in statewide labor costs, rather than being evidence of a KIPP-specific shortfall.) However, overall, we feel that the update on what KIPP Houston has done over the past 12 months supports our earlier hypothesis: that KIPP Houston does face an ongoing funding shortfall, and KIPP Houston has followed through on the plans it shared with us almost a year ago by making significant cuts to its programs.

KIPP Houston has told us that its next update will come during the summer when it approves its 2012-13 school-year budget. We intend to post another update once we’ve reviewed that document.

Update: As of November 2012, we have decided to discontinue the practice of publishing a list of “non-top-rated standout organizations,” so we no longer plan to publish an additional update on KIPP Houston (more here).

The worst killer of invisible children is not Joseph Kony

(updated on: ) | by Holden


Joseph Kony is evil and should be stopped. He has allegedly abducted 30,000 children in his long military campaign.

Malaria kills hundreds of thousands of children every year.

Joseph Kony has committed atrocities that make me furious. But malaria makes me angrier. Why? Because malaria deaths really do happen just because Americans don’t care enough.

The popular Kony 2012 video argues that Kony can be stopped just by making him famous. That might be true. But it might not be. We generally don’t focus our research on military interventions (more below on why we don’t), so we have little knowledge of the situation, but one thing that’s clear is that things aren’t as simple as Invisible Children is making them sound.

  • First off, American involvement with pursuing Joseph Kony did not start with Invisible Children’s campaign. It has a long history.
  • LSE faculty, writing in Foreign Affairs, state that “the LRA is, in fact, a relatively small player in all of this — as much a symptom as a cause of the endemic violence. If Kony is removed, LRA fighters will join other groups or act independently.”
  • Some are concerned that pursuing Kony could do more harm than good – not just by diverting resources and attention from more important problems, but via support to the Ugandan army and via provoking possible retaliation.

But we can stop a lot of malaria if we can just care about it more. Insecticide-treated nets drastically reduce malaria; they’ve been tested time and again; they’ve worked on a small scale and on a large scale; they’re safe, they’re proven, they’re cheap and they save lives. (Details at our investigation of insecticide-treated nets.)

The same can likely be said for some other malaria control interventions. The missing ingredient in malaria control? More money – it’s that simple. And you don’t have to lobby Washington to make that happen (though you can); you can also just write a check or get your friends to do so.

Africa has many problems that are like malaria: devastating, but also preventable with donor dollars. (Another one: parasite infections.) Raising awareness of these problems would, I believe, do far more good than raising awareness of Joseph Kony.

So why is Invisible Children focused on Kony?
I don’t know exactly how Invisible Children picked its cause, but I have a guess. Invisible Children is excellent at filmmaking and Joseph Kony – while not the worst problem in Africa – is probably the best movie-style villain. The atrocities he commits are unspeakable and emotionally gripping; he is a person, so we can identify with him enough just to truly hate him. He is a face of evil.

Individuals can change the world (and they’re already doing it)
Invisible Children is right when it says that the power of individuals is increasing. As a donor, voter and social networker, you have power. With that power comes responsibility. You have to decide whether you’re going to focus on the most important problems for Africans or the most cinematically apt problems for Americans. And whether you’re going to use your power to intervene in a complex, disputed situation that you don’t have the context to fully understand, or in a simple situation where all humanitarians really do agree.

If Invisible Children has inspired you to care more about Africans, that’s great news. We hope you’ll take that inspiration, passion and emotion and take it to the next level. If you can learn how to give as effectively as possible, you’ll join a worldwide community of individuals that is giving millions of dollars and saving children’s lives every year. GiveWell tries to be a crucial cog in that community by putting thousands of hours into research identifying the best charities available.

More errors in widely-cited figures: The case of mothers2mothers

(updated on: ) | by Holden

Note: mothers2mothers has provided a response to this post that can be viewed below.

Summary: mothers2mothers, a well-respected group that focuses on HIV programs in Africa, published figures on its website that we have recently come to believe are erroneous. We feel this finding is important not because of what it says about mothers2mothers, but because of what it says about the wider community that has been funding, awarding, and citing mothers2mothers’s figures.

mothers2mothers (m2m) is a group focused on prevention of mother-to-child transmission of HIV (PMTCT). It has as many awards – and major funders – as any nonprofit (its size) that we’ve seen.

Its published figures suggest that it serves a huge number of women – specifically, that it accounts for around 20% of all women on PMTCT in sub-Saharan Africa. Yet when we performed a simple check on its figures, we saw major anomalies:

  • In some countries, m2m’s stated number of women served exceeds the national total for women on PMTCT, from public UNITAIDS data. (For example, in Swaziland m2m reports ~25,000 mothers served; but UNITAIDS reports a total of ~10,000 Swaziland women on PMTCT for the same year.)
  • In other countries, m2m’s stated number of women served does not exceed the national total, but it is enough to account for 50-100% of it. However, looking at the trends in national data, one does not see an increase after m2m’s entry into the country. (Charts below.)

After corresponding with mothers2mothers, we believe that the anomalies we’ve seen are chiefly explained by flaws in mothers2mothers’s data.

mothers2mothers has told us it is now considering adding a disclaimer to its website. It has also provided a response, which is included below.

We feel this finding is important not because of what it says about mothers2mothers, but because of what it says about the wider community that has been funding, awarding, and citing mothers2mothers’s figures.

There is a lot more to the value of a nonprofit than the quality of its data, and there are a lot of questions that a good investigation ought to ask besides whether the numbers add up. We certainly don’t think the anomalies we’ve found show that m2m isn’t doing great work, or that its support and awards are undeserved (and we are still considering the possibility of further investigating m2m as a potential GiveWell-recommended group). Still, seeing this sort of problem from an organization that gets as much attention as m2m seems significant. It’s another piece of evidence that the philanthropic world – including many of the largest and best-resourced funders – is not asking all of the critical questions that it could be asking.

The implausible implications of mothers2mothers’ figures, and how we came across them

We’ve long found mothers2mothers to be potentially promising because of its focus on antiretrovirals for the prevention of mother-to-child transmission (PMTCT) of HIV, which we consider a priority intervention. We examined mothers2mothers in 2009, and concluded that the evidence of effectiveness it was pointing to didn’t meet our standards (details). In April of 2011, our interest was rekindled when a funder of mothers2mothers told us that mothers2mothers treats a substantial percentage of all the women needing PMTCT worldwide.

Our immediate reaction was: “If that’s true, we ought to be able to see the case for mothers2mothers’s impact in country-level data – perhaps via major improvements in country-level data following mothers2mothers’s entry into a country – and a convincing impact at that high a level would be impressive and compelling.” So we decided to collect the relevant data and see what sort of picture it presented.

Philanthropy’s success stories

(updated on: ) | by Holden

[Added August 27, 2014: GiveWell Labs is now known as the Open Philanthropy Project.]

The first item on the agenda for GiveWell Labs is to get the broadest possible view of philanthropy: its history (what’s gone well, what hasn’t) and its current state (what’s being funded and what isn’t).

On the history front, I’ve found very little of interest. Most books on philanthropy are how-to’s rather than histories, and most of the histories don’t appear to focus on listing specific cases where foundations had (or failed to have) impact. One exception is the Casebook for The Foundation: A Great American Secret, which lists and discusses “100 of the highest-achieving foundation initiatives” since 1900.

I thoroughly examined this volume, and collected some basic notes into a spreadsheet. My reflections follow. In brief, I felt there were some very strong cases here, particularly in the area of medical research, and I was surprised by philanthropy’s history of being active in shaping various graduate education programs. At the same time, I thought the Casebook’s history had important shortcomings – in particular, not putting successes in context along with failures – and I see a lot of room for improvement in the amount of information available about the history of philanthropic successes and failures.

Note that I am not well-informed about most of the cases discussed here and have relied on the Casebook for my information. The notes that follow are only loose impressions and are not backed by the sort of evidence that we usually seek (even for blog posts).

Philanthropy has some extremely impressive accomplishments. Among other things, foundations have been (in my view) reasonably credited for leading the way on building schools and hospitals in the rural Southern U.S., piloting the shoulder line on U.S. roads, successfully advocating for federal legislation in areas including health care for the homeless and nuclear deproliferation, the research that led to the Green Revolution, and many major advances in medical research (including the first combination drug therapy for AIDS and the development of the pap smear). There are many other projects that sound like they may have been impactful, but which the Casebook doesn’t give enough context on for me to have a strong view.

The most impressive cases (in my view) are mostly the earlier ones. Though the Casebook focuses on more recent philanthropy (78 of its 100 cases are post-1950), 9 of the 14 cases I found most impressive are pre-1950 (and a 10th is from 1952).

A possible explanation is that the space of doing good has become more crowded over time. For example, note that