Context: VillageReach focuses on health-system logistics in the developing-world. It was our top-rated charity from July 2009-November 2011, and GiveWell donors contributed over $2 million to it. These funds have primarily been directed towards a scale-up of VillageReach’s approach to health supplies in Mozambique. We have been posting regular updates on VillageReach’s progress.

We’ve just published an update on VillageReach’s progress. This post provides highlights from that update. In brief, progress has been discouraging on multiple fronts; VillageReach has made significant changes to its project plan and budget as a result; its resulting cost-effectiveness is likely to be substantially less strong than originally anticipated. All of these observations are made possible by VillageReach’s continuing transparency and commitment to collecting meaningful data. We always prefer discouraging observations to no observations.

VillageReach’s scale-up of its pilot project has hit multiple setbacks:

• Obstacles getting the program running. The program has run into significant obstacles in the two provinces in which VillageReach has been operating since late-2010/early-2011. The primary problem has been accessing funds from non-VillageReach sources to pay the health workers who implement the program. Lack of funding for these workers led them to stop implementing the program and no vaccine deliveries were made in either province for several months in the mid-to-late 2011. Work in two additional provinces has begun but is now significantly behind schedule. VillageReach has decided, going forward, to step in and provide funding itself when necessary; it reports that in the four months following this change, distributions occurred to all health centers on time.
• Increased expected costs. Actual costs have been higher than expected and VillageReach now projects roughly twice the costs it initially did. This is a primarily a function of (a) actual costs exceeding expectations and (b) increased expected costs due to VillageReach’s deciding to fill gaps, when necessary, left by other funders to ensure the project runs smoothly.
• Scaled-back future plans. Because of these problems, VillageReach has significantly scaled back its plans for the project, intending for the time being to work in 4 provinces rather than 8.

We now believe that VillageReach has room for more funding of approximately $1.5 million for 2012. This represents a change from our October 2011 statement that VillageReach did not have short-term room for more funding; the change is primarily due to the passage of time and some changes in the timing of expected expenditures rather than to any major change in VillageReach’s total projected costs.

We have not yet determined where VillageReach should rank on our top charities list. Its commitment to transparency and meaningful data collection - which have allowed the discouraging observations above - are major points in its favor, in our view, and we will likely attempt to ensure that it continues to have enough funding (something we believe we have ample time to do at the moment).

We are also more deeply examining the original evidence of effectiveness for VillageReach’s pilot project. Our standards for evidence continue to rise, and our re-examination has raised significant questions that we intend to pursue in the coming months. The deeper examination comes about because:

• Our research process has changed. In 2009 and 2010, VillageReach’s impact assessment was the best we had ever seen from a charity. Our research process has evolved, and there are now questions we would have asked of VillageReach in 2011 that we did not ask back in 2009-2010. For instance, we have always known that factors other than VillageReach’s work may have led to the increase in immunization coverage in Cabo Delgado between 2003 and 2008, but our investigation of this question was limited to (a) asking VillageReach whether other NGOs had significant operations that might have caused this and (b) looking at country-level immunization rates across Africa to see whether the change was part of a general trend. We now place more weight on other factors - particularly province-level government commitment - that could have led to this change. We intend to investigate this question and learn more about what else might have been happening in Cabo Delgado during the period of VillageReach’s pilot project.
• New information is available. This data includes: (A) health surveys released in 2010 that measure immunization rates in Mozambique. These surveys offer another source for data relevant to VillageReach’s project that was not available when we first assessed the pilot project. (B) In preparation for its scale-up, VillageReach returned to Cabo Delgado (the province in which it ran its pilot project) and conducted its own survey of immunization rates there. The newest survey raises questions about the impact of the pilot program and we will be conducting and publishing further analysis in the coming months. So far, we’ve conducted a re-analysis of VillageReach’s stockout and vaccination rate data.

As the first charity we directed significant funding to, VillageReach represents one of our best available learning opportunities. This is particularly true due to VillageReach’s continuing transparency and commitment to collecting meaningful data - qualities we believe are rare and outstanding. We intend to continue to take full advantage of this learning opportunity, even if it means publishing more discouraging news.

This is a guest post from John Beale, VillageReach’s Director, Strategic Development & Group Lead, Social Business Group.

VillageReach has been reviewed by GiveWell since mid-2009, and was recommended as its #1 recommendation for two years, until November 2011. In providing this updated review of our work in Mozambique, we see a need to explain the context for what we do.

Two key principles define the organization:

1. our mission is to save lives and improve health by increasing access to quality healthcare for remote, underserved communities; and
2. an emphasis on measurement, results and transparency in reporting its plans, challenges, failures and results.

The organization was established in 2000, and for years directed all of its efforts at a demonstration project in Mozambique. During the past three years, the scope and scale of VillageReach has increased dramatically: we are engaged in numerous new projects supported by our donors, foundations, USAID and other international development organizations. All of this additional work to improve health systems was borne out of the initial experience we gained in working on the initial Mozambique demonstration project.

Our interest in transparency is common to many of our donors and core to GiveWell’s mission. In our case, we see the need for new approaches to improving health systems in low-income countries that can yield greater impact at lower cost. Our focus is to strengthen existing health systems through improvements in health system capacity (supply chain, management and personnel training), information technology (to improve the quality of data reported from the field) and the creation of social businesses (that create shared infrastructure to improve transportation, energy supply and communications for rural health facilities and surrounding communities).

We focus on the last mile of the health system, where a lack of human resource capacity and infrastructure can limit the ability of the system to serve its communities. Instead of looking top-down, we see greater improvements being made possible by looking bottom-up. To highlight the benefits of this approach, it is clear to us that we must be objective in publicly documenting both the challenges and successes the approach records, and to draw attention to the need to allocate more global health resources to improving access to healthcare.

There are naturally risks in trying new approaches in search of significant rewards. We accept that there are risks and that we will not succeed all the time; but, we believe that through taking chances, sometimes making mistakes, learning and adjusting, we will achieve results for communities whose basic need for healthcare have been left unmet for too long. We believe many of our supporters share our vision.

For the Mozambique expansion, we tested the sustainability of the system with a new approach under which local governmental health authorities assume responsibility to operate and fund the distribution system. What we found, is that when government funding is erratic, the vaccines are not distributed. As a result, the availability and quality of healthcare becomes erratic, and ultimately the communities we seek to serve suffer. We evaluated the program, saw that there were too many months when vaccines were not getting distributed adequately, and quickly decided to intervene to achieve our primary goal. Because improving health outcomes is a higher priority for us than sustainability, we have agreed – at the cost of about $25,000 per province per year - to step in to fill gaps in government funding when necessary to ensure the distributions occur on a regular basis as we develop new approaches that can smooth the government’s funding streams. This sort of adaptation is important to achieving results, and since this change we’ve seen the vaccine distributions happening every month.

It’s also in our best interests to expose the challenges in what we are attempting to do as much as the successes. We’re looking for system change in global health: our effort is to lead by example and document the results. This system change perspective hasn’t been covered by GiveWell because the focus is more purely on transparency and measureable success.

VillageReach’s view is there is a lot of innovation in global health but insufficient effort to ensure the innovations reach the underserved. We’re engaged in improving health systems in order to save lives, but the broader goal is to see governments and other organizations doing this type of work because the need extends well beyond VillageReach’s modest resources. Some contributors clearly prefer to support only organizations they deem to be successful, but many of our supporters are interested in our approach because we’re trying to achieve something that’s difficult and unpredictable, but still worthy and representing a needed change.

We will continue to work to reach the underserved, documenting what worked, as well as what didn’t.

John Beale
Director, Strategic Development &
Group Lead, Social Business Group

Our current top-rated charity is VillageReach. In 2010, we directed over $1.1 million to it, which met its short-term funding needs (i.e., its needs for the next year or so).

VillageReach still has longer-term needs, and in the absence of other giving opportunities that we consider comparable, we’ve continued to feature it as #1 on our website. However, we’ve also been focusing most of our effort this year on identifying and investigating other potential top-rated charities, with the hope that we can refocus attention on an organization with shorter-term needs this December. (In general, the vast bulk of our impact on donations comes in December.) We believe that we will be able to do so. We don’t believe we’ll be able to recommend a giving opportunity as good as giving to VillageReach was last year, but given VillageReach’s lack of short-term (1-year) room for more funding, we do expect to have a different top recommendation by this December.

We haven’t been updating our rankings continuously; we prefer to do very deep investigations of top contenders, and aim for an all-at-once refresh in time for December. This is largely because we’ve continued to raise the bar for what it takes to become a top charity. For example, since we’ve found field visits to be useful, we now have a strong preference to avoid naming a charity “top-rated” before we’ve seen its work on the ground (for this reason, staff is currently split up between Malawi and India, visiting contender charities; we will post notes and pictures after we return and get the content approved by charities we’ve visited). More generally, we are looking to examine a charity from many different angles and have a high level of confidence before we start directing significant funds to it.

Bottom line - by December, we will have a new “top-rated” charity. This is not a “demotion” of VillageReach; rather, it reflects our success in directing enough funding to it to close its short-term gap.

When someone asks me what makes GiveWell different from other third-party charity evaluators, I often answer by listing all the things we’ve done in order to investigate our current top-rated charity, VillageReach.

• We’ve done extensive background research on international aid and argued for key takeaways such as the overall promise of international aid for donors, the particular promise of health aid, and, within health aid, the particular promise of immunization-focused programs (such as VillageReach’s).
• We’ve written up a comprehensive discussion of VillageReach’s different projects, the funders behind them, the funds allocated to them and what we know about their impact.
• We’ve examined VillageReach’s evidence for the effectiveness of its pilot project, listed and examined possible objections to this evidence, discussed these objections with staff and investigated staff’s responses.
• We’ve created multiple estimates of how much VillageReach’s activities cost per life saved.
• We’ve closely tracked VillageReach’s room for more funding and its expansion plan, and continually updated our analysis on these points as its plans and revenues have changed.
• We’ve spent several days with VillageReach staff in Mozambique (the site of its major project) and posted notes, reflections, pictures, and audio recordings of conversations.
• We’ve also visited VillageReach’s U.S. headquarters in Seattle, and we will soon be visiting the other site where VillageReach has a significant field presence, in Malawi.
• We’ve spoken to other funders of VillageReach, and to funders who have considered but ultimately declined funding VillageReach.
• Starting this year (following our moving over $1 million to VillageReach in 2010, most of it in late 2010), we’ve been posting quarterly updates on the progress of VillageReach’s work in Mozambique and the extent to which more donations are needed/warranted. We intend to continue these updates even if we stop recommending donations to VillageReach (which we expect to do shortly given the fact that it does not currently have a short-term funding need), since we want to learn how our recommendation turns out.

All in all, we’ve spent hundreds of hours examining VillageReach - yet we still feel very far from being “settled” on the question of how promising its activities are. Like any outstanding opportunity to do good, VillageReach’s work involves large and complex challenges. We’ll never have 100% of the relevant information or 100% certainty on its merits, but because we’ve recommended VillageReach so highly and moved over $1 million to it, it’s important to us that we do the best we can.

It isn’t realistic to do this kind of in-depth investigation for thousands (or even hundreds) of charities. We have to save our resources for the most promising charities if we want to have a reasonable level of confidence in our top recommendations. That means we take shortcuts on less promising charities, and we don’t put in the work it would take to distinguish between “worst,” “bad,” “mediocre” and “decent” groups - we’re laser-focused on the ones that we consider “best.”

Other independent charity evaluators tend to measure themselves by how many charities they rate. They exist largely for donors who already know where they want to give, and want a basic legitimacy check before they finalize the donation. To accommodate this goal, these other evaluators need to be far less thorough and more simplified than we are. That means - in our view - that they have no realistic chance of ever meaningfully rating impact, i.e., the degree to which a charity is succeeding at its mission.

GiveWell isn’t for everyone. Donors looking to check the charity they already want to give to are better off with other resources. But for donors who don’t already have a charity in mind and are looking to maximize their impact, we don’t know of any other group that provides a comparable product.

VillageReach is the first (hopefully not the last) charity for which (a) GiveWell has raised enough money to make a qualitative difference in VillageReach’s activities; (b) GiveWell has a clear enough picture of the room for more funding situation to be able to say with some confidence what that difference is.

We intend on providing periodic (aiming for quarterly) updates on VillageReach’s (a) progress in its Mozambique expansion, for which unrestricted funds have been sought; (b) revenue, projected expenses, and room for more funding.

We have posted the first such update to the Updates on VillageReach page. In a nutshell:

• Currently, VillageReach is active in Cabo Delgado, Niassa and Maputo, and VillageReach has provided a 6-month assessment and baseline vaccination coverage survey for Cabo Delgado.
• The overall expansion is substantially behind the schedule set out in July 2010 (on time in Cabo Delgado but 6-12 months behind for other provinces) for funding-related reasons (though revenue in 2010 closed the entire “stretch funding gap” for the year, the vast majority of that revenue was not received or anticipated until December).
• In Cabo Delgado, data collection appears to have improved, but there are not yet other signs of improvement in the health system’s performance. This is consistent with the expected trajectory of VillageReach’s 3-year involvement in a province, and we do not find it to be a major cause for concern.
• There has been a global supply disruption in a key vaccine that is expected to be resolved by the third quarter of 2011.
• VillageReach had a total of about $2 million in unrestricted revenue in 2010, of which a little over $1 million can be attributed to GiveWell’s recommendation.
• We believe that this $1 million has been key to VillageReach’s being able to plan to expand its program into the provinces of Gaza and Inhambane.
• We believe that VillageReach has a total funding gap of about $3.6 million remaining for this project. Taking into account expected revenue from non-GiveWell-related sources leaves about $1.4 million in room for more funding. These funds would allow it to proceed as quickly as possible with its full 8-province expansion plan.

The news is not all good. The global supply disruption is a major concern for us, and a reminder of how any attempt to help can be held up by circumstances beyond one’s control. VillageReach’s reports also reveal a problem with the flow of funds between a funder and the government, and make it quite clear that there has not yet been any noticeable improvement in health system efficiency (though there has been an improvement in data collection, and this is arguably what should be expected at this stage in the project).

Overall, though, we feel better about VillageReach than we did 3 months ago because we feel we continue to have a window into the true - not stylized - impact that GiveWell-related funds are having. In our view, one of the most problematic things about charity is that people purchase something but never get to fully understand what they purchased, so the feedback loops that lead to continuous improvement in most markets are broken. This is the first case we know of where individual donors can expect to get full and frequent updates on the impact of their donations (not just how their donations were spent), and is thus a first step toward improving those feedback loops.

An anonymous donor got some buzz a few weeks ago with its call on Innocentive for “extraordinary and unorthodox” philanthropic opportunities. It seeks a project that “holds the potential for a transformational impact,” “is unlikely to attract funding elsewhere due to its risky, unorthodox, and/or neglected profile,” and will be able to “attract additional capital from other sources” after the initial “catalytic” funding.

Perhaps ironically, this RFP sounded very familiar to me. In fact, it’s hard for me to see a big difference between it and the $100 million Gates Grand Challenges Explorations, “a unique initiative that supports innovative research of unorthodox ideas” in global health (though the Innocentive proposal above does not explicitly specify a sector, all three of its examples are in global health as well).

Speaking more informally, I’ve heard similar concepts emphasized by most major funders I’ve spoken with. Anyone who has dealt with major foundations should recognize the desire to find a completely new, revolutionary, neglected opportunity that just needs some seed funding to explode.

I do believe that the best opportunities are the under-funded ones. Yet I’m not sure that tiny, neglected innovations are the best places to look for these opportunities - precisely because that’s where all the major funders seem to be looking. I submit that the better place to look for neglected opportunities is the “valley of death” between proof of concept and large-scale rollout.

“Valley of death” is a term from both the investment and research communities. In both cases, broadly speaking, it refers to the difficulty of getting funding between the earliest stages of a project and the latest stages of a project, where investors have neither a sure thing nor a “lottery ticket” for the glory of being “first.”

I believe that the world of philanthropy suffers from its own “valley of death.”

Programs that have enough traction often aim for funding by the government (for example, the Nurse-Family Partnership program has had some success with this). Programs in their earliest stages appeal to foundations’ hunger for bragging rights. But what about VillageReach, which had a single successful “proof of concept” pilot project (initially funded by the Gates and Skoll foundations and the World Bank Development Award) and now needs transitional funding to roll out its model across Mozambique? Large foundations have, to date, declined to fund the rollout, in some cases specifically citing their preference for establishing models rather than rolling them out.

There’s no glory in funding the VillageReach rollout. VillageReach already has shown that what it’s doing has worked; nobody can claim to be brilliant for spotting it. And VillageReach doesn’t need help designing its program (this has been cited to me explicitly as a drawback from the perspective of some major funders).

But if it doesn’t get funded, it isn’t going to happen. If it does, it could change the world. It is, in fact, a fantastic opportunity.

If you asked me which funders are going to have the greatest impact over the next decade or so, I’d bet on the funders extraordinary and unorthodox enough to forget about being “extraordinary and unorthodox” - and instead put in a bid for the Smart Money Award.