Context: VillageReach focuses on health-system logistics in the developing-world. It was our top-rated charity from July 2009-November 2011, and GiveWell donors contributed over $2 million to it. These funds have primarily been directed towards a scale-up of VillageReach’s approach to health supplies in Mozambique. We have been posting regular updates on VillageReach’s progress.

We’ve just published an update on VillageReach’s progress. This post provides highlights from that update. In brief, progress has been discouraging on multiple fronts; VillageReach has made significant changes to its project plan and budget as a result; its resulting cost-effectiveness is likely to be substantially less strong than originally anticipated. All of these observations are made possible by VillageReach’s continuing transparency and commitment to collecting meaningful data. We always prefer discouraging observations to no observations.

VillageReach’s scale-up of its pilot project has hit multiple setbacks:

• Obstacles getting the program running. The program has run into significant obstacles in the two provinces in which VillageReach has been operating since late-2010/early-2011. The primary problem has been accessing funds from non-VillageReach sources to pay the health workers who implement the program. Lack of funding for these workers led them to stop implementing the program and no vaccine deliveries were made in either province for several months in the mid-to-late 2011. Work in two additional provinces has begun but is now significantly behind schedule. VillageReach has decided, going forward, to step in and provide funding itself when necessary; it reports that in the four months following this change, distributions occurred to all health centers on time.
• Increased expected costs. Actual costs have been higher than expected and VillageReach now projects roughly twice the costs it initially did. This is a primarily a function of (a) actual costs exceeding expectations and (b) increased expected costs due to VillageReach’s deciding to fill gaps, when necessary, left by other funders to ensure the project runs smoothly.
• Scaled-back future plans. Because of these problems, VillageReach has significantly scaled back its plans for the project, intending for the time being to work in 4 provinces rather than 8.

We now believe that VillageReach has room for more funding of approximately $1.5 million for 2012. This represents a change from our October 2011 statement that VillageReach did not have short-term room for more funding; the change is primarily due to the passage of time and some changes in the timing of expected expenditures rather than to any major change in VillageReach’s total projected costs.

We have not yet determined where VillageReach should rank on our top charities list. Its commitment to transparency and meaningful data collection – which have allowed the discouraging observations above – are major points in its favor, in our view, and we will likely attempt to ensure that it continues to have enough funding (something we believe we have ample time to do at the moment).

We are also more deeply examining the original evidence of effectiveness for VillageReach’s pilot project. Our standards for evidence continue to rise, and our re-examination has raised significant questions that we intend to pursue in the coming months. The deeper examination comes about because:

• Our research process has changed. In 2009 and 2010, VillageReach’s impact assessment was the best we had ever seen from a charity. Our research process has evolved, and there are now questions we would have asked of VillageReach in 2011 that we did not ask back in 2009-2010. For instance, we have always known that factors other than VillageReach’s work may have led to the increase in immunization coverage in Cabo Delgado between 2003 and 2008, but our investigation of this question was limited to (a) asking VillageReach whether other NGOs had significant operations that might have caused this and (b) looking at country-level immunization rates across Africa to see whether the change was part of a general trend. We now place more weight on other factors – particularly province-level government commitment – that could have led to this change. We intend to investigate this question and learn more about what else might have been happening in Cabo Delgado during the period of VillageReach’s pilot project.
• New information is available. This data includes: (A) health surveys released in 2010 that measure immunization rates in Mozambique. These surveys offer another source for data relevant to VillageReach’s project that was not available when we first assessed the pilot project. (B) In preparation for its scale-up, VillageReach returned to Cabo Delgado (the province in which it ran its pilot project) and conducted its own survey of immunization rates there. The newest survey raises questions about the impact of the pilot program and we will be conducting and publishing further analysis in the coming months. So far, we’ve conducted a re-analysis of VillageReach’s stockout and vaccination rate data.

As the first charity we directed significant funding to, VillageReach represents one of our best available learning opportunities. This is particularly true due to VillageReach’s continuing transparency and commitment to collecting meaningful data – qualities we believe are rare and outstanding. We intend to continue to take full advantage of this learning opportunity, even if it means publishing more discouraging news.