Note: mothers2mothers has provided a response to this post that can be viewed below.
Summary: mothers2mothers, a well-respected group that focuses on HIV programs in Africa, published figures on its website that we have recently come to believe are erroneous. We feel this finding is important not because of what it says about mothers2mothers, but because of what it says about the wider community that has been funding, awarding, and citing mothers2mothers’s figures.
mothers2mothers (m2m) is a group focused on prevention of mother-to-child transmission of HIV (PMTCT). It has as many awards – and major funders – as any nonprofit (its size) that we’ve seen.
Its published figures suggest that it serves a huge number of women – specifically, that it accounts for around 20% of all women on PMTCT in sub-Saharan Africa. Yet when we performed a simple check on its figures, we saw major anomalies:
- In some countries, m2m’s stated number of women served exceeds the national total for women on PMTCT, from public UNITAIDS data. (For example, in Swaziland m2m reports ~25,000 mothers served; but UNITAIDS reports a total of ~10,000 Swaziland women on PMTCT for the same year.)
- In other countries, m2m’s stated number of women served does not exceed the national total, but it is enough to account for 50-100% of it. However, looking at the trends in national data, one does not see an increase after m2m’s entry into the country. (Charts below.)
After corresponding with mothers2mothers, we believe that the anomalies we’ve seen are chiefly explained by flaws in mothers2mothers’s data.
mothers2mothers has told us it is now considering adding a disclaimer to its website. It has also provided a response, which is included below.
We feel this finding is important not because of what it says about mothers2mothers, but because of what it says about the wider community that has been funding, awarding, and citing mothers2mothers’s figures.
There is a lot more to the value of a nonprofit than the quality of its data, and there are a lot of questions that a good investigation ought to ask besides whether the numbers add up. We certainly don’t think the anomalies we’ve found show that m2m isn’t doing great work, or that its support and awards are undeserved (and we are still considering the possibility of further investigating m2m as a potential GiveWell-recommended group). Still, seeing this sort of problem from an organization that gets as much attention as m2m seems significant. It’s another piece of evidence that the philanthropic world – including many of the largest and best-resourced funders – is not asking all of the critical questions that it could be asking.
The implausible implications of mothers2mothers’ figures, and how we came across them
We’ve long found mothers2mothers to be potentially promising because of its focus on antiretrovirals for the prevention of mother-to-child transmission (PMTCT) of HIV, which we consider a priority intervention. We examined mothers2mothers in 2009, and concluded that the evidence of effectiveness it was pointing to didn’t meet our standards (details). In April of 2011, our interest was rekindled when a funder of mothers2mothers told us that mothers2mothers treats a substantial percentage of all the women needing PMTCT worldwide.
Our immediate reaction was: “If that’s true, we ought to be able to see the case for mothers2mothers’s impact in country-level data – perhaps via major improvements in country-level data following mothers2mothers’s entry into a country – and a convincing impact at that high a level would be impressive and compelling.” So we decided to collect the relevant data and see what sort of picture it presented.
The data was publicly accessible and easy to find. We got data for “total number of women on PMTCT” from World Health Organization reports, and data for “number of women served by mothers2mothers” from mothers2mothers annual reports. The data was consistent with the statement the funder had made. (Details at our spreadsheet.)
We charted the data for each country in which m2m works, to get a sense of whether there were major country-level improvements taking place when m2m entered a country.
In some countries, the “number of women served by mothers2mothers” exceeds the “total number of women on PMTCT,” which doesn’t seem logical. The most extreme example is Swaziland:
Full details are at our review of mothers2mothers.
We didn’t see the country-level data point to a visible impact of mothers2mothers; we did see an apparent contradiction between official country-level data and mothers2mothers data.
We contacted the mothers2mothers funder who had first piqued our interest, who in turn connected us with mothers2mothers. After some back and forth, mothers2mothers sent an explanation for the above anomalies. After reviewing the explanation, our view is that the anomalies pointed to above are best explained by inaccuracies of mothers2mothers data (in particular, double-counting participants).
Why we shouldn’t have been the first to notice these issues
mothers2mothers has a very high profile.
- According to its current website, mothers2mothers receives substantial funding from USAID (the main agency for U.S. government aid), CIDA (Canada government aid), UNICEF, the Gates Foundation, the U.S. Centers for Disease Control and Prevention, the Clinton Health Access Initiative, the Skoll Foundation, the Wallace Global Fund, and the Michael and Susan Dell Foundation – all major, staffed funders.
- mothers2mothers and its founders have received numerous awards and honors, including the Schwab Foundation’s award for outstanding regional social entrepreneurs 2009, the Global Health Council’s Best Practices in Global Health Award (2010), and the Presidential Citizens Medal, given by the U.S. President in person (2008). It has spoken before Congress, been visited by U.S. senators and Michelle Obama, and been hosted by Laura Bush at the White House.
In addition, it appears that the “number of women served” figures that we find problematic have long been front and center.
- They are currently very prominently displayed on the mothers2mothers home page; they are prominent in the last several years of annual reports.
- Figures relating to the “number of clients served” have been cited by UNICEF, the Skoll Foundation and the Schwab Foundation. (Note that funders generally publish very little about their grantees, so there is not much to work with when determining whether mothers2mothers funders have considered these figures important.)
Any of mothers2mothers’s impressive list of supporters could have asked the same question we did: “Given that mothers2mothers’s clients-served figures are so high, can we see the impact of mothers2mothers in country-level data?” We believe that if they had asked that question, they would have arrived at our answer: the clients-served figures are flawed and overstate mothers2mothers’s scale. We believe that none of mothers2mothers’s supporters asked the question, because (based on our exchange with mothers2mothers) we believe we are the first to notice the apparent flaws.
Bottom line
The fact that mothers2mothers has flawed data does not mean it isn’t doing good work, or that it doesn’t deserve the support and accolades it has gotten. Maintaining good data in this sort of work is likely to be a substantial challenge, and is far from being the only – or most important – aspect of tackling the challenge mothers2mothers seeks to tackle.
This isn’t a post about mothers2mothers or the work that it does. It’s a post about the work that mothers2mothers’s supporters – which include several of the big names in philanthropy – apparently haven’t done: critically questioning mothers2mothers’s headline “clients served” figures, and looking opportunistically for data that might paint a picture of mothers2mothers’s impact.
Because major funders share so little about what goes into their thinking, we rarely get an opportunity to glimpse what sorts of questions do – and don’t – factor into their processes. There may be many important questions they are asking that we are failing to ask (and we wish they would share more information that would help us determine whether this is the case). But we take this episode as evidence that there are at least some instances of important questions that we ask, that these funders apparently don’t.
Response from mothers2mothers
Following an email exchange, and after reviewing a draft of this post, mothers2mothers provided the following response:
Despite dialogue we have had with GiveWell in recent months, we disagree with many of the assumptions they continue to make and the conclusions they have drawn, both about our organization and the generous donors and partners who support our efforts. We would be pleased to respond personally to questions readers may have about opinions in this article or issues GiveWell is raising. Please feel free to contact us at: MitchellBesser@m2m.org.
Mitchell Besser, Founder and Medical Director, mothers2mothers
Comments
Could you disclose your analysis of m2m’s figures that lead to suspect their numbers rather than the official ones? Without fully disclosing your analysis, m2m’s current funders will discount it in favor of m2m’s explanation (which they’ve suspiciously chosen to keep private).
I’m unsure of why you made this post. You’ve broken a big norm by criticizing another “good guy” in public, potentially starving them of funding. Personally, I admire GiveWell for it. My concern is that you may be curtailing your effectiveness. Charities may become more hesitant to talk to you if you publicly shame them. You may spend too much time defending against the inevitable counterattacks.
I was unclear from the post as to how the discrepancy could have occurred too. I’d assume that when WHO is computing totals it would sum those served by different organizations, but obviously the anomalies couldn’t exist if that were the case.
Samuel – thanks for the comments. We did discuss the data with m2m in order to try to ensure that our analysis was painting an accurate picture. They asked that we keep that discussion confidential and we’ve complied, so we’re not able to say more about why we believe the errors are more likely in m2m’s data than the WHO’s. In general, I think the prior should be that a large international organization like the WHO will have more accurate data than any particular NGO.
We conducted the above analysis based on publicly available data before talking to m2m. Our policy (discussed more here) is that we won’t publish private documents or conversations with charities without permission and we won’t penalize a charity publicly for anything they reveal to us in a private setting. We have this policy because, as you point out, we don’t want to create an incentive for charities not to talk to us.
Ron – mothers2mothers does not directly administer PTMCT (more in our review); rather it recruits and trains mothers who have received PMTCT to help new mothers understand the benefits of PMTCT so that they adhere to the prescribed regimen.
Our understanding is that the WHO obtains its data from national HIV programs which presumably collect data from the healthcare facilities themselves.
Comments are closed.