What’s wrong with this “evidence of impact” for high-profile charities?
- The Millennium Villages project‘s main evidence of impact so far appears to be increasing crop yields by a factor of 4-15x in 3 villages. (H/T to the recent coverage of Millennium Villages on Aid Watch)
- A KickStart evaluation (available on our website (DOC)) reports that “Family/household incomes of 3,000 farmers increased by 100% within 24 months following acquisition of micro-irrigation pumps.”
- The Carter Center‘s agriculture program “provides the tools and means to help farmers in sub-Saharan Africa increase their crop yields through agricultural development, sometimes two- or even threefold.”
Among other possible problems, two major issues jump out:
1. No context on what “normal” variation in incomes looks like for poor farmers. Some years have more favorable weather – and local economic situations – than others. Enough that one year’s income or crop yield could be double another’s? 4x? 20x?
Unfortunately, one of the better pieces of “evidence” that jumps to mind is a 75-year-old novel, The Good Earth, whose farmer protagonist is comfortable one year and has literally zero income the next, for no other reason than the weather. If a given year’s yield were close enough to zero, the next year could be a huge increase (2x, 4x, 20x or more) simply by returning to normal.
I have seen little information on the local year-to-year volatility that poor farmers can experience, but I imagine that it (a) varies greatly from region to region and (b) could easily involve incomes falling and jumping by enormous amounts.
None of the above reports provide any context on this question, beyond qualitative statements about how favorable the rains were in each year examined. None of them employ any sort of “comparison group” of farmers (aside from one vague reference to “farms not using improved seeds and fertilizers” in the Malawi Millennium Village). Ultimately, none accomplish one of the most basic goals of an evaluation: giving a sense of how likely the “gains” they describe are to have arisen by pure chance.
With larger sample sizes, we might be able to use country-level volatility for context. But that brings me to the next problem.
2. We have no assurance that the described gains are representative, as opposed to “cherry-picked.”
All of the above organizations have reputations for consistent and thorough monitoring and evaluation, yet in all cases, we find ourselves looking for “impact” from a tiny subset of their projects.
- The Millennium Villages evaluation “includes results from the Sauri baseline survey, annual reports from Sauri and Koraro and crop-harvest data from Malawi.” How were these three chosen from among the 78 villages mentioned in the abstract?* The report doesn’t say.
- KickStart implies that it is constantly and consistently tracking impact, yet the report linked above discusses a single project over a single year.
- The Carter Center is somewhat explicit that its number is a “high-end” one (“sometimes two- or even threefold”). Examining the most recent annual report for its agriculture program shows a very small upward trend, if any, in crop yields across the countries in which it is working.
Some ways to produce more compelling evidence of impact
- Be clear about what is being measured and what is being published, and when. It seems to us that in this area, charity evaluation lags far behind clinical trials, which are constantly registered before they are complete so people can track their progress. (The Poverty Action Lab is similarly transparent with its own ongoing projects.)
- More sample size; more context; use of comparison groups. Discussed above.
- Look for more sustained improvements in people’s lives. One measure I find superior to straight “income” or “crop yields” is asset accumulation. A jump in income could be temporary; if someone upgrades their roof or sanitation, it’s likely that at least they expect the gain to be a real and lasting one. The Village Enterprise Fund’s evaluation is one of the better charity evaluations I’ve seen in the area of economic empowerment, partly because it focuses on standard of living rather than a simple measure of income.
*It’s possible that the yields mentioned are for “clusters” of villages rather than individual villages; there are only 12 clusters. However, the source documents available for Sauri and Koraro appear to be at the village rather than the cluster level, and the details of how the measurements were made are unclear.