The GiveWell Blog

Agriculture charity evaluation: Incomes boosted are not the same as lives changed

What’s wrong with this “evidence of impact” for high-profile charities?

Among other possible problems, two major issues jump out:

1. No context on what “normal” variation in incomes looks like for poor farmers. Some years have more favorable weather – and local economic situations – than others. Enough that one year’s income or crop yield could be double another’s? 4x? 20x?

Unfortunately, one of the better pieces of “evidence” that jumps to mind is a 75-year-old novel, The Good Earth, whose farmer protagonist is comfortable one year and has literally zero income the next, for no other reason than the weather. If a given year’s yield were close enough to zero, the next year could be a huge increase (2x, 4x, 20x or more) simply by returning to normal.

I have seen little information on the local year-to-year volatility that poor farmers can experience, but I imagine that it (a) varies greatly from region to region and (b) could easily involve incomes falling and jumping by enormous amounts.

None of the above reports provide any context on this question, beyond qualitative statements about how favorable the rains were in each year examined. None of them employ any sort of “comparison group” of farmers (aside from one vague reference to “farms not using improved seeds and fertilizers” in the Malawi Millennium Village). Ultimately, none accomplish one of the most basic goals of an evaluation: giving a sense of how likely the “gains” they describe are to have arisen by pure chance.

With larger sample sizes, we might be able to use country-level volatility for context. But that brings me to the next problem.

2. We have no assurance that the described gains are representative, as opposed to “cherry-picked.”

All of the above organizations have reputations for consistent and thorough monitoring and evaluation, yet in all cases, we find ourselves looking for “impact” from a tiny subset of their projects.

Some ways to produce more compelling evidence of impact

  1. Be clear about what is being measured and what is being published, and when. It seems to us that in this area, charity evaluation lags far behind clinical trials, which are constantly registered before they are complete so people can track their progress. (The Poverty Action Lab is similarly transparent with its own ongoing projects.)
  2. More sample size; more context; use of comparison groups. Discussed above.
  3. Look for more sustained improvements in people’s lives. One measure I find superior to straight “income” or “crop yields” is asset accumulation. A jump in income could be temporary; if someone upgrades their roof or sanitation, it’s likely that at least they expect the gain to be a real and lasting one. The Village Enterprise Fund’s evaluation is one of the better charity evaluations I’ve seen in the area of economic empowerment, partly because it focuses on standard of living rather than a simple measure of income.

*It’s possible that the yields mentioned are for “clusters” of villages rather than individual villages; there are only 12 clusters. However, the source documents available for Sauri and Koraro appear to be at the village rather than the cluster level, and the details of how the measurements were made are unclear.

Comments

  • Ian Turner on October 21, 2009 at 1:24 pm said:

    Regarding the Millenium Villages, it should be noted that the project is adding roughly $100 (*) per person per year to villages’ local economy. Note that GDP per capita in these villages would be somewhere around $150 (*) per year, again raising the specter of how these changes would compare to just handing out cash.

    (*) Note that figures named in this comment are measured at market exchange rates, not PPP.

  • Ian Turner on October 21, 2009 at 1:32 pm said:

    Also, regarding sustainability, there is really only one measure that you can use to find out if a project is making a sustainable difference: Withdraw the funds and see if the subject thrive or wither. Even checking to see if roofs are built doesn’t tell you whether or not they will be maintained.

    These facts are unfortunate, partly because withdrawing funds is hard, and partly because it takes years before you get usable data.

  • Holden on October 22, 2009 at 5:37 pm said:

    Ian, it depends somewhat on what is meant by “sustainability.” I agree that your test is the only one for long-term impact of a project on a community. However, I believe we can fairly infer a major change in an individual’s life from less. Asset accumulation could be misleading in many ways, but I believe the measure has fewer pitfalls than the measures discussed in this post.

  • Ian Turner on October 24, 2009 at 6:07 pm said:

    Holden,

    Suppose that you take someone who is earning $150 per year and give them an extra $100 per year in cash for five years. Wouldn’t you expect them to build up assets every year during those five years? But in the sixth year, when you stop giving, this person will survive on $150 plus whatever assets they saved in the prior five years. By the time you get to year 15, is there any reason to think that the person’s life will be much better off than it was in year zero? Perhaps, but it’s far from obvious to me.

    My problem with the millennium villages is that it’s impossible to tell what part of their success comes from injecting $100 per person into the local economy, and what part of their success comes from improving crop yields, reducing malaria, etc.

    Cheers,

    –Ian

  • Holden on October 27, 2009 at 9:23 am said:

    Ian, I think it’s hard to tell whether Millennium Villages has had “success” in any area, and harder to break it down as you describe.

    I have much more confidence that lives were improved if I see asset accumulation than if I see a one-time bump in incomes/crop yields, though less confidence than if I see a major health improvement (particularly a life-saving one) or – as you say – a long-term, sustained improvement in living standards.

  • Sam Lee on January 1, 2010 at 6:42 pm said:

    Holden,

    On economic empowerment long term success metrics, in addition to typical income gain, and asset accumulation that you prefer, what other indicators have you found be used in practice?

  • Holden on January 5, 2010 at 7:03 pm said:

    Sam, speaking off the cuff, I’d say the types of evidence we’ve seen collected fall into the following broad categories:

    • Agricultural output (as discussed in this post)
    • “Income,” sometimes inferred using a relatively complex methodology, in PPP-adjusted dollar terms.
    • Standard-of-living and quality-of-life measures, which can include asset accumulation but also access to water, enrollment of children in school, health & nutrition measures, and more (example: Grameen Bank’s 10 Indicators).

    The last category is the one we find most compelling.

  • Bryan on March 29, 2010 at 6:44 pm said:

    Analytically weak. There are troves of information about the Millennium Villages out there and that givewell has decided to base their evaluation on a couple 600 opeds with obvious bias is unfortunate.

    Though this is somewhat dated for those of you who don’t like reading: http://video.google.com/videoplay?docid=-2617281089086064784&ei=KDqxS9amFZ6yqQLL1KHiBg&q=millennium+villages+cross+cutting+initiatives&view=3#

    Otherwise, check out the ODI report, the peer reviewed stuff and the real literature on this. Don’t hold things to unreasonable burdens of proof without doing any substantive research into this, that’s not productive or fair.

Comments are closed.