The GiveWell Blog

Agriculture charity evaluation: Incomes boosted are not the same as lives changed

(updated on: ) | by Holden

What’s wrong with this “evidence of impact” for high-profile charities?

Among other possible problems, two major issues jump out:

1. No context on what “normal” variation in incomes looks like for poor farmers. Some years have more favorable weather – and local economic situations – than others. Enough that one year’s income or crop yield could be double another’s? 4x? 20x?

Unfortunately, one of the better pieces of “evidence” that jumps to mind is a 75-year-old novel, The Good Earth, whose farmer protagonist is comfortable one year and has literally zero income the next, for no other reason than the weather. If a given year’s yield were close enough to zero, the next year could be a huge increase (2x, 4x, 20x or more) simply by returning to normal.

I have seen little information on the local year-to-year volatility that poor farmers can experience, but I imagine that it (a) varies greatly from region to region and (b) could easily involve incomes falling and jumping by enormous amounts.

None of the above reports provide any context on this question, beyond qualitative statements about how favorable the rains were in each year examined. None of them employ any sort of “comparison group” of farmers (aside from one vague reference to “farms not using improved seeds and fertilizers” in the Malawi Millennium Village). Ultimately, none accomplish one of the most basic goals of an evaluation: giving a sense of how likely the “gains” they describe are to have arisen by pure chance.

With larger sample sizes, we might be able to use country-level volatility for context. But that brings me to the next problem.

2. We have no assurance that the described gains are representative, as opposed to “cherry-picked.”

All of the above organizations have reputations for consistent and thorough monitoring and evaluation, yet in all cases, we find ourselves looking for “impact” from a tiny subset of their projects.

Some ways to produce more compelling evidence of impact

  1. Be clear about what is being measured and what is being published, and when. It seems to us that in this area, charity evaluation lags far behind clinical trials, which are constantly registered before they are complete so people can track their progress. (The Poverty Action Lab is similarly transparent with its own ongoing projects.)
  2. More sample size; more context; use of comparison groups. Discussed above.
  3. Look for more sustained improvements in people’s lives. One measure I find superior to straight “income” or “crop yields” is asset accumulation. A jump in income could be temporary; if someone upgrades their roof or sanitation, it’s likely that at least they expect the gain to be a real and lasting one. The Village Enterprise Fund’s evaluation is one of the better charity evaluations I’ve seen in the area of economic empowerment, partly because it focuses on standard of living rather than a simple measure of income.

*It’s possible that the yields mentioned are for “clusters” of villages rather than individual villages; there are only 12 clusters. However, the source documents available for Sauri and Koraro appear to be at the village rather than the cluster level, and the details of how the measurements were made are unclear.

Are charities helping? We don’t know

(updated on: ) | by Holden

In a recent debate, David Hunter’s article on the nonprofit sector has taken heat for its assertion that “While nonprofits work incredibly hard, with passion and dedication, and often in incredibly difficult circumstances to solve society’s most intractable problems, there is virtually no credible evidence that most nonprofit organizations actually produce any social value.”

We agree with the claim for the sectors we’ve examined, which we believe are similar to the sectors Mr. Hunter has examined: particularly thorny areas such as charities working to improve education and international charities addressing extreme poverty overseas. These are problems on which experts have struggled for decades to make any progress, and while we don’t necessarily agree that most charities are failing to produce value, we agree that most charities cannot produce any credible evidence that they are. This is different from the claim that Sean Stannard-Stockton attributes to Mr. Hunter (“most nonprofits and the social sector as a whole is not currently producing social value”), but it still means that it’s very hard for a donor to give with confidence.

The information we have

Our belief is based on two years of looking for this evidence; we’ve published the full details of our findings online, and you can see our summary of international charities (only 19 out of 320 examined publish any impact-related evaluation reports) and U.S. equality of opportunity charities (only 6 of 83 examined provide credible impact-related reports, and 2 of these show negative or no impact).

In addition, in a guest post on the GiveWell Blog, David Anderson of the Coalition for Evidence-Based Policy estimates that 75% of rigorous evaluations show weak effects, no effects, or negative effects.

More information needed

On the other hand, we also believe the criticism that Mr. Hunter doesn’t support his own claim with evidence has merit. We would like more clarity on which sectors Mr. Hunter has examined and is referring to, and information on where he has looked for evidence and what he has found.

In addition, we feel that examples of failing/harmful programs, such as “Well intentioned but ineptly run mentoring programs where failed matches reinforce in youngsters a sense of their low worth and poor prospects” (and the other items on the list on page 2 of the article) should be clearly referenced to summaries of evidence.

The truth is that we cannot have a very informed debate about how much value nonprofits create because we have so little evidence of any kind. Some people adamantly believe that nonprofits create enormous value; others are skeptical that they create any; and there is very little to go on, at least in the sectors under discussion.

Nonprofits that do have credible evidence of their social impact

The good news for donors is that they need not be in the dark if they give to the right charity. Our top-rated charities do produce credible evidence of their social impact. We encourage individual donors to expand and fund these charities until and unless others follow suit.

Is fundraising stuck in the world of Mad Men?

(updated on: ) | by Elie

Advertising is based on one thing: Happiness. And you know what happiness is? Happiness is the smell of a new car… It’s freedom from fear. It’s a billboard on the side of the road that screams with reassurance that whatever you’re doing is okay. You are okay. — Don Draper on AMC’s Mad Men. (Video on YouTube.)

I sometimes wonder how Don Draper would react to someone telling him that in 2009, many consumers make their buying decisions based on a product’s quality rather than the feeling a product’s advertising instills in them, or the narrative in which the ad places them.

I picture Don saying, “User reviews? Restaurant ratings? Product testing? No way. That’s just not how consumers are hardwired to think.”

When I see Sasha Dichter and Nathaniel Whittemore saying the same thing, I wonder whether they’re right that the problem is donors as opposed to the information donors have access to.

Sure, it’s possible that donors are hardwired a certain way and may never care about the real impact their gift has. But it’s also possible that many donors would look past happiness and use information on impact – if only it were easier to find. That’s what we’re trying to bring about.

What’s different about Kiva

(updated on: ) | by Holden

Why has Kiva been singled out for so much criticism lately (see GiveWell Board member Tim Ogden’s summary)?

Part of the answer is that Kiva has arguably been misleading donors – but that can’t be the full answer. David Roodman’s original post could never have come about without the fact, as Mr. Roodman puts it, that “the way Kiva actually works is hidden in plain sight.” And our followup analysis on repayment rates was only possible because Kiva makes all its repayment data publicly and easily available. As Elie said in that post, “Similar analysis would be impossible for nearly every other charity I can think of.”

Contrast Kiva with, for example, UNICEF. Kiva makes it possible to trace the path of your donation, to the extent that such tracing is realistic (and it largely turns out to be more along the lines of “you funded a certain MFI” rather than “you funded a certain person”). UNICEF doesn’t even seem to have a breakdown of how much money is going to each continent. We definitely can’t find information on questions like (a) What specific projects are you funding? (b) What is your role in each? (c) What new projects are planned, and where? (d) How is each project going, whom is it affecting, and how?

There are no strange patterns in UNICEF’s numbers because there are no numbers. There are no contradictions because there is no concrete information. And the intent here isn’t to single out UNICEF – it’s merely one of the vast majority of international aid organizations about which we know essentially nothing.

Giving an impression to donors that’s undermined by the facts is a minor scandal. When will complete opacity – simply sharing no information at all – be a major one?

DonorsChoose vs. Kiva

(updated on: ) | by Holden

This Tactical Philanthropy post implies that the salient difference between DonorsChoose and Kiva is that DonorsChoose is more “authentic” in terms of connecting donors to projects. (Update: Sean points out in the comment below that most of the post I cited was a quote from a former DonorsChoose employee and doesn’t necessarily reflect TacticalPhilanthropy’s opinion.)

We think there’s another important difference. Kiva and DonorsChoose work on very different types of problems, that offer donors vastly different opportunities to cause significant impact. And if we had to pick one, we’d bet on Kiva as the better option if you want to change lives.

The problem of improving children’s education is far, far more difficult than most people suspect – for our most recent coverage, see our multi-part series on the recent Harlem Children’s Zone study, and how excited the academic community has been over a one-time and modest improvement. And DonorsChoose does not offer the chance to support any of the most promising and tested educational interventions, such as the creation of intensive charter schools. Funding DonorsChoose is making a bet that classroom materials are where it’s at, even when the teachers, school systems and children’s outside environments remain the same. As far as we know, there is zero evidence that improved school supplies have resulted in any measurable improvements in the U.S., and even in the developing world the proposition seems iffy.

Funding Kiva, even if it doesn’t mean funding individuals, means funding microfinance charities. We are very ambivalent about microfinance; recent posts on this topic include questions about whether there’s any empirical support for microfinance charities’ claims and an interview with David Roodman about microfinance charity in which many unresolved questions come up. But the bottom line is that when you fund microfinance charities, your money ends up in the poorest parts of the world, and often (though we don’t know how often or how much) helps people there manage their volatile financial lives with credit, savings and other assistance.

(In fact, the way in which your money is most helpful may be by doing exactly what it has been criticized for doing – effectively funding partners’ other projects, such as savings vehicles, rather than the loans you see on your screen.)

Fund financial services in the poorest parts of the world, or fund an untested and low-intensity approach to a problem that higher-intensity programs have struggled with for decades. $900 can be five digital cameras for a classroom or a year’s income for 3 people.

That’s the decision as it looks to us, as we ask not “How can I make sure that I can see the exact person who’s getting my money?” but “How can my money accomplish as much good as possible?”

And on that note, we feel that our top-rated charities offer far better opportunities to improve lives than either, even if they can’t deliver the same emotional experience. As a donor, which would you rather have?

Kiva repayment data

(updated on: ) | by Elie

GiveWell Board member Tim Ogden asks, is Kiva reporting phantom payments?

It’s hard to say for sure, but it seems that the answers is either (a) yes, they are or (b) no, but borrowers listed on Kiva’s website are not representative of the average borrowers visiting that MFI.

The table below shows default rates for Kiva’s largest field partners (in terms of total loans provided through Kiva) and compares the default rates reported on Kiva to default rates reported for those same institutions on MixMarket, a site devoted to sharing data on microfinance.

Data from the two sources aren’t perfectly compatible but the broad story is clear: Kiva’s partners have higher repayment rates on their Kiva-listed loans than on their portfolio overall, as listed on MixMarket. This could come about in a couple ways:

  • The borrowers that partners list on Kiva could be substantially more likely to repay than “other” borrowers. If Kiva loans are effectively fungible between partners (as the recent discussions imply), this would mean that donors are likely “effectively” funding clients who are less likely to repay than those in the profiles they’re scanning.
  • Kiva’s partners could be “covering for” their clients to keep their Kiva-listed repayment rates high.

(Kiva data comes from a superb site, Kiva Data, a site stating that it is unaffiliated with Kiva as well as the partners’ pages on the Kiva website itself. MixMarket data comes from the MixMarket website. For definitions of items in the table, see the notes section below it.

It’s worth noting that this analysis is only possible because Kiva makes its data available and due to the efforts of those at Kiva Data and MixMarket who have created extremely useful information resources. Similar analysis would be impossible for nearly every other charity I can think of.)

Loans and default rates for Kiva’s 10 largest partners

Partner Number of loans on Kiva Number of loans on Mix Start date on Kiva Kiva-reported default rate Mix-reported default rate over similar time period p
PRISMA 4,493 22,306 8/18/07 0.0% 0.4% <.001%
LAPO-NGR 3,180 329,384 12/3/06 0.0% 0.4% <.001%
Manuela Ramos 3,918 16,982 9/24/07 0.0% 0.0% 100%
CREDIT 2,172 52,013 6/3/06 0.0% 0.1% 15%
SAT 1,645 73,268 12/7/07 0.0% 0.0% 100%
FINCA – PER 2,704 13,084 8/24/07 0.0% 0.3% 0.115%
BRAC – TZA 1,174 69,502 1/22/08 0.0% 6.6% <.001%
SPBD 1,863 7,816 9/1/06 0.0% 0.8% <.001%
MLF MicroInvest 2,240 13,248 4/6/07 0.0% 0.7% <.001%
Fundación Paraguaya 1,956 34,222 6/23/07 0.0% 2.4% <.001%

Notes on the data:

  • The data in the table above comes from Kiva’s 10 largest partners (in terms of number of loans made to date) who provided full MixMarket data for 2008. 2 of Kiva’s 10 largest partners doesn’t have this data accessible at the moment. AMK, Kiva’s 7th largest partner, reported no outstanding loans for 2008 on MixMarket and ADMIC, Kiva’s 9th largest partner, has not reported any data for 2008 yet.
  • p refers to the probability that, if the MFI’s Kiva borrowers were equally likely to default as its borrowers in general, the MFI would happen to have a 0% default rate for its Kiva loans. (Note that this calculation assumes that the time periods and the two definitions of “default rate” are comparable.) Low values mean it is extremely unlikely that this phenomenon would arise by chance, implying instead that the Kiva borrowers are systematically better than the normal borrowers, or that defaults are simply not being reported to/by Kiva.
  • The data on MixMarket is self-reported. Nevertheless, Mix is a respected source, and we think it’s reasonable to rely on it.
  • The date-ranges used for the Kiva data and Mix data above don’t sync up perfectly. Mix reports data for each calendar year. Kiva reports all-time data for each of its partners, some of whom joined in the middle of a calendar year. We’ve roughly adjusted for this by ignoring a Kiva partner’s first partial year and weighting the Mix data for the remaining years by the number of loans outstanding each year.
  • MixMarket doesn’t use the term “default rate.” Instead, we’ve used “Write-off ratio %,” defined as Write Offs / Loan Portfolio, gross, average. We’d prefer to also look at Portfolio at Risk figures. However, Kiva only reports delinquencies for loans currently outstanding and therefore can’t be matched well to Mix data.
  • MixMarket reports data as a % of the portfolio’s dollar value. Kiva reports data as a % of loans made and value loaned. Given that Kiva reports 0% default rates and Mix reports >0% default rates, there’s clearly some inconsistency. Nevertheless, without information regarding the size of the loans defaulted on at Kiva, we can’t guess at the magnitude of the inconsistency.
  • All data used in this table is available in an Excel file. If you’re interested see Kiva major partners data (XLS). This file also contains links directly to Kiva’s partners page both on Kiva and MixMarket.