I wonder how much of the difference between our approach to charity and others’ approach comes from this very simple fact:

We think that improving people’s lives is really hard.

It might be easy to brighten someone’s day, even their week. But to get someone from poverty and misery to self-sufficiency and a world of opportunity … you have to change not just their resources, but their skills, often their attitude, and always their behavior. You can try to do this early in life or late; either way, it’s an uphill battle.

As we’ve been over and over, that’s why we think “money spent” is a poor proxy for “good accomplished,” and tend not to get as excited as others over dollars raised or dollars spent. But as we get deeper into reading apps, I’m finding there’s more and more to this rift.

I’m very skeptical of any program that claims great effects with relatively low amounts of intervention, whether it’s a one-time class on condom use or a once-a-week tutoring session. I think about how easy it is for the people I know to sit through some class, walk out full of ideas, and forget them a week later, and I think – if you’re doing anything meaningful for people with this little investment, you must be some sort of sorceror.

I trust survey data about as far as I can throw it (note: data cannot be thrown). Katya’s post today gives a sniff of the argument that “people are notoriously bad predictors of their behavior”; there’s actually an enormous amount of literature out there ramming home this point (if you really want to see it, let me know). We’ve seen a lot of survey data along the lines of “97% of participants felt the program improved their ability to ___,” which would be great if wishes were horses and horses were great.

I tend to feel the same way about anecdotes and personal experiences. They can be useful to get a picture of a program, but as evidence that it works? People are easy to change in the moment and incredibly hard to change in the long run. How can you possibly tell from watching a group of children for a year that they’re headed for better lives?

And in the end, though I keep reminding myself that evidence of effectiveness can come in all different forms, I can’t keep down that longing for randomized studies, or at least studies that employ some sort of comparison group. These are very rare in the applications we’re receiving: charities tend to look only at their clients. While this clearly saves them a lot of money and hassle, it leaves me wondering whether they’re really helping people … or just picking out the ones who are going to succeed anyway (you’re probably familiar with this question in the context of Ivy League schools). I’m sure this is heresy, but if we started a Placebo Foundation that sought out the “most motivated” poor people as clients, did absolutely nothing for them, and then examined their outcomes, would we find the same “successes” that many charities claim?

And yet every study I’ve read from applicants – even when limited to survey data, even when lacking a comparison group of any kind – concludes that the program in question was a success.

Unless there’s willful deception going on here, this implies to me that the people who work at charities think helping people is really, really easy; that there’s no need to worry about all the questions above; that the flimsiest and most perfunctory of evidence is good enough to walk away from feeling that people in need have been truly helped.

Maybe we’re wrong. Maybe helping people is that simple. If that’s what you think, keep writing those checks to whomever sends you mail, and make sure they aren’t blowing a penny more than they have to on salaries. If you’re concerned as I am, though, I’ve got some tough news from you: aside from GiveWell, I don’t think you have much company.

GiveWell started in the hedge fund world (as a collaboration between coworkers), and our staff and board is heavy on for-profit experience. When we talk about metrics and dashboards, a lot of people assume we’re applying business concepts, and that we want to see charities run more like businesses. That isn’t true.

Do businesses conduct academic-style studies employing randomization and statistical significance tests to figure out how they’re doing? If any do, I doubt they’re very good ones. If you’re running a chain of pizza joints, you don’t need to know anything about statistics, and you can get by without much of a dashboard too. That’s because the single most important thing you need to measure – profit – is hitting you in the face every day. And most of the other things you need to measure take care of themselves too. It’s easy to know how much you’re paying delivery boys and how that impacts your bottom line – and if they slack off, you’ll be hearing about it from your consumers. No followup surveys necessary.

The more complicated things get, and the more difficult-to-observe things you have that might affect you in long-term and difficult-to-notice (but still important) ways, the more you need to audit. That’s why companies do it. But there’s no business whose operations are as difficult to understand – and whose outcomes are as difficult to measure – as even the simplest charity trying to fight poverty.

If a charity doesn’t follow up with its clients, it will never know whether its efforts are very successful, moderately successful, or entirely worthless. It will have no way of figuring out when the plan that made sense in its director’s head is falling short in reality. It could keep doing things that make logical sense, but don’t work at all, for hundreds of years – and never find out. None of that is true of any business.

Reading fundraisers’ arguments for why their charities are good, I keep hearing things like “We’re very experienced,” “We’ve been around for a long time,” “We’re well recognized/well respected/well established.” These are all points that make perfect sense as praises for a for-profit business, and no sense as praises for a charity. Charities are mission-driven, not self-gain-driven. Continued existence is not evidence of success; fundraising excellence isn’t either. The fact is that without evaluation, a charity’s success is something no one can see.

Believe me, I know about the inherent limitations of measurement, and I know how expensive and annoying it is too. I don’t like to fill out a survey every time I blow my nose; if I were running a business, I wouldn’t want to be spending half our budget on figuring out whether our pizza really was the cause of improved customer happiness, or whether selection bias were involved. But if I were running a charity, I wouldn’t see a choice. It’s tough but true: unlike a business, a charity needs all that annoying stuff – or it’s working in the pitch dark.

How do you evaluate an organization that does all of this?

Elie and I have been wrestling with this problem for the past few days, especially today. We’ve put together what we think is a reasonable final-round application (see below), and sent it to five of our strongest huge-comprehensive-giganto-mega-charities to see how they feel about it. Generally, they haven’t felt very good, and we’re trying to figure out what this means. We’d love your thoughts.

In our Face/Off a few days ago, Elie argued that a good organization ought to be able to give us a picture of what they do and whether it works – no matter how big they are. If they’re too big to give this picture, then – well, they’re too big. I found this pretty convincing, so we crowned Elie as the winner of our inaugural Face/Off (don’t despair, Holden diehards, there will be more to come) and created the following application:

• Application for mega-charities: instructions
• Application for mega-charities: The Matrix

We’ve dubbed this app The Matrix, because its key feature is a gigantic matrix of regions and indicators – we want to know what each charity does and doesn’t have data on, in every region it works in. It’s visually gargantuan, but we’re not asking applicants to fill in statistics in the cells. All we’re asking is that they tell us what they do and don’t do – and what they do and don’t measure – in each of their regions.

All of the people we got to talk to today agreed that the application was off-putting/overwhelming, at least at first glance. We then explained how we want them to go about it: send us what they already have, and use The Matrix just to tell us what they sent (we’re not asking them to do any writing, interpretation, or summarizing – just send what they have and classify it). One of the applicants said this was fine, but the others both hesitated even then. We were told that just compiling what activities are done in each region could take a major project; that pulling up all the relevant reports (and categorizing them) could take weeks.

So my response was: How do you tell what’s going on at a bird’s-eye view level? How does your Executive Director know what’s going on and how you’re doing? How does your Board know? What do you look at to decide which programs to expand and which ones to change?

The basic picture we got is that priorities are often driven project-by-project: the people on the ground (whether local organizations seeking help, or staff in the field) make proposals, and the central office reviews them individually. After some discussion, we came to differing agreements with each organization, and we’ll see what they send us (and keep you posted).

But in my mind, the biggest and most important question (the one in bold above) remains largely unanswered. It seems to me like if you’re running 200 different programs in 20 different countries, you need to be monitoring the heck out of them, and you need to have some kind of summary view that you can show to the people in charge and give them a real picture of what’s going on. Otherwise, how are those people in charge?

I don’t hold small organizations, or simple organizations, to the same standard of measurement and organization. A bicycle doesn’t need a dashboard, because you can tell immediately if something’s wrong; unmetaphorically, if you work in one place, doing one thing, you can be part of the day-to-day activities and understand them intuitively, without ever measuring or documenting a thing. But for the life of me, I can’t understand how it’s possible to have an “intuitive” feel for your work when you’re trying to help thousands of different people, thousands of miles away, living in cultures and regions you didn’t grow up in and will never truly understand. It seems like the only way to have any idea of what’s going on is to collect an enormous quantity of facts and put great care into interpreting and organizing them. Elie and I recognize that we aren’t experienced in these matters … but the idea that an organization would take weeks to put together a summary of what it does and whether it works is just hard for us to swallow.

What do you think? Are we barking up the wrong tree? Is it unreasonable to ask charities for this much organized information? Should we be trying to evaluate charities without getting the full picture of their activities? How can we?

Can you be comfortable donating to a mega-vehicle, without seeing the dashboard?

There’s been a lot of excitement about microlending, especially since the 2006 Nobel Peace Prize created a wave of stories about it. The basic idea is to make loans to the destitute, helping them pull themselves out of poverty. The stories and the numbers floating around from fundraisers paint a picture that’s simply too good to be true (below) … we can’t authoritatively say that we have the real story, but I think I’m starting to understand the difference between these stories and how microlending actually helps people.

First, the good stuff.

The microlending you’ve heard of: beggars to billionaires, thousands of times a day
gramm
When I first heard about microfinance, everywhere I turned was a story like this one, or this one, or perhaps this one, or all these ones. (Dang. That was the easiest time I’ve ever had finding examples of anything.) Basic story: poor entrepreneur has a killer business model (OK, usually a fruit stand) that’s ready to expand, if only someone would lend the funds. In swoops a microlender; the entrepreneur borrows, expands the business, succeeds, and changes her life for good.

Now imagine how I felt after combining those basic stories with the aggregate numbers: repayment rates like 97%, on billions of dollars lent, with a typical loan being around $50 ??? So if each loan repaid is a person who has built a business and escaped poverty … then if I donate $50, it can be lent out repeatedly and help a family escape poverty every 6-12 months for the rest of eternity? And if Grameen Bank has lent out ~$5 billion … that’s … 100 million people lifted out of poverty forever?

I know what you’re thinking: “Sign me up! How could Holden doubt any of these claims?” Well, call me the Grinch, but here were a few things that bugged me:

1. If every loan were really going to expand a successful business, the lenders wouldn’t be nonprofits – they would for-profits, and Muhammad Yunus and his friends would all be katrillionaires. Oh, and poverty wouldn’t exist.

2. Expanding a business is going to involve serious risk, unless you’ve got new customers already lined up for miles. I know I never gave much thought to being an entrepreneur until I had some cash saved up … seems hard to believe that people living hand-to-mouth are all chomping at the bit to do it. This paper makes this general argument (hard to be an entrepreneur when you’re in poverty) a bit better than I just did.

3. Seriously … how many people do you know who could start a profitable business if you loaned them a million dollars?

But then again, if these people aren’t lifting themselves out of poverty, why and how are they borrowing money and paying it back? What’s really going on here?

The microlending that happens: credit as basic need

Here’s a paper you’ll be hearing more about in future posts. It’s a review of many studies of the actual impact of microfinance on poverty. Pages 17-20 describe one of the more rigorous attempts at this, and the debate over what it showed; the one thing all three of the studies agree on is that people with credit available had smoother consumption: “household consumption increased most during the lean Aus season, when the poor often go hungry.”

I don’t know about you, but I’ve always taken my ability to smooth consumption (saving, borrowing, etc.) so much for granted that it didn’t even occur to me what it would be like to live without it. But as this paper argues in great detail, the world’s poorest often are less badly hurting for food/medicine than they are for the most basic support networks and mechanisms we use to manage our lives. If your income is seasonal and you’ve got no bank, forget about starting a business – you can’t even plan for the next week.

In this context, giving loans isn’t about creating and expanding ventures, it’s about meeting a basic need that might be as vital as the classic food, water, and shelter: the ability to manage risk and plan for changes. Loans aren’t the key to ending poverty, but they’re one of many things that we have to make available.

So I’ve done it, I’ve killed Santa Claus. Microloans won’t end global warming and baldness; they’re just one more way of helping people, along with bednets, condoms, and water purification. Donors, how do you feel? Disgusted? Disillusioned? Ready to swear off charity for good? Or interested in learning more?

Thanks to Tim of Philanthropy Action for discussing this issue with me and pointing me to two of the papers above (Dichter; Banerjee/Duflo).

See this post for the ground rules of this Face/Off.

Most of the organizations we’re covering in Africa don’t just do one thing, they do many. I want to get a picture of what the organization as a whole is trying to accomplish (my benchmark is to understand 80% of programs) and the evidence that supports the effectiveness of those programs. I can’t think of any other way to evaluate the efficacy of an organization. It sounds like Holden is worried that asking for 80% is going to be too hard on the charities we’re evaluating. Too hard to explain 80% of what you do? How could that be? If you can’t explain 80% of what you do relatively easily, then there’s just no way that your organization is running effectively. The organizations we fund have to be able to do that.

Holden’s approach, I think, relies on asking the charities what they plan to do in the future and why and relying on that to guide our grantmaking process. I just don’t put a lot of weight on what a charity tells us they’re going to do. We’re funders and they’re going to tell us what we want to hear. I don’t want to fund a charity based on who can give us the slickest presentation and who can whisper sweet nothings in my ear.

Even if I did mostly buy what the charities were telling me, I question their ability to reasonably predict their program’s success. Why is it so difficult? Really, we just need a charity to tell us what they’re planning to do, argue that it’s smart, and articulate how they plan to execute the program. But, it’s just not that simple because it’s really difficult to figure out what’s going to work. Africa has a lot of complicated, interconnected problems, and I don’t want to judge the best organization based on what I think will work. I want to see definitive results of what works, and then I want to throw money at it. I want to fund the organization that’s proven it can effect change (save lives, increase economic opportunity, etc.) despite all the obstacles. Holden would probably object here that if the problems are so complicated why do I trust any organization to decide what to do next? Well, I don’t just think that those that have proven, effective programs are more likely to make better decisions in the future (though the probably would). I want to fund organizations that are going to keep executing on their proven programs. As long as the same problems persist in Africa, I just want to keep throwing money at the organizations with proven, effective programs.

That’s the heart of my argument, but there are two more pieces of support rattling around in my head. First, I can imagine funding something new if it were necessary. But, there’s so many ways to save lives in Africa that have already been proven, that I don’t see any reason to go off and try something new. Second, I want to fund organizations for which it’s incredibly clear why we’ve chosen them. I want to fund organizations that allow anyone to read our site and agree or disagree with our judgments based on the facts. I want to take as much as possible of the subjectivity – how well a new program will likely work in the future – out of the equation.