My thoughts on the latest Giving Carnival:

Fundraising today is all about the pitch; 10 years from now, I hope it will be about the product.

Fundraising today reminds me a lot of car commercials. Car commercials try to convince you that a Ford is better than a Toyota or a Volvo outclasses a Volkswagen by showing pictures of cars speeding around tight curves on steep mountains. Those commercials don’t tell you anything about what distinguishes one car from another. Similarly, my experience with fundraising materials from non-profits is that they’re virtually interchangeable (all are heavy on pictures of extremely cute 5 year-old children) and give me no information about what distinguishes one non-profit from another.

In the case of cars, the world has changed such that I have the ability to distinguish between powerful cars and less powerful cars, reliable cars and less reliable cars, safe cars and less safe cars using any number of resources (Edmunds, Consumer Reports, etc.).With charities the same is not yet true, but it will be.

In the future, fundraisers will have to support their programs by clearly explaining what their programs accomplish (as Holden and I keep learning, a pretty difficult enterprise) and providing evidence for their claims. They’ll have to do this because donors will demand it. No longer willing to accept vague descriptions of program accomplishments and no longer scared to criticize organizations which, though they mean well, accomplish little, donors will force a new age of fundraising, one where the conversation revolves around how best to help those in need.

That’s the utopian future I hope for and envision. The future that frightens me most (and the one that other carnival participants, like here and here, envision) is a future that’s a stronger, faster, super-powered version of today. More efficient strategies, super-slick presentations, and lightning fast, fully-linked technologies allow people to give more with virtually no thought at all. Where the world might be improving but then again, it might not, and it doesn’t even matter because no one will care. “People are giving more than ever!”, they’ll say, and that will be that.

I wonder how much of the difference between our approach to charity and others’ approach comes from this very simple fact:

We think that improving people’s lives is really hard.

It might be easy to brighten someone’s day, even their week. But to get someone from poverty and misery to self-sufficiency and a world of opportunity … you have to change not just their resources, but their skills, often their attitude, and always their behavior. You can try to do this early in life or late; either way, it’s an uphill battle.

As we’ve been over and over, that’s why we think “money spent” is a poor proxy for “good accomplished,” and tend not to get as excited as others over dollars raised or dollars spent. But as we get deeper into reading apps, I’m finding there’s more and more to this rift.

I’m very skeptical of any program that claims great effects with relatively low amounts of intervention, whether it’s a one-time class on condom use or a once-a-week tutoring session. I think about how easy it is for the people I know to sit through some class, walk out full of ideas, and forget them a week later, and I think – if you’re doing anything meaningful for people with this little investment, you must be some sort of sorceror.

I trust survey data about as far as I can throw it (note: data cannot be thrown). Katya’s post today gives a sniff of the argument that “people are notoriously bad predictors of their behavior”; there’s actually an enormous amount of literature out there ramming home this point (if you really want to see it, let me know). We’ve seen a lot of survey data along the lines of “97% of participants felt the program improved their ability to ___,” which would be great if wishes were horses and horses were great.

I tend to feel the same way about anecdotes and personal experiences. They can be useful to get a picture of a program, but as evidence that it works? People are easy to change in the moment and incredibly hard to change in the long run. How can you possibly tell from watching a group of children for a year that they’re headed for better lives?

And in the end, though I keep reminding myself that evidence of effectiveness can come in all different forms, I can’t keep down that longing for randomized studies, or at least studies that employ some sort of comparison group. These are very rare in the applications we’re receiving: charities tend to look only at their clients. While this clearly saves them a lot of money and hassle, it leaves me wondering whether they’re really helping people … or just picking out the ones who are going to succeed anyway (you’re probably familiar with this question in the context of Ivy League schools). I’m sure this is heresy, but if we started a Placebo Foundation that sought out the “most motivated” poor people as clients, did absolutely nothing for them, and then examined their outcomes, would we find the same “successes” that many charities claim?

And yet every study I’ve read from applicants – even when limited to survey data, even when lacking a comparison group of any kind – concludes that the program in question was a success.

Unless there’s willful deception going on here, this implies to me that the people who work at charities think helping people is really, really easy; that there’s no need to worry about all the questions above; that the flimsiest and most perfunctory of evidence is good enough to walk away from feeling that people in need have been truly helped.

Maybe we’re wrong. Maybe helping people is that simple. If that’s what you think, keep writing those checks to whomever sends you mail, and make sure they aren’t blowing a penny more than they have to on salaries. If you’re concerned as I am, though, I’ve got some tough news from you: aside from GiveWell, I don’t think you have much company.

GiveWell started in the hedge fund world (as a collaboration between coworkers), and our staff and board is heavy on for-profit experience. When we talk about metrics and dashboards, a lot of people assume we’re applying business concepts, and that we want to see charities run more like businesses. That isn’t true.

Do businesses conduct academic-style studies employing randomization and statistical significance tests to figure out how they’re doing? If any do, I doubt they’re very good ones. If you’re running a chain of pizza joints, you don’t need to know anything about statistics, and you can get by without much of a dashboard too. That’s because the single most important thing you need to measure – profit – is hitting you in the face every day. And most of the other things you need to measure take care of themselves too. It’s easy to know how much you’re paying delivery boys and how that impacts your bottom line – and if they slack off, you’ll be hearing about it from your consumers. No followup surveys necessary.

The more complicated things get, and the more difficult-to-observe things you have that might affect you in long-term and difficult-to-notice (but still important) ways, the more you need to audit. That’s why companies do it. But there’s no business whose operations are as difficult to understand – and whose outcomes are as difficult to measure – as even the simplest charity trying to fight poverty.

If a charity doesn’t follow up with its clients, it will never know whether its efforts are very successful, moderately successful, or entirely worthless. It will have no way of figuring out when the plan that made sense in its director’s head is falling short in reality. It could keep doing things that make logical sense, but don’t work at all, for hundreds of years – and never find out. None of that is true of any business.

Reading fundraisers’ arguments for why their charities are good, I keep hearing things like “We’re very experienced,” “We’ve been around for a long time,” “We’re well recognized/well respected/well established.” These are all points that make perfect sense as praises for a for-profit business, and no sense as praises for a charity. Charities are mission-driven, not self-gain-driven. Continued existence is not evidence of success; fundraising excellence isn’t either. The fact is that without evaluation, a charity’s success is something no one can see.

Believe me, I know about the inherent limitations of measurement, and I know how expensive and annoying it is too. I don’t like to fill out a survey every time I blow my nose; if I were running a business, I wouldn’t want to be spending half our budget on figuring out whether our pizza really was the cause of improved customer happiness, or whether selection bias were involved. But if I were running a charity, I wouldn’t see a choice. It’s tough but true: unlike a business, a charity needs all that annoying stuff – or it’s working in the pitch dark.

How do you evaluate an organization that does all of this?

Elie and I have been wrestling with this problem for the past few days, especially today. We’ve put together what we think is a reasonable final-round application (see below), and sent it to five of our strongest huge-comprehensive-giganto-mega-charities to see how they feel about it. Generally, they haven’t felt very good, and we’re trying to figure out what this means. We’d love your thoughts.

In our Face/Off a few days ago, Elie argued that a good organization ought to be able to give us a picture of what they do and whether it works – no matter how big they are. If they’re too big to give this picture, then – well, they’re too big. I found this pretty convincing, so we crowned Elie as the winner of our inaugural Face/Off (don’t despair, Holden diehards, there will be more to come) and created the following application:

• Application for mega-charities: instructions
• Application for mega-charities: The Matrix

We’ve dubbed this app The Matrix, because its key feature is a gigantic matrix of regions and indicators – we want to know what each charity does and doesn’t have data on, in every region it works in. It’s visually gargantuan, but we’re not asking applicants to fill in statistics in the cells. All we’re asking is that they tell us what they do and don’t do – and what they do and don’t measure – in each of their regions.

All of the people we got to talk to today agreed that the application was off-putting/overwhelming, at least at first glance. We then explained how we want them to go about it: send us what they already have, and use The Matrix just to tell us what they sent (we’re not asking them to do any writing, interpretation, or summarizing – just send what they have and classify it). One of the applicants said this was fine, but the others both hesitated even then. We were told that just compiling what activities are done in each region could take a major project; that pulling up all the relevant reports (and categorizing them) could take weeks.

So my response was: How do you tell what’s going on at a bird’s-eye view level? How does your Executive Director know what’s going on and how you’re doing? How does your Board know? What do you look at to decide which programs to expand and which ones to change?

The basic picture we got is that priorities are often driven project-by-project: the people on the ground (whether local organizations seeking help, or staff in the field) make proposals, and the central office reviews them individually. After some discussion, we came to differing agreements with each organization, and we’ll see what they send us (and keep you posted).

But in my mind, the biggest and most important question (the one in bold above) remains largely unanswered. It seems to me like if you’re running 200 different programs in 20 different countries, you need to be monitoring the heck out of them, and you need to have some kind of summary view that you can show to the people in charge and give them a real picture of what’s going on. Otherwise, how are those people in charge?

I don’t hold small organizations, or simple organizations, to the same standard of measurement and organization. A bicycle doesn’t need a dashboard, because you can tell immediately if something’s wrong; unmetaphorically, if you work in one place, doing one thing, you can be part of the day-to-day activities and understand them intuitively, without ever measuring or documenting a thing. But for the life of me, I can’t understand how it’s possible to have an “intuitive” feel for your work when you’re trying to help thousands of different people, thousands of miles away, living in cultures and regions you didn’t grow up in and will never truly understand. It seems like the only way to have any idea of what’s going on is to collect an enormous quantity of facts and put great care into interpreting and organizing them. Elie and I recognize that we aren’t experienced in these matters … but the idea that an organization would take weeks to put together a summary of what it does and whether it works is just hard for us to swallow.

What do you think? Are we barking up the wrong tree? Is it unreasonable to ask charities for this much organized information? Should we be trying to evaluate charities without getting the full picture of their activities? How can we?

Can you be comfortable donating to a mega-vehicle, without seeing the dashboard?

There’s been a lot of excitement about microlending, especially since the 2006 Nobel Peace Prize created a wave of stories about it. The basic idea is to make loans to the destitute, helping them pull themselves out of poverty. The stories and the numbers floating around from fundraisers paint a picture that’s simply too good to be true (below) … we can’t authoritatively say that we have the real story, but I think I’m starting to understand the difference between these stories and how microlending actually helps people.

First, the good stuff.

The microlending you’ve heard of: beggars to billionaires, thousands of times a day
gramm
When I first heard about microfinance, everywhere I turned was a story like this one, or this one, or perhaps this one, or all these ones. (Dang. That was the easiest time I’ve ever had finding examples of anything.) Basic story: poor entrepreneur has a killer business model (OK, usually a fruit stand) that’s ready to expand, if only someone would lend the funds. In swoops a microlender; the entrepreneur borrows, expands the business, succeeds, and changes her life for good.

Now imagine how I felt after combining those basic stories with the aggregate numbers: repayment rates like 97%, on billions of dollars lent, with a typical loan being around $50 ??? So if each loan repaid is a person who has built a business and escaped poverty … then if I donate $50, it can be lent out repeatedly and help a family escape poverty every 6-12 months for the rest of eternity? And if Grameen Bank has lent out ~$5 billion … that’s … 100 million people lifted out of poverty forever?

I know what you’re thinking: “Sign me up! How could Holden doubt any of these claims?” Well, call me the Grinch, but here were a few things that bugged me:

1. If every loan were really going to expand a successful business, the lenders wouldn’t be nonprofits – they would for-profits, and Muhammad Yunus and his friends would all be katrillionaires. Oh, and poverty wouldn’t exist.

2. Expanding a business is going to involve serious risk, unless you’ve got new customers already lined up for miles. I know I never gave much thought to being an entrepreneur until I had some cash saved up … seems hard to believe that people living hand-to-mouth are all chomping at the bit to do it. This paper makes this general argument (hard to be an entrepreneur when you’re in poverty) a bit better than I just did.

3. Seriously … how many people do you know who could start a profitable business if you loaned them a million dollars?

But then again, if these people aren’t lifting themselves out of poverty, why and how are they borrowing money and paying it back? What’s really going on here?

The microlending that happens: credit as basic need

Here’s a paper you’ll be hearing more about in future posts. It’s a review of many studies of the actual impact of microfinance on poverty. Pages 17-20 describe one of the more rigorous attempts at this, and the debate over what it showed; the one thing all three of the studies agree on is that people with credit available had smoother consumption: “household consumption increased most during the lean Aus season, when the poor often go hungry.”

I don’t know about you, but I’ve always taken my ability to smooth consumption (saving, borrowing, etc.) so much for granted that it didn’t even occur to me what it would be like to live without it. But as this paper argues in great detail, the world’s poorest often are less badly hurting for food/medicine than they are for the most basic support networks and mechanisms we use to manage our lives. If your income is seasonal and you’ve got no bank, forget about starting a business – you can’t even plan for the next week.

In this context, giving loans isn’t about creating and expanding ventures, it’s about meeting a basic need that might be as vital as the classic food, water, and shelter: the ability to manage risk and plan for changes. Loans aren’t the key to ending poverty, but they’re one of many things that we have to make available.

So I’ve done it, I’ve killed Santa Claus. Microloans won’t end global warming and baldness; they’re just one more way of helping people, along with bednets, condoms, and water purification. Donors, how do you feel? Disgusted? Disillusioned? Ready to swear off charity for good? Or interested in learning more?

Thanks to Tim of Philanthropy Action for discussing this issue with me and pointing me to two of the papers above (Dichter; Banerjee/Duflo).