We’ve written a heck of a lot about what “measurements” should and shouldn’t be applied to charitable work – that’s what we spend a lot of our time thinking about, seeing as how we’re trying to figure out where to donate and all. Here’s a roundup of what we think. It’s a long post, but there’s candy for you: we offer up our actual, defined, concrete metrics for you to look at and think about, rather than sticking to abstract thoughts about whether you can quantify philanthropy (as I predict most others will).

First, though, the abstract stuff. Like many others, we are concerned about the point where measurement tries to do too much. Improving people’s lives is complex and difficult; you can never really be sure of what you’re accomplishing; and there are philosophical decisions to be made as well. An overambitious metric risks a conclusion like “Building a new charter school in New York City has a GORP (Good over RePlacement) of 17.3, whereas distributing medication to those with AIDS in Africa has a GORP of 18.5 – I think it’s pretty obvious where to donate, no?” More on this pitfall (which we’ve seen a lot of) here.

We also hate metrics that do too little. I’m constantly amazed at the way people will accept any ranking that someone cares to throw together, regardless of whether it makes any sense whatsoever (see the U.S. News and World Report rankings of just about anything, as well as this flagrant violation of common sense that didn’t so much as give me an honorable mention). The charity version of this is what we call the Straw Ratio, a seductively easy-to-calculate metric that is roughly as helpful in deciding between the best charities as this link is. I’ve written no less than 10 posts on why this metric, featured by Charity Navigator among others, is the worst thing since, uh, unsliced bread.

But to let the matter end there is unacceptable. There are too many options, and there is too much at stake, to throw up our hands, as I argued here. Metrics can’t do everything, but they have too much to offer for us to abandon them. We need to figure out what our goal is in donating, and measure what we can. So, enough generalizations. If we had all the information, the following would be our ultimate measures of charities within each of our seven areas of focus.

—————

Ideal metricsCauses 1-3: aid the poorest of the poor, focusing on Africa.
Cause 1: Provide for basic human needs including basic health care, food, water, and shelter.
Ideal metric: number of people for whom all such basic needs are met, and wouldn’t have been met without the charity’s activities, per dollar per year.
Cause 2: Fight epidemic curable/treatable diseases, including malaria, diarrhea, tuberculosis, AIDS, measles, and pneumonia.
Ideal metric: number of people who are alive and functioning, but would have been killed or debilitated without the charity’s activities, per dollar per year.
Cause 3: Enable economic opportunity through microcredit, job assistance and training, and education.
Ideal metric: number of people whose jobs produce the income necessary to give them and their families a relatively comfortable lifestyle (including health, nourishment, relatively clean and comfortable shelter, and some leisure time), but would have been unemployed or working completely non-sustaining jobs without the charity’s activities, per dollar per year. (Systematic differences in family size would complicate this.)
Causes 4-7: remove barriers to opportunity in wealthy societies, focusing on New York City.
Cause 4: Provide for basic human needs including basic health care, food, and shelter.
Ideal metric: number of people for whom all such basic needs are met, and wouldn’t have been met without the charity’s activities, per dollar per year. Note that this cause remains philosophically distinct from Cause 1, because living like this in the developed world is a fundamentally different experience – and means different things to different donors – relative to living like this in the developing world.
Cause 5: Aid youth development (pre-high school) through after-school activities, child care programs, etc.
Ideal metric: number of children who enter high school with normal levels of learning abilities and mental health, but wouldn’t have without the charity’s activities, per dollar per year.
Cause 6: Improve educational opportunities at the high school level through charter schools, summer schools, and public school reform.
Ideal metric: number of children who graduate from high school well equipped for college (as demonstrated by later college grades), but wouldn’t have done so without this charity’s activities, per dollar per year.
Cause 7: Enable economic opportunity through microcredit, job assistance and training.
Ideal metric: number of people whose jobs produce the income necessary to give them and their families a relatively comfortable lifestyle (including health, nourishment, relatively clean and comfortable shelter, some leisure time, and some room in the budget for luxuries), but would have been unemployed or working completely non-sustaining jobs without the charity’s activities, per dollar per year. (Systematic differences in family size would complicate this.)

—————

OK. On one hand …

I think these metrics rock. There is a “click” for me when I read them – “Yeah, that’s what I want out of this charity! Yeah, that matches with common sense! That’s right – if Group A and Group B are both doing microfinance, and if it can actually be shown [forget for the moment that it can’t be] that $1000 leads to 3 sustainably employed people through A and only 2 through B, I feel good about donating to A!” This quality in a metric is far from given, and I think the key is making sure that everything is measured in people fully served. We make no attempt to make a conversion factor between someone whose life improves a little and someone whose life improves a lot – that factor would be arbitrary and would lead to numbers that don’t have clear meaning. Instead, when we start having to decide between fundamentally different ways of improving people’s lives, we stop comparing charities. This way, we can measure everything in terms of people, not any abstraction.

And I think it’s cool and useful to have these metrics. They help to focus our thoughts, when we’re trying to evaluate charities, and the fact that they’re well-defined may explain why we tend to have a more detailed idea of what we want than other donors do. We know that when we’re talking to a microfinance organization, it isn’t enough to see $ loaned – we know we need to know how many loans were made, how many were paid back, and (if possible) what ultimately happened to the borrowers; and we know how to weight these things and combine them into a sense of what ultimately got accomplished. So, go us.

On the other hand … we’ll never actually calculate a single one of these things.

Oh, we might estimate them, very, very loosely. We never have yet, because we’ve never even been in the ballpark of enough information and certainty. When we’re looking at public school reform, for example, it’s huge just to see that a program made any impact. Calculating how many lives were changed is a dream.

Once you start actually looking at charities’ specific activities, you find that there are no more generalizations to be made on this topic. We know what we’re aiming for (the metrics above), but what information actually ends up being relevant is completely case-specific. There’s no way to go about it except by combining analysis with judgment, common sense, intuition, and improvisation.

So, we welcome comments on our metrics and we continue to think about them, but we are also wary of hyperfocusing on them. We think that people who hyperfocus on metrics are putting too much faith in numbers and experts. Nobody will ever know for sure how many people they’re helping, and the estimation involves literally hundreds of judgment calls that no degree can qualify anyone to make. That’s why we want to put the focus squarely on blowing up the black box. We seek to be the first people ever – including any advisor or foundation you like – to publicly share everything that goes into our giving decisions. We think that’s the most valuable thing you could ever have in an evaluator.

I’ve done an arguably excessive amount of ranting on what I’ve dubbed the “Straw Ratio”: the idea that we can find the best charities by seeing what percentage of their total expenses goes to program expenses, or, in Straw Man parlance, goes “straight to the people who need it.” It’s about time for me to stop devoting every post to this, but before I do, I want to wrap up with why I’m so obsessed with this issue.

To me, the Straw Ratio is more than a bad metric: it’s an emblem of the biggest single problem with charitable giving today. The problem is the fallacy – specific to charity – that how much you spend is more important than how you spend it. That, in the end, is the fallacy behind the Straw Ratio: that what matters about a donation is not what it pays for, but how much it pays for.

It’s the same problem that sits behind the success of “microcharity” sites such as DonorsChoose and GlobalGiving and Kiva, which I will complain about more later – the attitude that getting your money straight to where it’s needed, bypassing administration and planning, is an exciting and intelligent way to give it.

It’s the same problem that sits behind some of the excessive restriction/designation of donations, both by individuals and foundations, which I will also complain more about later. Too many donors feel that if “someone else” pays for all the planning, evaluating, and administration (or if they simply aren’t paid for at all?), they’ve gotten more bang for their buck.

It might be the same problem that makes charities so reluctant to embrace transparency and admit that not everything they do is perfect – the obvious benefits of global dialogue, in terms of figuring out how to spend money, are offset by a fear that there will be less money to spend.

And in the end, it is probably the same problem that explains why donors are not smart about their giving, and nobody aside from GiveWell seems interested in helping them to become more so. From the (RED) campaign to the “Don’t Almost Give” campaign to, well, everything, “being charitable” is identified with “giving a lot.” The nonprofit sector is naturally obsessed with getting people to give more, and donors think that the only thing that determines how good you are at giving is how wealthy you are. That’s wrong.

The world’s needy don’t just need your dollars, they need your thoughts. If your friend gives more than you but gives it worse, this is pretty much the equivalent of their having a vastly superior tennis racket and being a vastly inferior tennis player. You are better. Why does that make so much sense in every context except charity, but sound so foreign here?

In all, I can think of a whopping two activities where money alone will buy you success. 1. Arcade games (the ones where you can continue as long as you have more quarters). 2. Strip clubs. Nothing else comes to mind.

Money can’t buy you a great movie. Money can’t buy you a great product. Money can’t buy you love. It can’t buy you a better world either.

Anyone who has ever had a job knows that throwing money at a problem is worthless, and that a single great idea can dwarf the impact of a bigger budget. Anyone who has ever watched a tiny startup business take down a Goliath knows the same thing. Anyone who has ever had a hobby knows that how much you spend is going to account for half, at most, of how good you are.

Anyone who has ever had 5 minutes of consciousness knows that intelligent thoughts are gold, and that next to them, money is cheap. So next time you give, expect your donation to help pay for salaries and self-evaluations. And when it comes to choosing where to donate, don’t use the Forbes list or the first slickster who sends you a picture of a baby. Put in some time, and donate some of your own intelligence as well. The world needs it.

If my extended, multi-post rant on the Straw Ratio has been at all convincing to you, you should be concerned about what the Internet thinks “charity evaluation” means. I’m not just talking about the top result, but about the whole shebang.

Give.org’s criteria don’t say a single thing about the nature or effectiveness of a charity’s program activities – only its organizational policies and, of course, our friend the Straw Ratio (see “FINANCES”). The next link is referencing the American Institute of Philanthropy, whose ratings criteria will look familiar to fans of Charity Navigator’s: program expenses good, administrative and fundraising expenses bad, lots of assets/reserves good, content of programs irrelevant. Next one down gives a whole list of links to charity evaluators with more (or less) of the same.

Then you’ve got the bold Put your money where our math is (I’ll admit that’s a great pun), which says “Don’t worry. We’ve done your homework for you,” before laying out criteria for the charities that are “run the most efficiently” – criteria that, again, look mighty familiar, except these guys penalize charities for saving a lot where the above watchdogs reward them. (There are arguments both ways … and both arguments are pretty bad. I haven’t spelled out why I think how much a charity saves is a pretty crazy thing to look at when trying to help people … leave a comment if you want me to.)

And on and on … not to mention the ads for charities claiming to be “Top ranked in efficiency” and the like. One guess as to how they’re measuring that.

Not cool, Internet.

Measuring success at helping people is hard; it has inherent limits; it’s time-consuming; and it’s expensive. But it has to be done.

The first reason is that no matter how much sense an idea makes in your head, translating it to reality is another matter. I’d argue that acceptance of this basic idea is the single reason that we now have medical alternatives to prayer.

The second reason is that there are a a LOT of different charities out there for a donor to choose from – and without some sense of what they’ve actually accomplished, a donor has nothing to go on but theories and brochures. To me, that’s not much better than flinging our money randomly around the globe, with anyone who has a good story and a good accountant getting a chance to play. That isn’t a reasonable approach to solving the world’s problems.

The question of how to evaluate a given charity’s activities is a question for another day. It depends on the charity and it’s pretty much never easy. The question for today is, why isn’t it done more thoroughly and more often? And the answer for today (though it’s only one factor) is that funders don’t want to pay for it.

Children’s Aid Society has explicitly told us they’re concerned about funders’ reactions to the amount of their budget that is going to evaluation, “as opposed to” helping people – and that they’ve been unable to execute a major community school evaluation they’ve mapped out because it exceeds the “evaluation budget” designated in their grants. We asked New Visions for Public Schools why they don’t seem to have had the same problem, and they told us that they have, but that they’ve made a priority of fighting for larger evaluation budgets from their own funders. Over and over again, when we ask charities why they haven’t measured things that seem measurable, they’ve responded that the people who fund them don’t want it: many of their funds are often officially earmarked for non-evaluation purposes, and even when they’re not, they’re concerned about donors’ wishes to “get the money right to the people who need it.”

You and I are lucky that the people who build our bridges, cars, and airplanes don’t decide they can get more money directly to us if they cut their testing budget. The people served by nonprofits should be so lucky.

It’s very unlikely – maybe impossible – that the Straw Ratio bothers you as much as it bothers me. So I’m going to try to spill out all the ranting I have left on this topic over the next few posts, and then we’ll completely be done with it, at least until the next time I tell someone about our project and they go “Oh yeah, I’m really into giving intelligently. Like have you seen these websites that show you how much of your dollar REALLY goes to helping people?”, causing me to completely black out with rage and do things I can’t remember days later, like write incoherent blog posts or light Form 990s on fire. (FYI, this happens about 2x per day on average.)

So. My last post focused on one component of those evil, evil administrative expenses: salaries. Another common administrative expense that jumps to mind (although “administrative expense” is subject to some interpretation, as has been pointed out to me) is information technology. This is something I haven’t seen from the inside, but anecdotes about the generally poor state of nonprofit tech abound. Here’s a pretty good example: Today I Cried, an aborted (as of a few days ago) blog by a nonprofit IT director.

“I think I’m done with nonprofit. I’m looking at corporate – the bigger the better.”

“So much energy wasted on bull[bunnies].”

“Do you know what a modem-router is?”

Oy.

But the most relevant post here is Non-Profit Technology vs. For-Profit Technology. First overarching difference? “Money (restricted/unrestricted funding; discount pricing; used/donated technology; Tech Soup; technology grants)”. What do you know. Those dollars that you’ve so diligently been marking “FOR NEEDY CHILDREN ONLY” made it past that greedy tech department after all.

Ever try to do good work with bad technology?

Is that what you want to be funding?

Whoa, no way – young nonprofit sector workers don’t want to stay in the sector because of low pay? Young people don’t aspire to become nonprofit leaders? Well where are we going to get great leaders? We need great leaders at nonprofits if we want them to get great things done, right?

Man, that’s rough. Anyway, my attention span isn’t so hot, so I forget what I was just talking about, but you know what pisses me off? Charities that blow their money on fat salaries for their greedy executives!! Oh, man. Don’t get me started. At least I have a friend in Charity Navigator, which puts the CEO compensation right on that profile page so I can spot crooks like that Curtis R. Welling. Guy makes 275 THOUSAND DOLLARS (as much as a mid-level investment banker!) for running Americares, which barely cracks the top 20 largest nonprofits in the country. I mean, the AVERAGE nonprofit CEO makes ~$150k/yr – who needs that kind of money? That’s, like, entry-level MBA money!

Listen to me. I want my donation going to needy children, not to some jerk in a suit. I want great leaders in the nonprofit sector. I am confused.

Oh, actually, I think I’ve sorted it out. The problem seems to be that pesky Straw Ratio again, and the mentality that goes along with it: that we should evaluate charities by nitpicking their balance sheets and questioning their operating costs, rather than by looking at what they do and whether it works.