We’ve written a heck of a lot about what “measurements” should and shouldn’t be applied to charitable work – that’s what we spend a lot of our time thinking about, seeing as how we’re trying to figure out where to donate and all. Here’s a roundup of what we think. It’s a long post, but there’s candy for you: we offer up our actual, defined, concrete metrics for you to look at and think about, rather than sticking to abstract thoughts about whether you can quantify philanthropy (as I predict most others will).
First, though, the abstract stuff. Like many others, we are concerned about the point where measurement tries to do too much. Improving people’s lives is complex and difficult; you can never really be sure of what you’re accomplishing; and there are philosophical decisions to be made as well. An overambitious metric risks a conclusion like “Building a new charter school in New York City has a GORP (Good over RePlacement) of 17.3, whereas distributing medication to those with AIDS in Africa has a GORP of 18.5 – I think it’s pretty obvious where to donate, no?” More on this pitfall (which we’ve seen a lot of) here.
We also hate metrics that do too little. I’m constantly amazed at the way people will accept any ranking that someone cares to throw together, regardless of whether it makes any sense whatsoever (see the U.S. News and World Report rankings of just about anything, as well as this flagrant violation of common sense that didn’t so much as give me an honorable mention). The charity version of this is what we call the Straw Ratio, a seductively easy-to-calculate metric that is roughly as helpful in deciding between the best charities as this link is. I’ve written no less than 10 posts on why this metric, featured by Charity Navigator among others, is the worst thing since, uh, unsliced bread.
But to let the matter end there is unacceptable. There are too many options, and there is too much at stake, to throw up our hands, as I argued here. Metrics can’t do everything, but they have too much to offer for us to abandon them. We need to figure out what our goal is in donating, and measure what we can. So, enough generalizations. If we had all the information, the following would be our ultimate measures of charities within each of our seven areas of focus.
Cause 1: Provide for basic human needs including basic health care, food, water, and shelter.
Ideal metric: number of people for whom all such basic needs are met, and wouldn’t have been met without the charity’s activities, per dollar per year.
Cause 2: Fight epidemic curable/treatable diseases, including malaria, diarrhea, tuberculosis, AIDS, measles, and pneumonia.
Ideal metric: number of people who are alive and functioning, but would have been killed or debilitated without the charity’s activities, per dollar per year.
Cause 3: Enable economic opportunity through microcredit, job assistance and training, and education.
Ideal metric: number of people whose jobs produce the income necessary to give them and their families a relatively comfortable lifestyle (including health, nourishment, relatively clean and comfortable shelter, and some leisure time), but would have been unemployed or working completely non-sustaining jobs without the charity’s activities, per dollar per year. (Systematic differences in family size would complicate this.)
Causes 4-7: remove barriers to opportunity in wealthy societies, focusing on New York City.
Cause 4: Provide for basic human needs including basic health care, food, and shelter.
Ideal metric: number of people for whom all such basic needs are met, and wouldn’t have been met without the charity’s activities, per dollar per year. Note that this cause remains philosophically distinct from Cause 1, because living like this in the developed world is a fundamentally different experience – and means different things to different donors – relative to living like this in the developing world.
Cause 5: Aid youth development (pre-high school) through after-school activities, child care programs, etc.
Ideal metric: number of children who enter high school with normal levels of learning abilities and mental health, but wouldn’t have without the charity’s activities, per dollar per year.
Cause 6: Improve educational opportunities at the high school level through charter schools, summer schools, and public school reform.
Ideal metric: number of children who graduate from high school well equipped for college (as demonstrated by later college grades), but wouldn’t have done so without this charity’s activities, per dollar per year.
Cause 7: Enable economic opportunity through microcredit, job assistance and training.
Ideal metric: number of people whose jobs produce the income necessary to give them and their families a relatively comfortable lifestyle (including health, nourishment, relatively clean and comfortable shelter, some leisure time, and some room in the budget for luxuries), but would have been unemployed or working completely non-sustaining jobs without the charity’s activities, per dollar per year. (Systematic differences in family size would complicate this.)
OK. On one hand …
I think these metrics rock. There is a “click” for me when I read them – “Yeah, that’s what I want out of this charity! Yeah, that matches with common sense! That’s right – if Group A and Group B are both doing microfinance, and if it can actually be shown [forget for the moment that it can’t be] that $1000 leads to 3 sustainably employed people through A and only 2 through B, I feel good about donating to A!” This quality in a metric is far from given, and I think the key is making sure that everything is measured in people fully served. We make no attempt to make a conversion factor between someone whose life improves a little and someone whose life improves a lot – that factor would be arbitrary and would lead to numbers that don’t have clear meaning. Instead, when we start having to decide between fundamentally different ways of improving people’s lives, we stop comparing charities. This way, we can measure everything in terms of people, not any abstraction.
And I think it’s cool and useful to have these metrics. They help to focus our thoughts, when we’re trying to evaluate charities, and the fact that they’re well-defined may explain why we tend to have a more detailed idea of what we want than other donors do. We know that when we’re talking to a microfinance organization, it isn’t enough to see $ loaned – we know we need to know how many loans were made, how many were paid back, and (if possible) what ultimately happened to the borrowers; and we know how to weight these things and combine them into a sense of what ultimately got accomplished. So, go us.
On the other hand … we’ll never actually calculate a single one of these things.
Oh, we might estimate them, very, very loosely. We never have yet, because we’ve never even been in the ballpark of enough information and certainty. When we’re looking at public school reform, for example, it’s huge just to see that a program made any impact. Calculating how many lives were changed is a dream.
Once you start actually looking at charities’ specific activities, you find that there are no more generalizations to be made on this topic. We know what we’re aiming for (the metrics above), but what information actually ends up being relevant is completely case-specific. There’s no way to go about it except by combining analysis with judgment, common sense, intuition, and improvisation.
So, we welcome comments on our metrics and we continue to think about them, but we are also wary of hyperfocusing on them. We think that people who hyperfocus on metrics are putting too much faith in numbers and experts. Nobody will ever know for sure how many people they’re helping, and the estimation involves literally hundreds of judgment calls that no degree can qualify anyone to make. That’s why we want to put the focus squarely on blowing up the black box. We seek to be the first people ever – including any advisor or foundation you like – to publicly share everything that goes into our giving decisions. We think that’s the most valuable thing you could ever have in an evaluator.