Transparency is the one thing about GiveWell that everyone seems to like. Our focus on measurement is much more contested. I believe that the connection is tight, though, because both are necessary consequences of humility, which is probably the last word you ever thought I’d ask to be associated with.

Transparency is a really big deal to us because we believe that no matter how much we learn and no matter how hard we work, we can always be wrong. That’s why we invite as many people as possible into the conversation.

(When I look at large foundations making multimillion-dollar decisions while keeping their data and reasoning “confidential” – all I see is a gigantic pile of the most unbelievably mind-blowing arrogance of all time. I’m serious. Deciding where to give is too hard and too complex – with all the judgment calls and all the different kinds of thinking it involves, there is just no way Bill Gates wouldn’t benefit from having more outside perspectives. I don’t care how smart he is.)

Measurement is about inviting someone else into the conversation: The Facts. The Facts have a lot to say, and they often contradict what we would have thought. That’s why we have to listen to them. Like transparency, measurement takes a lot of extra effort and expense; like transparency, it can’t solve all your problems by itself; and like transparency, it’s easily worth it if you agree that the issues are extremely complex, and that no matter how much sense something makes in your head, The Facts might disagree.

(And yes, when I hear people talking about how they’re “too busy helping people” to measure, I hear arrogance there too. The only way you could ever take that position is if you are so sure of what you’re doing that you think learning more about it isn’t worth the 10% more you can spend doing the same untested thing. How high on your own infallibility do you have to be to come to that conclusion?)

To a lot of people, humility means speaking with a certain tone of voice, or just plain keeping your mouth shut. If that’s what you think humility is, we don’t have it. To us, humility is constantly saying “The things that make sense to me could be wrong – that’s why I’m going to do everything I can to test them, against others’ ideas and against reality.” Instead of being silently dissatisfied with charity, we’re loudly dissatisfied, so that anyone who disagrees can respond. Instead of happily assuming our dollars are doing good, we demand to see The Facts, so they can respond too.

Laugh if you must, but in the end humility is the defining value of the GiveWell project. We’re not here to impose our solutions on the sector; we’re here because we want to see more questions.

Audio for last Monday’s board meeting is on the way; in the meantime, here’s a summary. The meeting ran about five hours and had heated arguments, tension, drama, a couple car chases, and down-to-the-wire votes. The highlight was the Elie and I ended up reversing our position on 2 of the 3 causes we voted on. Here’s the story.

Employment assistance: HOPE Def. Year Up in Double OT

Elie and I walked into the meeting having settled on a basic judgment call: we preferred high-skill to low-skill job training, for a variety of reasons. The Board (particularly Greg and Bob) questioned all of these reasons heavily.

• We argued that without good comparison-group data, we couldn’t really use empirical evidence to establish whether a charity was making clients much better off than they would be without its help. Therefore, we preferred charities that were clearly teaching a highly specific skill – something that clients would clearly need training in – rather than working on improving motivation, which seems much harder to change in a 12-month or shorter program. By contrast, Greg and Bob felt that the easiest thing to help a person with is neither motivation nor specific skills, but extremely basic things like how to present oneself for an interview – and that this is the sort of coaching they’re unlikely to be able to obtain without a nonprofit’s help.
• Our cost-of-living analysis suggested that lower-skill jobs, paying $10/hr, don’t represent a true living wage – and that they are unlikely to lead naturally to raises, whereas high-skill jobs allow a reasonable and improving standard of living. Greg and Bob were of the mind that getting a first job is the “hardest hump to get over,” and that someone with a job can find a way to get further training and move up, without needing an intensive nonprofit-run program.

The frustrating thing is that everyone in the room was completely guessing on all of this, and I feel that the right kind of evaluation could settle these questions. We voted on this central question – high- vs. low-skill wages – and initially, high-skill won 3 votes to 2, and we prepared to vote between the high-skill organizations Year Up and St. Nick’s.

But even though my instincts still lean toward high-skill training, my confidence was substantially weakened by Greg’s and Bob’s arguments, and it occurred to me more just what a huge tossup/judgment call this all is. With that in mind, I switched my vote, preferring to “break the tie” between approaches by going with The HOPE Program, which is (a) the charity I have the highest confidence in as an organization, due to its impressive and thorough self-evaluation; (b) the charity whose effect is best supported by the (admittedly sketchy) empirical evidence we have.

The rest of the Board agreed with my way of framing the new decision, and after walking in expecting to award Year Up, we unanimously elected HOPE as the winner of our grant.

Saving lives in Africa: PSI 3, PIH 2

All of us agreed that Population Services International and Partners in Health appear to be truly excellent organizations, with dramatically different strengths and weaknesses. On one hand, PSI is a standout in terms of being able to track and document itself, and appears to conduct extremely large-scale, cost-effective activities that save huge numbers of lives. PIH is far inferior by the “dollars per life saved” metric, but of course there’s so much more to the story than that. Their model of providing comprehensive health care is extremely appealing because it is straightforward and community-based, and therefore (in my mind) a much more reliable way of ensuring that we’re truly addressing disadvantaged people’s needs.

This argument got heated and it got impolite. I took PSI’s side and accused Lucy and Tim (PIH defenders) of avoiding my arguments by using “jargon,” and focusing on PIH’s “revolutionariness” instead of just trying to figure out who’s going to help as many people as much as possible. They, in turn, accused me of being too naive to understand PIH’s awesomeness, and they may be right (though for the record, I do think PIH is awesome and would defend it over any other organization I know of but PSI, including many organizations that annihilate it on the “cost per life saved” metric).

Greg, Bob and I carried PSI over Tim and Lucy. Yes, the three outsiders outvoted the two more experienced people. Yes, this is a concern, and yes, I still feel far from 100% on the decision and still think PIH is great. But as of today, I’d still rather grant PSI for a variety of reasons, and like any donor, I’m going with what makes sense to me. All five of us agreed that it was a tough call. The conversation will continue.

Global poverty: Opportunity International rallies from 1st-quarter deficit

Another one on which I completely flipped my position on a key judgment call. Elie and I came in having read all the literature we could possibly find evaluating microfinance programs, and we’d concluded that there is reason to believe that microfinance helps people – but the evidence isn’t nearly as compelling or rigorous as generally believed. We argued that there are just too many questions about when and how microfinance helps people – and that none of the microfinance organizations we’d evaluated were able to give us good information on when and how their programs worked. We therefore preferred to go with something simpler, more concrete, and more tangible: KickStart, which markets a water pump that directly improves farm production.

Greg and Tim agreed with us about the problems (and how little is known) with microfinance, but challenged the idea that a pump is more reliable just because it is more tangible. Both are convinced that there is not enough available credit in the developing world (something I don’t have as strong an opinion on); and they pointed out that the mere fact someone repays a loan generally tells you they used it for something reasonable and presumably life-improving, even if not to create a thriving new business.

I found their arguments compelling. We all hesitated to grant a microfinance program because we don’t have nearly enough information about any of them, and they – of all organizations – should be able to provide hard data, since they essentially are banks. But ultimately, we went with Opportunity International, which had given us the best picture of what was going on in their programs. This is a cause we need to do a lot more work on, but we all agree that it’s best to give today (based on a best guess) and keep learning – not hold our charity until we’re 100% confident.

Other causes: coming soon

Despite the wish to give today, we simply need to do more work in Causes 3 and 4, so we put those off.

People get tired and cranky

Lucy and Tim were sick of jabbering, and asked that we cover only the things we had to cover for the rest of the meeting. So we skipped the Progress Report on Year 1 (good thing, because the attention we got later in the week has altered the report substantially), approved the 2007 budget retroactively, and pulled up our salaries for approval through the next Board meeting. Lucy said she would approve these salaries only on condition that we commit to discussing (at the next meeting) what metrics GiveWell will use to evaluate itself. She is insistent that we apply the same high standard of metrics and evaluation to ourselves that we want to apply to other charities. She is right, and we are working on it.

The moral of the story

Deciding where to give was exhausting, demanding, and affected significantly by the presence of different perspectives, from Greg’s financial/economic intelligence to Lucy’s and Tim’s expertise and experience. My #1 thought coming out of it all is that we need more of these heated, diverse conversations, and we need them to be happening between more different kinds of people and in more public settings. Making the world a better place is not easy; it deserves to be argued about as intensely and critically as everything else important that we do.

The things I’m proudest of about this meeting:

1. That I changed my mind on two key issues (and ended up switching my vote on 2 of the 3 causes). I feel that it always takes vigilance to keep an open mind, especially when arguing over something I’ve been working on and convincing myself of for months, so it’s always good to be monitoring whether I am operating in a manner consistent with open-mindedness.

2. That Board members were able to challenge our decisions in an informed way. The Board didn’t have access to any special materials (except on global poverty, which isn’t on the website yet but will be). They just read givewell.net , and doing so allowed them to understand the key issues enough to truly apply their own critical thinking. That’s what GiveWell is really about.

6 months ago, GiveWell pledged to give a $25,000 grant to the best organization we found in each of our five causes, and we’re going to follow through. But, I wish we didn’t have to.

We recognized that donors are more likely to trust us for some decisions (comparing organizations with similar goals) than for others (for example, trying to compare the value of saving a child’s life in Africa to that of helping improve a child’s education in the New York City). That’s why we divided charities into different causes, with the aim of giving one grant in each cause.

Having learned what I have about organizations that aim to raise incomes in New York City and those that aim to save lives in Africa, I would not donate to any of our applicants aiming to raise incomes in New York. I’d donate instead to Population Services International, our top choice for saving lives in Africa. I have two reasons for this:

1. The relative cost of the programs.
2. The logic underlying the approach.

1. Relative cost. The organizations we recommend – the ones in which we have the most confidence – spend $10,000-50,000 to help an unemployed/under-employed person become self supporting. For $50,000, I’d expect to save at least 50 lives in Africa by donating to Population Services International. There are reasonable arguments in favor of improving lives in the developed world – there’s value in helping people “get over the hump” that’s preventing them from living a fullly enabled otherwise happy life – and our “lives saved” number is of course a simplification that leaves out a lot. But these are not in the same ballpark. I think that the ratio of people helped (10-50:1) is just too much.

2. The logic underlying the approach. I don’t understand the “theory of change” (or, the basic logic behind the intervention) that explains the outcomes that Employment Assistance charities achieve. Are some basic soft skills (resume writing, interviewing tips, etc.) preventing their clients from finding employment? Do their clients need specific vocational skills (computer technician, environmental remediation technician, etc.)? We’ve wondered if it makes more sense to run job-training programs or just give the people money to support their basic needs, or otherwise, provide directly for their needs. In Africa, I understand the logic behind the interventions much more clearly (for example, selling insecticide treated nets that prevent mosquitoes from biting children, and thus prevent the transmission of malaria).

I end up feeling like a donation can make a huge difference in many people’s lives, with high confidence (through our top Cause 1 organizations) – or might make a difference in someone’s life, with low confidence and an unclear understanding of the logic (through our top Cause 5 organizations).

Our aim at GiveWell is to help donors find great charities to donate to by identifying the best charities in a cause (in which we can compare apples to apples). Our goal isn’t directly to choose their causes for them because the choice of apples versus orgnges is often a question of personal taste. But we hope that as we show donors more concretely what they get for their dollar, that will affect what causes they choose too. Speaking very broadly and roughly, imagine that for $25,000 you could save 25 lives in Africa or help 1 person get a better job in New York City. I know how I’d choose. What would you do?

I want to take the time to explain a couple views I hold that offend some people. These views are:

• The label “philanthropy expert” means very little to me.
• I see no strong reason to believe that charitable foundations are generally capable, competent, or effective at anything at all.

These aren’t insults aimed at particular people, and they aren’t hyperbole intended to generate controversy. They’re opinions I hold based on my observations about how philanthropy is structured. I can explain them. Both of these beliefs come from the fact that I have no way of determining whether the “respected experts” in philanthropy represent a set of True Experts, or an old boys’ club.

True Experts

True Experts have two qualities. (1) They have learned a lot from experience. (2) This translates to a demonstrable ability to do something.

Regarding (1): Experience can be educational … or it can not be. It’s educational when it consists of trying something, learning whether it worked, and trying something else, all in an environment where “what works” is fairly stable. A highly experienced glovemaker will know all kinds of tricks of the trade and have all kinds of instincts that a young but “naturally talented” glovemaker doesn’t.

In the world of investing, by contrast, it’s much less safe to bet on the more experienced person. The environment is so complicated and the results so random that people can spend 50 years day-trading without ever learning from it. Writing music is even more this way: I don’t think there’s any strong connection between a musician’s level of experience and their ability to write great music (if anything the relationship is negative).

Now imagine an activity that consists of investing without looking at your results. In other words, you buy a stock, but you never check whether the stock makes money or loses money. You never read the news about whether the company does well or does poorly. How much would you value someone with this sort of experience – buying and selling stocks without ever checking up on how they do? Because that’s what “experience in philanthropy” (or workforce development, or education) comes down to, if unaccompanied by outcomes evaluation.

The peculiar thing about philanthropy is that because you’re trying to help someone else – not yourself – you need the big expensive study, or else you literally have no way of knowing whether what you did worked. And so, no way of learning from experience. I wish I had an analogy for this but I don’t. There is no analogy. Philanthropy is the one sphere where you don’t get any info about your effects unless you explicitly study them.

Regarding (2): In most fields, “expert” doesn’t just mean “experienced” – it means “proven.” Tons of people have been investing their whole lives and are still dreadful at it, worse than a bright young 18-year-old would be. But Warren Buffett has a strong track record of investment success; he’s a True Expert in investing. On a smaller scale, your local doctor hopefully has a track record of helping people get better. OK, there’s no randomized-design evaluation of him, but we have a lot of independent information about how medical ailments progress if untreated, and the people your doctor treats get better instead. Unlike a Middle Ages leech supplier, your doctor is a True Expert in medicine.

As an aside, it’s pretty rare for a True Expert to refuse to share their knowledge. Even in the hypercompetitive finance industry, Warren Buffett basically tells you how he thinks because he knows you can’t replicate it. In more world-serving fields such as academia, every expert’s beliefs are peer reviewed, published, and available to any schmoe with a library card.

What am I to make of the statement that Person X is a respected expert in philanthropy? It could mean that they have a track record of solving social problems, and I just haven’t seen the evidence due to a lack of information sharing. Or it could mean …

Old boys’ clubs

We’ve all seen old boys’ clubs. They’re groups of people who think alike, act alike, and have some sort of power or privilege that they don’t have to continually earn.

I’ll use the example of a typical college fraternity. Nominally, the fraternity has a mission, something about “upholding honor and serving the community” or something. Nominally, the President is the person most qualified to help the fraternity promote this mission, the Treasurer is the best person to keep the books, etc. But really? The President is the most popular guy. The Treasurer is ~4th-most popular. The guy who slept with the President’s girlfriend is at the bottom of the totem pole. Etc.

These clubs’ members usually have a lot of weird, idiosyncratic beliefs about the world, because of how similar they are to each other and how little they check these beliefs with the outside. You can usually bet that the “accepted wisdom” at these clubs plus $2 is worth about one subway ride. You can usually not bet that the “top people” at these clubs are better in any way than the others. And it’s not clear that Louis XVI and his cronies did, in fact, run France better than a 26-year-old punk might have.

Foundations

The thought that much of the world’s well-meaning money is managed by old boys’ clubs is scary – scary enough to make one want to deny it, even without evidence. But the fact is, everything about the way foundations are structured is consistent with the idea that they may be old boys’ clubs. I have no evidence that they do the measuring necessary to learn from experience. (They might, and they might not.) I have no evidence that they interact meaningfully with people outside their bubble. (They might, or they might not.) I have no evidence that they have made the world better off, consistent with their mission. (They may have. They may not have.)

I have no evidence that their power and privilege are connected in any way with performance or merit. Unlike with a private company, the fact that they have money means only that someone gave them money decades ago.

The only thing I do know is that they don’t share their information as much as would be consistent with devotion to a better world. That’s why I have a negative view of them.

I’m not committed to this view. Maybe foundations do have ways of learning from their activities. Maybe they do have a track record of success. Maybe they know far more about philanthropy than GiveWell can ever hope to. That’s definitely possible.

So which is it? Foundations, are you an old boys’ club – or can you demonstrate that you’re True Experts?

I am proud to announce the winner of the 2007 Holden Award for Excellence in Imperfection. This award has a rich history dating back to 3 minutes ago. The rules are as follows:

The first charity to make a negative correction to its GiveWell review wins the award.

Since we’ve published our review drafts, we’ve gotten many corrections – “you’ve underestimated ___ advantage of our organization,” “you’ve left out how great it is that we ___,” etc. Most of these corrections have been correct or at least worth considering, and I’m glad we’ve gotten them. But, not until today did someone write in, “You’re overstating our lives saved per $.” For that, this charity wins the 2007 Holden Award for Excellence in Imperfection, in recognition of its honesty, humility, and recognition that getting the truth out there is more important than looking good.

The award comes with a $500 donation from Holden (not from The Clear Fund). This means a lot now that I no longer make a hedge fund salary.

Your winner: Population Services International, for sending us downward revisions of its own estimates of HIV/AIDS infections averted. Well there’s a shocker for you. The very same organization that has demonstrated the strongest commitment to monitoring and evaluating itself is also the first one to ask us to remove information that exaggerates its impact! Maybe monitoring and evaluation really do go hand in hand with humility. Would you guess that combination also goes hand in hand with effectiveness?