…a doctor who is just out of medical school is not a True Expert because although they can give medical advice, they have no track record of helping people get better.

However, I would submit that the track record is not as important as the ability to do something which can be demonstrated. For instance, a doctor who is just out of medical school in my opiniioin IS a True Expert because they still have a lot of independent information about medical ailments and what treatments work (i.e. OTHERS have used the same treatments to get people better).

I think the definition should be….

1)The person has learned something EITHER from experience or through the study of the experience of others
2)This translates to a demonstrable ability to do something.

I don’t think anyone would tell you that helping people is easy, if you asked them point-blank (and were specific about the issues: education, global health, etc.) But I do think there’s an implicit assumption on the part of those who don’t think about these issues very hard. The obsession with “minimizing overhead” seems to be a reflection of this mentality. Anyway, it doesn’t matter too much. We agree that helping people is hard.

The evaluation resources I provided are indeed abstract. They are academic sources that provide conceptual underpinnings for evaluation. I just wanted to make the point that evaluation has a long history that would inform and enhance your project.

I also want to point out that the Big Brothers Big Sisters study did indeed made strong claims for the effects of BBBS mentoring on the life outcomes of the kids who participated in the study as part of the “treatment” group. What is important about the study is that is was randomized, which is difficult to achieve when one is studying the outcomes of interventions on people (since in order to conduct a randomized study there has to be a control group of kids who are not given the benefit of the intervention) and really expensive! Mentoring reduced by almost half the rate of first-time drug and alcohol use and reduced school absentism by half. At any rate, I encourage you to revisit that study, because I think it’s an example of exactly what you are promoting here.

Nobody is suggesting that people donate randomly. I think I was put off a bit by an entry you wrote some time ago, where you stated that most people think improving people’s lives is easy. But now I’m not sure you actually believe that–maybe you were just trying to stir the pot. It would be fanatastic if you could help create a climate that encourages foundations and donors to fund more evaluation studies and develop more systematic reporting and translation mechanisms so that donors could easily access and understand them.

*I tried to answer your point about tone on the “Apples and Oranges” comment thread. Basically, it is true that my usual tone toward foundations is highly critical and obnoxious, but this is because I’m usually focused on their opacity. I am much more agnostic about the quality of their analysis, but wanted this post to distinguish between being agnostic and assuming that they’re good just because they’re “experienced.”

*I agree with you that we don’t know much and that it would be much better to get the opinions of subject matter experts. I also agree with you that it is hard to find good stuff out there that relates directly to the question of where to give. We have certainly tried, but have often found relevant literature so focused on policy (again relating to my belief that most people think of charity as either unimportant or easy, when it is neither) that we have found it more productive to focus on a particular question and find what’s written on that question. Basically, the Africa expert’s guide to what kind of charity works in Africa isn’t something we’ve been able to find. If we could, we would draw on it heavily.

Michael/Carl: I may be overstating the case for the value of “expertise.” I don’t feel strongly about this or feel that it is important for me to feel strongly. The attitude of GiveWell is that seeing no one else asking our questions or doing what we’re doing, we are going ahead with it until and unless someone convinces us that it is a bad idea. This is consistent with maximum skepticism about experts, or just 50% skepticism.

Andrea: as others mentioned on the apples & oranges thread, your resources seem to be more about evaluation in the abstract, rather than actual studies of what works. This isn’t necessarily a bad thing, but it doesn’t leave me feeling much more informed about whether foundations are to be trusted.

I have read the Big Brothers/Big Sisters study. My vague recollection is that it focused on attitudes and did not give any analysis of the connection of these measures to life outcomes, so I ended up not knowing much about what to make of it. I didn’t spend much time on it, though, as Big Brothers / Big Sisters didn’t apply for our grant.

Steve B, we checked out EMCF quite some time ago and didn’t get very far, but now I am hearing buzz about their newer initiatives. I will check out their website again when I get a chance.

Thanks to everyone for commenting. The quality of dialogue on this blog has improved a lot lately. I’d like your thoughts on anything I can do to keep you all around and keep the discussion going. Should we move these kinds of discussions to a more robust forum system (such as the one we’re experimenting with at discussion.givewell.org)?

An ROI measure is relatively easy to get in a for profit environment, where the bottom line is actually simply dollars. There is no easy way to do what you want to do in an environment where human beings are attempting to improve the human condition. It is as complex as any social science research, which is so difficult that some people don’t even think social science is real science at all. If there were an easy way to evaluate and compare nonprofits, it would have been done. That it hasn’t been to the extend you advocate for does not imply incompetence or ignorance. But your attempts to develop comparable measures and the dialogue is certainly welcome.

There is an extensive evaluation literature, and professional organizations that connect practitioners from all over the world.

Examples:
http://ioce.net/index.shtml
http://www.campbellcollaboration.org/index.asp
http://www.eval.org/

Couple of books:
Evaluation: A Systematic Approach by Peter H. Rossi, Mark W. Lipsey, Howard E. Freeman
Qualitative Research & Evaluation Methods by Michael Quinn Patton
Responsive Evaluation: New Directions for Evaluation (J-B PE Single Issue (Program) Evaluation by Jennifer C. Greene (Editor), Tineke A. Abma (Editor)

Foundations and Evaluation: Contexts and Practices for Effective Philanthropy, eds. Marc T. Braverman, Norman A. Constantine, Jana Kay Slater, August 2004

And see:
http://www.ppv.org/ppv/youth/youth.asp
PPV did a huge randomized national level study of Big Brothers Big Sisters that was excellent, which BBBS has links to on its website, and you can see from reading about it what a major undertaking such evaluations are.

That said, I agree with the real thrust of Holden’s argument. In fact, I actually think he understates the negative impact of the current state of the foundation world. Beyond doing a relatively poor job of picking winners, the current structure has two additional pernicious effects:

1) Foundations distort non-profit leadership:
To be “successful” as a non-profit leader means you have to be successful at raising money from foundations, which means you either (1) have the charisma necessary to convince foundation leaders to give you money or (2) have personal connections with these leaders. Unfortunately, neither of these qualities necessarily translates into the ability to effectively lead an organization. Which means the non-profit sector gets stuck with too many executive directors who are good fundraisers but ineffective organizational leaders.

Plus, whether these leaders or ineffective or effective, it becomes more difficult for non-profits to transition organizational leadership, because they cannot risk losing the people who have developed personal relationships with their funders.

2) Foundations encourage people to leave the non-profit field:
The nicest office environments I have ever been in are foundations. And the people who work in these offices tend to get paid significantly more than their counterparts who are working in the non-profits they fund, despite working fewer hours in less intense jobs. If you work in the non-profit world, I think this is incredibly disheartening. And I think it leads to people questioning why they remain in the non-profit sector and choosing to leave to work in foundations or the private sector.