I wonder how much of the difference between our approach to charity and others’ approach comes from this very simple fact:

We think that improving people’s lives is really hard.

It might be easy to brighten someone’s day, even their week. But to get someone from poverty and misery to self-sufficiency and a world of opportunity … you have to change not just their resources, but their skills, often their attitude, and always their behavior. You can try to do this early in life or late; either way, it’s an uphill battle.

As we’ve been over and over, that’s why we think “money spent” is a poor proxy for “good accomplished,” and tend not to get as excited as others over dollars raised or dollars spent. But as we get deeper into reading apps, I’m finding there’s more and more to this rift.

I’m very skeptical of any program that claims great effects with relatively low amounts of intervention, whether it’s a one-time class on condom use or a once-a-week tutoring session. I think about how easy it is for the people I know to sit through some class, walk out full of ideas, and forget them a week later, and I think – if you’re doing anything meaningful for people with this little investment, you must be some sort of sorceror.

I trust survey data about as far as I can throw it (note: data cannot be thrown). Katya’s post today gives a sniff of the argument that “people are notoriously bad predictors of their behavior”; there’s actually an enormous amount of literature out there ramming home this point (if you really want to see it, let me know). We’ve seen a lot of survey data along the lines of “97% of participants felt the program improved their ability to ___,” which would be great if wishes were horses and horses were great.

I tend to feel the same way about anecdotes and personal experiences. They can be useful to get a picture of a program, but as evidence that it works? People are easy to change in the moment and incredibly hard to change in the long run. How can you possibly tell from watching a group of children for a year that they’re headed for better lives?

And in the end, though I keep reminding myself that evidence of effectiveness can come in all different forms, I can’t keep down that longing for randomized studies, or at least studies that employ some sort of comparison group. These are very rare in the applications we’re receiving: charities tend to look only at their clients. While this clearly saves them a lot of money and hassle, it leaves me wondering whether they’re really helping people … or just picking out the ones who are going to succeed anyway (you’re probably familiar with this question in the context of Ivy League schools). I’m sure this is heresy, but if we started a Placebo Foundation that sought out the “most motivated” poor people as clients, did absolutely nothing for them, and then examined their outcomes, would we find the same “successes” that many charities claim?

And yet every study I’ve read from applicants – even when limited to survey data, even when lacking a comparison group of any kind – concludes that the program in question was a success.

Unless there’s willful deception going on here, this implies to me that the people who work at charities think helping people is really, really easy; that there’s no need to worry about all the questions above; that the flimsiest and most perfunctory of evidence is good enough to walk away from feeling that people in need have been truly helped.

Maybe we’re wrong. Maybe helping people is that simple. If that’s what you think, keep writing those checks to whomever sends you mail, and make sure they aren’t blowing a penny more than they have to on salaries. If you’re concerned as I am, though, I’ve got some tough news from you: aside from GiveWell, I don’t think you have much company.