A couple questions for you. I think Cause 5 is going to come down to these questions.

1. Would you rather grant …

A. A program that helps severely unemployed/undermployed people, with barriers to employment including past convictions & drug abuse, get jobs paying $8-12/hr with no clear career path, such as security guard / nurse’s aid / administrative assistant, or

B. A program that helps already-employed people go from their $8-12/hr jobs to jobs starting around $30k/yr with a clear path to at least ~$40k?

2. Would you rather grant …

A. A program that takes a general-interest population and gets them into relatively white-collar-ish jobs (administrative assistant on the low end; computer support specialist on the high end), at great cost?

B. A program that takes people who are already interested in and capable of a particular, narrower, more “blue-collar” career (nurse’s aid; environmental resource technician; truck driver), spending far less to get them into these (equally well-paying) jobs because it’s primarily about getting them certified?

For (1), we need to know more about the connection between income and living standards in NYC … I personally feel like $8-12/hr does not count as “self-supporting,” especially when supporting a family is an issue, and so I’m leaning toward (B).

For (2), I feel that as long as there are still more people who want help than there are funds to help them, we should help the “low-hanging fruit” first: the people who just need a certification to get the job they want. So, that’s (B) as well.

What do you think?

I’m right sick of all the jabber about all the wonderful things that giving does for the giver.

Don’t get me wrong. I think that giving is beautiful. Yet, as soon as that becomes the reason you give, it becomes about 10x less beautiful to me. I think that giving because of what it does for you - whether you call it happiness or fulfillment or what - is crass and misguided and yuck.

And generally, this kind of giving looks very different from the kind I prefer. When the donor gives in order to achieve happiness or fulfillment or any other feeling, the donor is probably giving to his own town, or his own “people” (ethnicity, nationality), or whatever random disease has affected him personally. And making sure the gift is restricted so he can have the illusion that every dollar pays for heartwarming things like food instead of unsexy things like rent. And doing a lot of pointless/unhelpful volunteering and other interaction with the fundraisers and the beneficiaries and the site so that he can see and feel his dollars at work. And definitely, definitely not doing something boring and nerdy like picking apart the methodology used to examine the actual effectiveness of what he’s funding.

This kind of giving is still better than nothing, I guess, but it’s worse than a good cheeseburger.

Loud and clear: good giving is NOT about the donor. It’s about the people in need. Remember them?

One of my pet peeves is when donors are determined to fight the diseases and conditions that have affected them personally. I know, this is a personal/philosophical decision, and a lot of people I respect will disagree with my take. This blog is personal and unfiltered, as you’ve been warned before.

The Straw Man says, “I’ve had a loved one affected by cancer. I know how it feels, and that’s why I’m fighting cancer.” You know how what feels, Straw Man? Losing a loved one? Then why don’t you focus on that problem: try to get fewer people to lose their loved ones? How is losing a loved one to cancer different from losing a loved one to diarrhea?

You know why you’ve never lost a loved one to diarrhea, and why I haven’t either? Because it’s easy to prevent death from diarrhea. Because it’s totally insane that anyone ever dies from diarrhea. The people you know are privileged, and would never in a million years be at risk from diarrhea. That’s why diarrhea has never “touched” you or me directly. It’s also the exact reason that fighting diarrhea is such a good use of money.

Fighting diarrhea is simple and saves lives immediately. Nobody should be dying from it, anywhere, but they are - in large part because people who can help, people like you, aren’t around and haven’t seen them “up close,” and are focused on the specific, narrow kind of suffering you’ve seen, instead of on fighting human suffering as well as we can.

I used to think of myself as a “political junkie,” because I had strong opinions about what should and shouldn’t be left to the free market, what we should and shouldn’t be doing with our military, etc.

At a certain point I realized that I wasn’t a political junkie, and had never been one. Political junkies don’t actually talk about whether we should have universal health care; they talk about whether we’re going to get it. They almost never talk about who should be President; they talk about who from their party has the best chance of winning. “Political stories” in the media are the same. Every time I watch a debate, there is zero analysis afterward of whether one person’s positions seem more reasonable than the other’s; there are ungodly amounts of analysis of who “appealed to the voters” and “won the debate.”

I’m interested in the question, “How should the world be?” Politics isn’t for people who are interested in that question; it’s for people who are convinced they already have the answer, and are interested in how they can manipulate others to make it come about.

I want to form a theory (about, say, how to close the achievement gap); read what others have already done to test the theory; revise my theory; then try my theory out on a small scale; test it; revise it again. Political junkies are sure they already know how to close the achievement gap (they generally, though not always, inherit their solution from their party), and spend all their time fighting to get their idea implemented across the whole nation, which is of course a battle that takes decades and leaves no room for testing or learning about their solution. Charity seems like a better place for someone who wants to actually try something, see if it works, and try something else.

The problem is, in charity I’m running into the same phenomenon. I’m dealing with fundraisers who are sure that what they do is the best way to help people, and they’re spending their lives raising money for it.

Yes, there are some issues that I think are clear-cut, and I respect the people that fight for the right side. But I wish more people would step back and say, “Helping people is hard, not easy. My guess at how to do it is a guess, not a divine truth opposed only by evil people and the dupes who listen to them.

“There are evil people and interest groups on every issue, but even if we struck them down, we’d be left with the question of what to do. That’s the question I’m interested in.”

1) Going to a top college
2) Being smart enough to get into a top college

Here’s a strong case for #2. (Hat tip: Eduwonkette)

Needless to say, we’re not surprised by the answer. Selection bias is a cruel mistress.

So much of what makes me passionate about this project is the feeling that the world’s best minds are just thinking about the wrong things right now. I want to show them that charity isn’t just about being “nice,” that the world’s most important problems are interesting and challenging and worthy of their attention.

Mark Cuban is such a smart guy. If only, if only we could get him thinking about this 1/2 as much as he thinks about this!

We’re still right in the thick of things in Cause 1: we’ve received applications from about half our finalists and are still waiting on the other half. Two of those we received are causing me the most difficulty: Population Services International and UNICEF.

PSI didn’t fill out our matrix, but it sent us their version. PSI measures the units of pills, condoms, bed-nets, etc. that they sell every year (giving us data back to 1999), and they match up volumes sold with research on disease prevalence, treatment efficacy, and expected utilization rates. That, combined with clear measures of how much they spend each year allows us to create a picture that’s clear about exactly what PSI is accomplishing.

UNICEF, because it’s so big – roughly 5 times the size of PSI, itself one of the largest non-profits out there – sent us a variety of reports, outcomes, and research papers, all of which just scratch the surface on UNICEF’s activities. But, what UNICEF gave us raises the question that their approach could be better.

UNICEF provided a top down picture of the biggest killers of children around the world, the priority places to implement their programs, and a fully-integrated approach to attacking all the barriers to healthy life facing African children. In one of the programs detailed, UNICEF provides full medical care to children in specific areas and then measures the impact on the villages it works in relative to other surrounding villages that don’t have the necessary services. UNICEF doesn’t just care for a subset of disease or conditions – they focus on a group of people and work to keep them healthy. While there’s a lot to be said for distributing cheap, effective medicine to needy people, given the complexity of problems and the multitude of diseases an approach based on people might be smarter.

But, then again, it’s hard to know because UNICEF is huge and only provided data for a couple programs that barely scratch the surface of the array of programs they run. Though we think UNICEF’s approach is smart, we have no idea what good we’re buying with our grant.

So what do we do? Even if Holden and I had the capacity to read aggregate and understand every report UNICEF has, they don’t have time to collect them all and send them to us. They just have too much information. So, you could argue that PSI can prove what they accomplish, and UNICEF can’t, so PSI is the better choice. But, neither Holden nor I likes the idea of awarding a grant because someone measures what they do. We want to pick the organizations that do the most good, and measurement is the best way to figure that out. We’ve got one organization giving us the whole story, and another giving us part of what might be a better one. Thoughts?

I’ve been thinking a lot today about the idea of “investing in people.” We are pretty obsessed with finding charities that have good results, but many people (including our applicants, and including some of the friends I’m making overseas ), are urging me to put more focus on finding great people, and trusting them.

Here’s the thing: I love this idea. I am all for investing in great people. GiveWell wouldn’t exist if it weren’t for this phenomenon; in fact, nothing would exist, because you need to get past your startup phase before you can start demonstrating effectiveness. And there is a set of people I know that I would call confidently call “great people” (in the specific sense of competence - there are other people I like personally but wouldn’t trust to invest a coupon for 50% off peanut butter). Any of these great people could come up to me and say “I’m starting a charity” and already be 90% of the way to getting half or more of my charitable budget.

But here’s the other thing. None of these great people run charities; they all do something else. That isn’t a comment on the nonprofit sector, it just reflects that I’ve spent most of my life outside it. And, all of these people have impressed me through substantial interaction, and by showing themselves to be intelligent and capable in the things they do focus on. I can’t just dial up a “great person” in whatever area I want.

I can’t determine whether someone is a “great person” through a one-hour abstract conversation or a two-hour site visit. I can’t do it by looking up my friend’s friend’s friend’s friend’s friend, even if I use Facebook … there are only so many degrees out that I trust people’s opinions. (2.) By the same token, I definitely can’t conclude that someone’s a great person from their “reputation.” And even if I could do any of these things, why should donors who’ve never met me trust my assessment?

That leaves me with the other way of finding a great person: looking at what they’ve accomplished. That’s how I’ve come to think that Steve Jobs is great at what he does, and that Frank Thomas is great at what he does; that’s how I’ve identified all the great people I haven’t met. Not a perfect method, but not bad.

A lot of investing and donating comes through personal connections, and rightfully so. But the world is too big for all of it to work this way. Someone who is great at helping people, and is past the startup phase (as all our applicants are), will have evidence that s/he’s helped people. I might double-check my impression with a site visit; a relationship might develop eventually; but it seems fair to judge an executive starting with the organization they spend their life running, rather than the other way around.

What do you think? Any other ideas on how to integrate the “great person” test into our grantmaking?

I hereby decree a new publishing schedule. Major posts will be published on Tuesday and Thursday (including today and this Thursday).

Fundraisers seem to do a phenomenal job.

Somehow, you don’t see fundraisers making a lot of arguments that “Money spent on evaluation means less letters mailed out” or “Evaluation is difficult and you can never really isolate causality perfectly.” Instead, you see them testing. And testing. And testing. And learning things that are far from obvious. And testing again.

Maybe it’s because they’re the ones the nonprofits rely on to stay in business. Program, on the other hand, doesn’t have to be as good as it can be … unless we demand it.

All I was trying to say in the last post could be summed up like this:

This is an awesome literature review about early child care programs. It describes how the author found all the papers she discusses … it is totally straightforward about what the methodological strengths and weaknesses of each paper are … and, hold on to your seats - the tables on pages 218 and 223 can only be described as “rock star.” At a glance, you can see all the studies under discussion, who they looked at, how they looked, and what they found.

The literature review I discussed on Saturday is nothing of the sort. It’s unclear about study design, it makes broad claims whose support is unclear, and of course, there are no awesome tables.

If only people were as determined to take a tough look at microlending as they are at Head Start. But of course, Head Start is politics; it affects us all; it’s important; it’s difficult; it’s controversial; it needs to be argued about. Microlending is charity, so it’s none of those things. That all checks out, right?

Goodness, this post is long and dry. The headline is: I read the paper everyone points to as “hard evidence of microfinance’s effectiveness,” and I came out with tons of questions and a need to visit the library. I’ve learned nothing about how microlending works (is it financing investment? Smoothing consumption? What kinds of activity is it making possible?), and all of the data for how well it works leaves me with about 1000 methodological concerns, possibly just because how vaguely the studies are described.

The paper is published by the Grameen Foundation and available here.

Concerns about bias

As in education, selection bias is a major concern in evaluating microfinance. If Person X is ready to take out a loan, confident that she will pay it back, while person Y isn’t - who would you bet on, even if no loan is given? The Coleman study described on pages 20-21 gives an excellent illustration of this issue. It’s the only study in the paper that uses what I would call the “ideal” design: inviting twice as many participants to a program as it has slots, a year in advance, and then choosing the participants at random. The study found that participants in the credit program were generally wealthier than non-participants, but that once you controlled for this, the program didn’t appear to make them any better off.

The review author points out that the study was done in Thailand, which already has a government-sponsored subsidized credit system. So we agree that the paper doesn’t tell us much about the impact of microlending in general … but it does show the perils of selection bias, and I’ve led with it because this problem affects so many of the other studies.

The worst are the studies - and there are many - that simply compare living standards for clients vs. non-clients, without examining whether the two groups may have been different to begin with. These could easily simply be showing the same effect as the study above: borrowers are wealthier before they ever borrow. Most of the studies discussed early in the paper look likely to have this problem (or at least the description doesn’t make it clear how they deal with it): Hossain 1988 (page 16), SEWA study (24-25), Mibanco study (26 - it isn’t entirely clear who’s being compared to whom, but all the differences discussed are between client and non-clients with no discussion of where they started out), ASHI study (27), and the FINCA studies (28). The last two look at changes in incomes, not just incomes, but if one group started off wealthier, I’d think they’d be more likely to increase their incomes too (regardless of any help from charities).

Incoming-client comparisons: still a long way from easing my mind

The vast majority of the studies discussed try to get around this problem by comparing incoming clients to existing clients. This seems better than simply comparing clients to non-clients in the same region: incoming clients presumably share most qualities of existing clients, aside from the fact that they haven’t yet benefited from microcredit. But, this test is still miles from rigorousness. Page 7 points out a couple potential problems with it - eager borrowers may differ from “wait and see” types, and more importantly (to me), MFIs may loosen their admission standards over time, which would mean that incoming clients are systematically worse off than existing clients for reasons that have nothing to do with the benefits of microloans. And then, of course, there’s just the fact that times change. For example, if microloan programs systematically attract people above a certain income level (as the Coleman study implies), and an economic boom makes everyone wealthier, you’ll see existing clients (originally the only people wealthy enough to enter the program) doing better than new clients (who have just now become wealthy enough).

A relatively simple (imperfect) way to adjust for all this would be to compare incoming clients both to existing clients today, and to those same existing clients at the time they entered the program. This would at least check whether existing clients were systematically better off to start with. Here’s the bad news: if the studies discussed do this, the literature review very rarely mentions it. It occasional points to a divergence between existing clients and incoming clients on some random quality like age (see page 36), rural/urban (36), schooling (33), etc., implying that the two groups are often not very comparable … but the review is generally short on the details, and in almost every case does not address the issue I’ve highlighted here.

Nearly all of these “incoming-clients” studies show significant positive differences between existing and incoming clients, implying that microfinance has improved lives for its clients, if these concerns are addressed. But I’m going to have to check out the papers myself before I feel very convinced.

Here’s what we’re left with:

These are all of the studies discussed that appear to address the concerns above in any way:

1. A 2004 study of a Pakistan program (33-34) compared clients to non-clients starting at similar levels of income, and showed a much larger increase in income for clients (though both experienced huge increases - it was ~30% to ~20%). This doesn’t account for “motivation bias” (the “optimism and drive” that taking out a loan may indicate), but at least it’s looking at people who started in about the same place.
2. A similar study was done on a Bosnia/Herzegovina program in 2005 (35-36), again showing much larger income gains for participants in the programs, and again adjusting for starting income though not for the “optimism and drive” bias.
3. The Second Impact Assessment of the BRAC program (29) compared changes in clients vs. non-clients; between 1993-1996, the % clients with a sanitary latrine went from 9 to 26, while the % non-clients with a sanitary latrine went from 10 to 9. The latrine variable is the only one where the paper makes clear that they started in the same place, and the rest of the discussion of the study seems to imply that they were pretty different to start with, in other ways.
4. Page 21 claims that Gwen Alexander “recreated” the design of the randomized Coleman study I led off with, using the same dataset that a bunch of the other studies were working off. It’s totally unclear to me how you recreate a randomized-design study using data that didn’t involve randomized design, and the paper doesn’t fill me in at all on this.
5. Finally, pages 17-20 discuss a back-and-forth between two scholars, in which the details of what it means to own a “half acre of land” - as well as a debate over a complicated, unexplained methodology that actually appears to be called “weighted exogenous sampling maximum likelihood-limited information maximum likelihood-fixed effects” - appear to make the difference between “microfinance is phenomenal” and “microfinance accomplishes nothing.” The part I’m most interested in is the final paper in this series, Khandker (2005) (discussed on page 19), which draws incredibly optimistic conclusions (crediting microfinance with more than a 15% reduction in poverty over a 6-year period). Unfortunately, the review gives no description of the methodology here, particularly how all the concerns about bias were addressed: all it says is that the methodology was “simpler” than whatever wacky thing was done in the first paper.

Bottom line:

So, bottom line: we have 3 studies (the first three above) showing promising impacts at particular sites, though they were not done by independent evaluators and may suffer from both “publication bias” (charities’ refraining from publishing negative reports) and the “optimism/motivation bias.” We have 2 studies that the review claims found great results with a rigorous methodology, but its description leaves the details of this methodology completely unclear. And, we have a host of studies that could easily simply have been observing the phenomenon that (relatively) wealthier people are more likely to take advantage of microlending programs.

My conclusion? We have to get to the library and read these papers, especially the ones that are claimed to be rigorous.

Conclusion of the review? “The previous section leaves little doubt that microfinance can be an effective tool to reduce poverty” (see page 22 of the 47-page study - before 80% of the papers had even been discussed!) And in the end, that’s why I’m so annoyed right now. This paper does not, to me, live up to the promise it makes on page 6, to “[compile] all the studies … and present them together, in a rigorous and unbiased way, so that we could finally have informed discussions about impact rooted in empirical data rather than ideology and emotion” (pg 6). It covers some truly low-information studies (like the first set I discussed) while presenting them as evidence for effectiveness; it discusses the most important studies without giving any idea of how (or whether) they corrected for the most dangerous forms of bias. It calls the fact that a paper stimulated debate, rather than unanimity, “unfortunate” (18). It’s peppered throughout with excited praise (like the quote above, and the super-annoying parenthetical on page 29). In the end, I don’t feel very inclined to take its claims at face value until I hit the library myself.

Without either the detail I need or a tone I trust, I don’t feel very convinced right now that microlending improves lives, especially reliably. (I’d put it around 63%.) I’m surprised that this is the paper everyone points to; I’m tempted to say that if more people read it, as opposed to just counting the pages and references, something clearer and more neutral would have become available by now.

Averages really annoy me. Average income, average test score, etc. When we’re talking about any kind of analysis of people, I have a hard time thinking of any case where you should be looking at the average of anything.

I much prefer “% of people above some threshold” type measures: % of students who graduated in 4 yrs or less, % of students scoring at proficiency level 3 or higher, % of families earning $20k/yr or less. This kind of metric is about 1.2x as complicated, and 2000x as meaningful, as an average.

Just thought you’d like to know.

So I’ve been watching the ballgame, and it struck me how much sports announcers have impacted my outlook on charity. I can explain.

The most common form of “evidence” we get from charities goes something like this: “We don’t have the data, but we’re here, every day. We work with the children, personally. We’ve been doing this for decades and we’ve accumulated a lot of knowledge that doesn’t necessarily take the form of statistics.”

Put aside, for a minute, the fact that we get that same story from all 150 charities we’re deciding between (all of which presumably think their activities are most deserving of more funding). There’s another problem with the attitude above, one that occurs to me every time I hear Michael Kay announcing a baseball game. In sports, unlike in charity (and really unlike in most things, which is why I find it an interesting case study), the facts are available - and when you look at them, you realize just how little that “on the ground” experience can be worth.

The fact is that baseball announcers and sportswriters spend their entire lives watching, studying, and thinking about sports. Many of them are former athletes who have played the game themselves. They are respected, they are paid to do what they do, and they are more experienced (i.e., they’ve seen more) than I’ll ever be in my life. And yet so many of them truly know absolutely nothing.

“Jeter’s a whole different player in October,” says Mr. Kay (demonstrably false). “You don’t want young pitchers carrying you in the playoffs.” (Comically false - 3 of the last 5 World Series champions had rookie closers.) I’m not giving any more examples - this post would hit 30,000 words in a heartbeat. But I’m happy to refer you to sources that give 2-3 examples per day of seasoned professionals - who’ve spent their whole lives on this stuff - saying things that are obviously, intuitively, factually, empirically, demonstrably, completely wrong.

It hits me over and over again, and I still haven’t quite gotten used to it. My only explanation is that humans have an incredible ability to ignore what they actually see, in favor of (a) what they expect to see (b) what they want to see. Now when I talk to an Executive Director or Development Officer whose life consists of running a charity and whose livelihood depends on convincing people that it’s the world’s best way to help people … I don’t know how much these factors cloud their judgment. Maybe not at all, in some (truly amazing, borderline inhuman) cases. But when they assure me that outcomes data isn’t necessary because they’ve been doing this for years, forgive me for having trouble swallowing this: I can’t help but think of Michael Kay, a man who’s done very little with his life but watch the Yankees, and still manages to know nothing about them.