We often hear calls to “standardize metrics” so that nonprofits’ outcomes can be compared directly to each other. As an example of why we find this idea unpromising, I’d like to review some of our work on the first cause we ever investigated: employment assistance in NYC.

We received 19 applications from employment assistance programs in NYC. 7 applicants were able to provide relatively long-term data on job placements and retentions. We initially hoped to compare these outcomes to each other and get some sense of which organization was delivering the most “jobs created per dollar.” It didn’t work out (to put it mildly).

Breakdown #1: youth vs. adults

The HOPE Program, Highbridge Community Life Center, St. Nick’s and CCCS serve unemployed/underemployed adults; Covenant House, Year Up and The Way to Work (formerly Vocational Foundation) explicitly focus on “disconnected youth.” It was immediately clear to us that we would have to subdivide the organizations, and could not directly compare something like Year Up to something like The Hope Program, since the challenges and opportunities are so different for a youth seeking a “first job” vs. a struggling adult.

Breakdowns beyond the mission statements

The three “youth” organizations listed above may appear similar, if you’re going only off of their basic mission statements.

But when we got into the details of how the different organizations recruit and select clients, it became clear that these three organizations cannot at all be compared in an apples-to-apples way.

Year Up, for example, not only requires a high-school degree (or GED) of all applicants, but conducts a competitive application process and – from the data we looked at – accepted fewer than 1/3 of applicants. (Details.) By contrast, Covenant House asserts that over half its clients have dropped out of school by tenth grade. Year Up places a substantially higher portion of its clients in substantially better-paying jobs than Covenant House, but given the differences in whom they serve, shouldn’t this be expected regardless of the impact of the actual employment assistance programs?

What about Way to Work (formerly Vocational Foundation)? It’s clear that the organization is far less selective than Year Up, as only 23% of its clients have a high school degree or GED, and there does not appear to be a “competitive” process (i.e., willing applicants’ being turned away). However, there does appear to be a good deal of “self-selection” going on – 2/3 of clients drop out early in the program. (Details.) We have no directly comparable data with which to compare this organization’s clients with Covenant House’s: we have 75% in “public housing” vs. 53% “homeless” (staying in Covenant House shelter) and 77% with no high school degree (or GED) vs. 50%+ having dropped out of high school by 10th grade. Covenant House has lower placement rates, and we would guess that it is serving the more challenged population, but we can neither verify nor quantify the extent to which it is.

The importance of differences in target populations

At one point we had hoped to – and attempted to – use Census data to figure out how important the differences in target populations were. This proved futile as anything but a super-rough contextualization: the Census data can only be narrowed down in certain ways, and we only had certain information about target populations, and there was no way to really make them match up. However, for this post I pulled together some data on 1999 wage earnings (focusing on the percentage of relevant people who earned more than $20k in 1999) to give a sense of how much small differences can matter.

• Among 18-24 year olds no longer in school, 27% of those with a high school degree (only) made $20k+; only 17% of those without a high school degree did. People with higher degrees made far more.
• Among 18-24 year olds no longer in school with a high school degree, earnings varied substantially by neighborhood. 59% of those in the same area as the Covenant House office (Chelsea/midtown) made $20k+, while 27% of those in the same area as the Year Up and Way to Work offices (Financial District) made $20k+.

Details here (XLS)

Apples to oranges

Looking over our overview table for finalists in this cause, it becomes clear how many differences make it impossible to compare their outcomes directly. Their programs target different populations, have different requirements (often varying significantly even within a single charity, which may offer different programs), and train them in different skills. The adult programs have even clearer differences than the youth programs. If St. Nick’s places 2/3 of its clients in Environmental Remediation Technician jobs while Highbridge places just under half in Nurse Aide jobs … what does this tell us about how the two compare?

I wouldn’t be ready to ascribe meaning to a direct comparison of job placements between two charities unless the two charities were working in the same region, with the same requirements, similar actual client populations (in terms of age, educational attainment, etc.) and essentially the same program (since self-selection effects would be different for different programs). Even then, something as simple as a difference in advertising techniques could cause differences in the target populations, differences that could swamp any effects of the programs.

Outcomes vs. impact

What if we could know, for each charity, not just how many clients they placed in jobs, but how many they placed in jobs who wouldn’t have gotten such jobs without the charity’s assistance?

If we had this information, I’d be more ready to compare it across charities. But this information is impact – what Tactical Philanthropy has called the “holy grail” of philanthropy. It’s extraordinarily rare for a charity even to attempt to collect reliable evidence of impact (more).

Our current approach is to seek out the few charities that can give at least somewhat compelling evidence of impact, and recommend them, with the quantification of outcomes and cost-effectiveness as a secondary consideration.

It is simply not feasible to compare charities across large sectors (something like “Employment assistance for disconnected youth in New York City”) in an apples-to-apples way. Even if the charities collected all the information we would like, the fundamentals of their programs and target populations would have to reach an unrealistic degree of similarity.

We’ve just published reviews of all of the organizations that applied for our economic empowerment grant. (Note that we previously announced the grant winners here.)

Whenever we publish new charity reviews, we get some push back from charities who aren’t happy that they’ve received a 0-star review.

Before we published our economic empowerment reviews, we sent non-public drafts to the charities so that they could review them for any inaccuracies. In the case of this grant, the push back was both louder and more consistent than we had heard before, and it led us to evaluate our process: what mistakes we had made, what we could do to rectify the situation, and what we should change to handle a process like this better in the future.

The criticism

The main criticisms we received from reviewed organizations were the following:

1. Your review implies that you’ve conducted an in-depth investigation of our entire organization and concluded that we’re doing a bad job. All you’ve done is determine that our application wasn’t strong enough to win this grant.
2. We didn’t know that information about our organization would be published on your website. We thought we were participating in a grant process, independent of the general evaluation work GiveWell does.

In some cases these criticisms were valid and spurred us to make changes.

What we’ve changed

Regarding criticism #1, we are generally very careful not to say that we have deemed a charity ineffective. Rather, we say that donors who choose to give to a charity should ask the charity to bear the burden of proof that their activities are accomplishing good. Our approach is to list questions we have about a charity’s activities and say that we have been unable to answer them.

Similarly, we’ve never implied in our reviews that we’ve conducted in-depth investigation of every charity we we rate 0-stars. In explaining why we rely on heuristics in our reviews, we write, “Our research is constrained by practical considerations. Our goal is not to be “perfect” in our assessments but rather to provide better information than donors can find anywhere else.”

Nevertheless, we understand that donors coming to one of our 0-star review pages could misunderstand them and think that we’ve deemed a charity’s programs ineffective. To remedy this, we’ve (a) added explanatory notes at the top of each review to further explain what a 0-star ratings means and (b) tried to specify as clearly as possible the steps we’ve taken for each charity’s review.

Regarding criticism #2, in most cases we feel we were extremely clear about our plans to publish materials.

Most charities we emailed were directed to our main grants page and our FAQ for the grant, both of which prominently discuss our process of publishing our full reasoning and analysis. In addition, many of the applicants (including the majority of those who objected after-the-fact to publication) submitted their applications online, clicking a box on our online application which stated, “I agree that with the exceptions specified in the text box above, all submitted materials are considered public information and will be posted on the GiveWell website.”

That said, there were a handful of organizations that we contacted in non-standard ways, and that can make a reasonable case that we didn’t adequately inform them about public disclosure (at least we can’t establish whether we did). We offered these charities the option not only to have their materials omitted, but to have us not disclose the fact that they applied for this grant from our site.

Note that we never publish an organization’s materials without explicit permission nor do we comment on confidential materials without that permission.

What we’ll do in the future

Two key items we recognized in this process are:

• It’s important to be completely clear and explicit with organizations about our plans to publish reviews and materials.
• Our process selects only the very few organizations that can meet our criteria. This means that a 0-star rating is the most likely outcome for an organization we consider. This may provide a disincentive to lesser-known organizations to apply for our grants and invite us to review them (by contrast, more prominent organizations generally appear on our site whether or not they apply).

One idea we’ve had for future grant applications is to create a page, first, for every organization we invite to apply for a grant. This way, the grant process allows organizations to improve their rating; it won’t cause them to receive a rating. It will also make our plans to publish a review crystal clear.

First it was shoes, then shirts. I don’t have much to add on questions like “Are supplies in fact clogging the roads?” and “Are there in fact people in Africa who don’t wear shirts because they can’t find/afford them?” But to me, the argument against in-kind donations is both simple and general (i.e., it doesn’t depend much on the specifics of the goods):

1. It seems nearly always preferable to sell your unwanted belongings and give the proceeds to charity.

I’m not sure how many (if any) people are “desperate for shirts.” But it does seem that an easy test of whether your old shirt is worth anything to anyone is to ask, “Will someone pay for it?”

This makes it likely that your shirt itself will not get to the poorest people – it will get to whoever pays the most for your shirt. But as long as you give to the proceeds to the best charity you can find, this seems like a good outcome. Imagine that a less-poor person is willing to pay $3 for your shirt; a poorer person wants your shirt but can’t (or won’t) pay $3 for it. That implies that the latter person is better off with the actual $3 (or $3 worth of the best charitable services you can find) than with the shirt.

Of course, if you sell, someone has to pay enough not only for the shirt, but for the costs of shipping and logistics. But that’s part of the point. If there’s no one in the world who will pay enough for your shirt to cover shipping costs, that’s a red flag that you’re looking at a wasteful transaction. Throw out the shirt and give cash. (Again: if someone in Africa is not willing or able to pay the shipping costs to get your shirt, that implies that giving them cash or charitable services equivalent to those shipping costs would be more valued than the shirt.)

The 1 Million Shirts campaign has asked for alternative ways to help people. I think “collect the million shirts, sell them off and give the proceeds to an outstanding charity” is a reasonable proposal.

This is also my understanding of how Goodwill works.

It seems that this argument holds for nearly all in-kind donations. There will be exceptions,* especially where a gift is driven by an acute local need for a particular good. But I would guess that such situations are the exception, rather than the rule, because I don’t think most in-kind donations are driven by the needs of the recipients. That brings me to the next point.

2. Just because your gifts are accepted doesn’t mean they’re wanted.

Good Intentions are Not Enough points to a USAID statement on gifts in kind (PDF). I think it’s an eye-opener, particularly this part:

The American public often … attempts to collect and donate commodities, also referred to as “gifts-in-kind” (GIK). GIK are most often inappropriate for relief programs and harmful to the environment and the local culture. They are expensive to transport, relative to the cost of procuring the same commodities locally. GIK use up scarce resources such as transportation routes, warehouse space, and staff time. They can adversely affect the regional economy by competing with similar commodities available locally. And GIK can contribute to negative images of the United States and its disaster response activities. Despite these problems, USAID often faces significant pressure to assist with or fund shipments of GIK to disaster settings. (Emphasis mine)

I think there is a possible analogy to volunteering. Charities have an incentive – and perhaps also pressure – to accommodate people asking them to give in-kind donations. (We’ve even seen a little of it ourselves; despite the abstractness of our own activities, people come to us asking where to donate leftover drugs or equipment, and it’s never fun to reply with “tough luck.”)

Giving away your goods feels better than taking my above suggestion, because it doesn’t force you to confront the difference between what you paid for the goods and what they’re now worth. (Giving away an old shirt feels better than selling it for $1.) And plenty of organizations are interested in delivering these good feelings. There may also be organizations that simply deliver gifts-in-kind without carefully weighing the shipping costs, logistics issues, etc.

We’ve noted before that much of charity seems set up to serve the donor rather than the recipient. Consider this next time you’re about to give old clothes to people who may not have asked for them (even if it’s through a charity that’s claiming they have).

*For example:

• Companies/producers can have reasons, relating to price discrimination, to donate the the things they produce. These reasons can apply to a drug company donating drugs, but not to an individual giving away old belongings, so I won’t elaborate on this point here.
• Sometimes there is an excellent match between what you have to offer and what someone else needs, to the point where (a) the person in need would be close to the highest bidder to buy it; (b) selling rather than donating would create unnecessary middleman/friction costs. (Analogy: if a friend is willing to pay $250 for an old computer that’s worth $300, I’ll take the friend’s offer instead of going through the hassle of selling. On the other hand, if the friend is only willing to pay $50, I’ll sell the computer.)
• There can be tax advantages to in-kind donations.