Following up on Thursday’s Alliance for Effective Social Investing meeting, Sean at Tactical Philanthropy writes:

A high performance nonprofit is a very well run organization. It has outstanding leadership, clear goals, an ethic of monitoring performance and making adjustments as needed, and it is financially healthy.

A high impact nonprofit is one whose efforts have been proven to cause sustainable, positive change.

Impact can be seen only in retrospect. Often many years later. Performance can be directly observed.

I think high impact nonprofits are the holy grail of philanthropy. But like any holy grail, it is something to journey towards, not something you demand now.

Sean goes on to argue that funders should put more focus on “high-performing,” as opposed to “high-impact,” nonprofits. At GiveWell, we focus on “high-impact” nonprofits, in that we look for evidence of past impact and not just future promise. Our response to Sean:

1. Assessing “high-performance” is much harder than assessing “high-impact.” This isn’t to say that either is easy. But we feel it’s very doable for charities to take the “form” of a “high-performance” nonprofit - collecting large amounts of data, executing activities competently, and describing those activities in a compelling and money-raising way - without actually being on a path toward impact (which requires that the data be the right data and that the activities be the right activities for the goal).

We see many charities with impressive-looking evaluation systems; far fewer with actual past outcomes to show. If anything, this makes us suspect that other funders are looking for the form and appearance of good evaluation, without holding charities accountable for actual results.

2. Because of this, funding “high-performance nonprofits” is not something that casual donors (as opposed to subject matter experts) are well positioned to do. This point parallels our argument that casual donors aren’t well positioned to fund the unproven and innovative. Like funding a small and unproven charity, funding a “high-performance” but not “high-impact” charity means trying to do something that hasn’t been done before, and introduces a greater need for understanding the full context of a program.

3. “High-impact” nonprofits might be rare - but that doesn’t make them overfunded. We believe that our top-rated charities can productively use more funds than they’re currently getting. As long as that’s the case, why should a casual donor give to a charity without past impact rather than a charity with past impact?

You donate some money to a charity in the hopes that it will (a) carry out a project that (b) improves people’s lives. In order to feel confident in your donation, you should feel confident in both of these.

In most areas of charity, we feel that people overfocus on “did it happen?” relative to “did it work?” People often worry about charities’ stealing their money, swallowing it up in overhead, etc., while assuming that if the charity ultimately uses the funds as it says it will, the result will be good. Yet improving lives is more complicated than charities generally make it sound (see this recent post of ours). This partial list of failed programs is made up entirely of programs that appear to have been carried out quite competently, and simply didn’t improve the lives of clients.

In international aid, the relative importance of “did it happen?” grows for a couple of reasons:

• International charities work far away and often in many different countries at once. It often isn’t feasible for their main stakeholders (Board members, major donors, etc.) to check that projects are being carried out.
• International charities are working within foreign political systems, cultures, etc. Materials can be stolen or misappropriated en route. Locals can take advantage of their superior knowledge and “game the system.”
• Many of the activities international charities carry out are proven to work (though many are not as well). Using insecticide-treated nets will reduce risk of malaria (more); an appropriate drug regimen will cure tuberculosis (more); vaccinations will prevent deadly diseases (writeup forthcoming). These claims have been proven and are essentially not subject to debate. This is not the case in the developed world - most of the programs charities work on have not been shown to improve outcome measures of health, standard of living, etc. (See, for example, this guest blog post.)

“Did it happen?” is a question that can largely be answered by informal, qualitative spot-checks. That’s why we would like to see more and better qualitative evidence. By contrast, to know whether a program worked, you need to somehow compare what happened to clients with what would have happened without the program - something that is often hard to have confidence in without formal outcomes tracking and evaluation.

Therefore, we believe that the role of site visits, qualitative evidence, spot-checks, etc. is likely more important in international giving than in domestic giving. In international aid, delivering proven programs (particularly medical ones) is a large part of the battle. In the U.S., most reputable charities are probably doing what they say they’re doing; the question is whether what they’re doing is effective.