[Added August 27, 2014: GiveWell Labs is now known as the Open Philanthropy Project.]

We previously listed our five chief criteria for GiveWell Labs (a new arm of our research process that will be open to any giving opportunity, no matter what form and what sector). This post further discusses the first two of these criteria – “upside” and “high likelihood of success” – and the tradeoff between them.

Upside

We use “upside” to refer to the possibility that a philanthropic project will have a huge/outsized impact. While it’s a good thing to fund projects that have this kind of potential – and while a single hugely successful project can make up for many failures – we also see danger in overthinking the upside of projects.

We don’t have experience with venture capital, but our impression is that in the for-profit world it’s extremely difficult to read the “upside” of different ventures, and thus many investments with “big win” potential are needed for one big win. Not only will many of the most hyped, funded, seemingly limitless-potential companies flop, but many of the biggest success stories won’t necessary look high-potential early on. For example, Twitter began as a frivolous-looking side project of another startup; Facebook’s founder was presenting a relatively unambitious-sounding vision of a college-only network a year after its founding. Another striking example of the danger of overthinking upside is provided by a published debate over AirBNB: Union Square Ventures saw a talented team and successful product, but declined to invest specifically because “I’m just not sure how big it’s going to be.” AirBNB later appears to have become a billion-dollar company.

Our feeling is that this sort of uncertainty extends to philanthropy as well. It’s hard to say in advance how generalizable and widely applicable an innovation is and how big the “latent market” for it is, and it’s also hard to say how an early-stage project will look in a few years. Because of this, we are against putting heavy weight on quantification of potential impact, and in favor of looking for projects with real-world feedback loops, so that they can learn and adapt as they go, hopefully eventually hitting on an approach that works and can be scaled and/or generalized.

Upside vs. likelihood of success

We do feel that there are probably some signs that distinguish high-upside-potential from low-upside-potential projects. The main one that strikes us is the extent to which a project’s goals revolve around development and testing of new knowledge, ideas and approaches, as opposed to repeating what has been done before or relying heavily on particular individual talents.

To the extent that that’s right, there is an inherent tension between upside and likelihood of success. To maximize the former, one looks for things that haven’t been done before; to maximize the latter, one largely looks for things that have worked before. For most of GiveWell’s history, we’ve focused on finding interventions and organizations with reasonably strong track records (though we’ve taken risks and considered upside as well); GiveWell Labs may give us more leeway for even higher-risk, higher-upside projects.

Our basic approach to the tradeoff:

• We consider likelihood of success much easier to evaluate than upside, and we’d prefer a project with only the former going for it to a project with only the latter going for it. However, we could recommend a project with no track record, high upside and strong performance on our other criteria (strong people, room for more funding, and plans for getting real-world feedback and learning from our results rather than a binary “success/failure” outcome).
• There may be “best of both worlds” opportunities in the “valley of death.” That is, it may be possible to find cases where there is a new approach whose basic promise has been demonstrated at a small scale, but which still needs more demonstration before it can reach very large scale. It isn’t clear to us whether funders often overlook these opportunities, but if they do, this could result in outstanding opportunities for us and our donors.