One of the challenges of GiveWell is evaluating the quality of our own research. There are no accepted standards of what charity recommendations should be or how they should be assessed.

Over the last year, we’ve been developing a method for self-evaluation based on structured writeups from people external to GiveWell (i.e., people who are not donors, Board or staff). Some of these people are early-stage volunteers with GiveWell; asking them to review us in this way gives us both valuable feedback and a window into how they think. Other reviewers have been recruited by us because their opinions seem particularly relevant due to their backgrounds (for example, Laura Freschi of Aid Watch). The goal here isn’t to produce a quantitative “score” for our research, but rather to provide some in-depth and credible outside perspectives that our audience (and we) can react to as we choose.

In most cases, the reviewer is asked to answer a very specific set of questions about a specific subset of our content: our overview of a particular issue such as developing-world education or HIV/AIDS, our heuristics for identifying promising charities, the fairness of how we’ve applied these heuristics, or our writeups on particular charities. Put these four areas together and you have our entire process for recommending charities.

However, we’ve also been experimenting with a broader approach that asks the reviewer to evaluate the whole of GiveWell’s value at once: finding the best charity to donate to within a set period of time, with no requirement that they use GiveWell to do so. We provide the reviewer with a list of all online resources we know of, including but not limited to GiveWell, and encourage the reviewer to do whatever it takes to get the best answer (including searching independently for information and calling charities directly).

We’ve run this “Finding the Best Charity” project a couple of times with volunteers, but the most recent posting is particularly interesting for a couple of reasons:

• The reviewer was Tobias Pfutze, an Assistant Professor of Economics who co-authored a paper rating official aid agencies with William Easterly.
• We took the project out of the hypothetical and into the real by offering a $2,500 donation to the charity of Prof. Pfutze’s choice, in exchange for the time (about 10 hours) he spent on the review. This donation was funded by Dario Amodei. Thus, this project literally represented Prof. Pfutze’s best attempt at allocating $2,500 to accomplish good.
• Prof. Pfutze chose a charity, Living Goods, that GiveWell does not recommend – though he did find this charity through GiveWell’s website.

Prof. Pfutze’s submission is available here. It includes a paraphrased transcript of a follow up conversation we had with him, exploring our areas of agreement and disagreement.

Overall we found this project very interesting and valuable. On one hand, Prof. Pfutze was clear that he “found Givewell to be by far the most helpful website,” that he agreed with the bulk of our criteria and conclusions, and that he found our recommended charities to have strong cases behind them. On the other, he came to a substantially different conclusion from us.

It appears that he placed a substantially higher weight on the “upside” of the donation – what the project would accomplish if it went as well as possible – and in particular, on the project’s model for achieving sustainability. He found Living Goods promising because it both (a) is aiming for a highly sustainable, lasting impact; (b) gains some credibility from its ongoing evaluation in partnership with Poverty Action Lab.

We remain in disagreement with this conclusion, but think his position is defensible, well-argued and thought-provoking.

We encourage interested readers to check out his full submission including the follow-up dialogue with GiveWell. We’ve found this “Finding the Best Charity” assignment to be valuable and interesting, and we are hoping to use this same format to get more feedback in the future.