This is the third post (of six) we’re planning to make focused on our self-evaluation and future plans.

This post answers a set of critical questions about the state of GiveWell as a donor resource. The questions are the same as last year’s.

Does GiveWell provide quality research that highlights truly outstanding charities in the areas it has covered?

Where we stood as of Feb 2011

• Internally, we were satisfied with the quality of our research as compared to other options for donors.
• We planned to complete a new round of research focused on international aid to find additional top charities.
• We planned to complete regular updates for VillageReach, our top-rated charity in 2009 and 2010 and the first charity to which we moved significant (now more than $2 million) funds.
• We felt a need for more substantial external checks on our research. In the previous year, we had several reviews completed, but we believed additional reviews were necessary.
• Our research process was constrained because of our inability to offer project-based funding.

Progress since Feb 2011

In Feb 2011, we wrote that we hoped for

More intensive examination of our top-rated charities (the ones that attract the lion’s share of our “money moved”), including in-person site visits, continual updates on room for more funding, and conversations with major funders who have agreed – or declined – to fund them.

In 2011, we did all of these things.

• The level of vetting to which we subject our top charities has increased significantly.
  • We visited all three leading contenders for our highest ratings in October before giving them our top rating. We intend to continue this and visit charities before we give them our top ratings.
  • We conducted extensive reviews of the cost-effectiveness and evidence of effectiveness for the programs run by our top charities, which went significantly beyond our independent assessments of research in previous years. (See our writeups on ITNs and deworming.)
• We have completed regular updates on VillageReach’s progress.
• We have received several additional external reviews of our research, but our attempts to significantly expand this process were not successful. We intend to revisit this in 2012.
• In addition, we launched GiveWell Labs, a new arm of our research process that will be open to any giving opportunity, no matter what form and what sector, in the hopes of improving our ability to find great giving opportunities.

As our influence has increased, our ability to get access to relevant people such as charity representatives, scholars, and major funders has improved. (For example, see our investigation of targeted vs. universal coverage for insecticide-treated nets.) This, along with growth in our team, have improved our ability to do in-depth research efficiently.

Where we stand

We feel that our current research is high-quality and up-to-date. We are not satisfied with our total “room for money moved.” We estimate that our top charities have roughly $15-20 million in available room for more funding, which is substantially more than we have directed to them, but not necessarily “enough” if our influence were to continue growing rapidly.

We also feel there are multiple areas that could offer outstanding opportunities that we have not yet researched as thoroughly as we could (particularly in the areas of nutrition, vaccinations, neglected tropical disease control, tuberculosis control, and research and development).

We also continue to see room for improvement in our coverage of top charities.

• We’d like to continue to increase our qualitative checks on top charities, particularly conversations with those who have funded them or chosen not to.
• We remain unsatisfied with the degree to which our research is “vetted.” It still seems to us that we could make a substantial mistake or error in judgment, with too high a probability that it would remain unnoticed. We feel that this is much less true today than ever before, because (a) our staff is larger and we subject important pages to multiple checks from different people; (b) our research draws more attention, including from donors who are giving large amounts and vetting our work fairly closely. We feel that the degree to which our work is “vetted” will grow as our overall influence and prominence grows, though putting more effort into formal external reviews help as well.

What we can do to improve

• Revisit the goal of having our work subjected to formal, consistent, credible external review.
• Continue to look for more outstanding giving opportunities for individual donors, particularly (a) in the areas we have identified as most promising (b) through GiveWell Labs.

Is it practical for donors to evaluate and use GiveWell’s research in the areas it has covered?

Where we stood as of Feb 2011

Last year, we wrote:

While we have created the basic process and template for both reviews and testimonials, and have some basic evidence of our credibility now easily available, we feel that the content of both pages could be much stronger.

Progress since Feb 2011

• We held an in-person research event in August 2011 and a conference call in December 2011 to publicly discuss our research and field questions. We posted audio and transcripts from both meetings. We held these meetings to enable donors to ask questions and engage with our research without requiring them to read all the details on our website.
• We substantially improved our collection of testimonials and now feel that it is fairly strong.
• We consolidated different forms of evidence for our credibility, which include both the improved collection of testimonials and documentation of our greatly improved influence.
• We completed more external reviews, though we did not progress as far on this front as we hoped.

Where we stand as of February 2012

We feel that evidence of our credibility has substantially improved and is now fairly strong, and that this is part of the reason for our increased money moved.

What we can do to improve

Aside from revisiting our approach to external evaluation, we believe the main path to improvement is to continue improving our knowledge of giving opportunities and the depth of our research. We also plan to increase the frequency of research meetings and conference calls to help donors engage with our research.

How much funding can GiveWell’s top-rated charities effectively absorb?

Where we stood as of Feb 2011

We estimated that VillageReach, our top-ranked charity at that time, had total room for more funding in its Mozambique scale up of approximately $1.5 million.

Progress since Feb 2011

We have identified two additional top charities, each of which has significant room for more funding.

Where we stand as of February 2012

We believe that AMF could absorb up to $15 million, and SCI could absorb a few million dollars as well.

We have made significant progress on this front in the past 12 months and expect to continue to progress in the next 12 months.

What we can do to improve

We intend to continue to look for more outstanding giving opportunities. Currently, we see the promising avenues to finding these opportunities as

• Deeply investigating the areas we have identified as most promising (such as the areas of nutrition, vaccinations, neglected tropical disease control, tuberculosis control, and research and development).
• Moving forward on GiveWell Labs.

This marks a change from the strategy we have generally used to date, including in 2011, of examining a large number of charities and deeply investigating the ones that stand out. All of the strongest opportunities we have found to date (VillageReach, AMF and SCI) are groups focusing on proven health/nutrition interventions that themselves have strong evidence bases; this greatly improves the ability of these charities to make a strong case (more discussion at our 2011 post on the appeal of global health and nutrition).

We feel that by focusing on the areas of aid with the strongest evidence bases and deeply understanding them, we are likely to find more outstanding giving opportunities; examining charities outside these areas appears at this point to have lower potential returns.

This does not mean we are closing ourselves off to the possibility of recommending charities in other areas. We have some particular promising charities in other areas on our list to investigate, and we expect to continue to have an “open submission process” such that any charity has the opportunity to make a case to us. However, we will likely raise the bar for “what it takes for us to investigate a charity outside our priority areas,” and we will likely focus on priority areas for at least the coming year.