The GiveWell Blog

GiveWell’s plan for 2012: Specifics of research

(updated on: ) | by Holden

[Added August 27, 2014: GiveWell Labs is now known as the Open Philanthropy Project.]

We previously laid out our high-level priorities for 2012. The top two priorities are “make significant progress on GiveWell Labs” and “find more outstanding giving opportunities under the same basic framework as our existing recommendations.” This post elaborates on our plans for these two priorities.

A note on relative priorities: our current top charities have significant room for more funding, so it would not be catastrophic (though it would be highly undesirable) to end 2012 without new top charities. Because of this, we view GiveWell Labs as slightly more crucial for 2012. However, we plan substantial work on both and anticipate that the quality of our standard research will continue to improve significantly.

GiveWell Labs

We believe that GiveWell Labs is very important for our long-term impact; it represents a substantial new opportunity to both find great giving opportunities and expand our potential target audience (more).

However, at this time GiveWell Labs is still in the very early stages. (We announced it in September, but a few weeks later put our entire focus on finding top charities in time for 2011’s holiday season.) The stage it’s at is somewhat comparable to the stage GiveWell was at in August of 2007; and like the GiveWell of 2007, we will probably go through a lot of experimentation, go down some significant dead ends, and possibly miss some deadlines and change our vision of what we’re trying to accomplish. So we don’t want to commit to highly concrete or definite goals at this time.

That said, here’s our current working framework.

The building blocks of GiveWell Labs

As we try to find the best giving opportunities, we believe it will be helpful to work separately on the questions of what the most promising sectors (general areas of philanthropy, such as “climate change mitigation” or “tuberculosis control” are) vs. what the most promising projects are within a sector. We’re thinking of GiveWell Labs as being divided into the following categories:

  1. Completely open-ended, sector-agnostic investigation (example: examining data on foundation grants to get a sense for what today’s foundations work on).
  2. Basic research to determine how promising a sector is (example: investigating climate change in a low-depth way, focusing on determining what strategies are open to philanthropists and whether their cost-effectiveness could be competitive with other sectors).
  3. In-depth work getting a deep understanding of a particular sector (example: trying to gather as many relevant ideas/conversations as possible for tuberculosis control).
  4. Researching a particular project, or kind of project, to determine whether to recommend it.

There is some justification for doing the 4 steps sequentially: #1 helps one choose the right sectors to research (#2), which helps one choose the right sectors to focus on (#3), which helps one choose the best projects to recommend (#4). However, there is also some justification for working on multiple tracks in parallel: learning more about specific projects and specific sectors will probably inform the way we go about deciding between sectors, and there are some sectors we already know well enough to consider them high-priority. In addition, we don’t ever expect to have final or rigid choices of the most promising sectors, and will always be open to particularly promising projects from any sector.

GiveWell’s plan for 2012: Top-level priorities

(updated on: ) | by Holden

[Added August 27, 2014: GiveWell Labs is now known as the Open Philanthropy Project.]

In previous posts, we discussed the progress we’ve made, where we stand, and how we can improve in core areas. This post focuses on the latter, and lays out our top-level strategic choices for the next year.

The big picture
Broadly, we see the key aspects of GiveWell – the areas in which we can improve – as

  1. Research expansion: finding additional outstanding giving opportunities.
  2. Research maintenance and systemization: keeping our research up to date, while allocating as much responsibility as possible to junior staff. This includes regular updates on charities that we have directed significant funding to.
  3. Research vetting: checking the quality of our research and providing evidence for this quality.
  4. Outreach: working to increase awareness of GiveWell, traffic to our site, conversion of website traffic into donors and followers, etc.
  5. Fundraising/operating: maintaining the organization.

These are broadly similar to the areas for improvement we’ve listed in the past. And as in the past, we feel that the first two of these – finding more outstanding giving opportunities and staying up to date on those we’ve found – are at the core of our work and remain our top priorities. The basic reasoning:

  • In 2011, as in 2010, we experienced substantial growth despite not making outreach a major priority for the year. We believe this is because producing quality research – while taking “low-hanging fruit” on the outreach side – leads to strong growth in referral links, organic search performance, and word-of-mouth. From surveying our largest donors, we believe that it is common for such donors to read our reports relatively carefully.
  • We believe that our chances for growing over the long term are highly dependent on our providing as much “room for money moved” as we can in the most outstanding giving opportunities possible, while doing as much due diligence as possible to maintain a strong reputation.
  • Letting up on the quality of our research would be a major risk. A highly dedicated effort to outreach would not clearly have greater returns (just in terms of attracting attention) than continuing to improve the quality of our research.

We are planning some work on #3-#5 above, but believe that we can perform strongly in both areas without making them major priorities for the year.

Our priorities and goals
#1: make significant progress on GiveWell Labs, the new arm of our research process that will be open to any giving opportunity, no matter what form and what sector. A future post will discuss the specifics of our plans and hoped-for progress on GiveWell Labs. This initiative represents a substantial new opportunity to both find great giving opportunities and expand our potential target audience (more).

#2: Find more top charities under the same basic framework as our existing recommendations. A future post will elaborate on our plans for this.

Our current top charities have significant room for more funding, so it would not be catastrophic (though it would be highly undesirable) to end 2012 without new top charities. Because of this, we view GiveWell Labs as slightly more crucial for 2012. However, we plan substantial work on both and anticipate that the quality of our standard research will continue to improve significantly.

#3: Expand our team. We are currently recruiting Research Analysts; we are also thinking about whether hiring for more specialized roles may more efficiently increase our capacity. We hope to have at least eight full-time employees by the end of 2012.

Other goals include:

  • Regular updates on the charities we have moved the most funding to.
  • Putting some time into more deeply investigating research questions that are particularly important to us, such as the risks of population growth and the benefits of deworming. These sorts of investigations are along the lines of the 2011 investigation that ended with our finding major errors in cost-effectiveness estimates published by the World Health Organization, and rethinking how we use these sorts of figures.
  • Updating our research on disaster relief.
  • Raising any funding needed to finance the expansion of our team.
  • Revisiting our process for having our research subjected to formal external review.
  • Improving our process for tracking and processing donations (more).
  • Low-hanging fruit on the “outreach” front:
    • Improving our website to reflect some specific feedback we’ve recently gotten.
    • Further conference calls and community events to discuss our research (rather than simply writing about it).
    • Any other opportunities we see to make reasonable potential gains on the “outreach” front without excessive investment on our part.

Conference call to discuss GiveWell’s annual review and plan

(updated on: ) | by Elie

We’re still in the midst of publishing our annual review and plan to the blog (4 posts down, 2 to go).

We’re planning to hold a conference call on Thursday, March 1 at 8pm EST to answer any questions people have about our progress to date and plans for the coming year..

We plan to post a recording and transcript from the call, so if you can’t make it but have questions, please submit them via email or blog comment.

If you’d like to attend this call, please register your interest using this signup form.

Self-evaluation: GiveWell as a project

(updated on: ) | by Elie

This is the fourth post (of six) we’re planning to make focused on our self-evaluation and future plans.

This post answers a set of critical questions for GiveWell stakeholders. The questions are the same as last year’s.

Is GiveWell’s research process “robust,” i.e., can it be continued and maintained without relying on the co-Founders?

Where we stood as of Feb 2011

We had two full-time, non-co-founder employees (analysts), a part-time employee, and plans to hire another in the summer. We wrote that, in total, we expected to have 3.5 analysts by the Fall of 2011.

Progress since Feb 2011

By the summer of 2011, we had 3 full-time analysts and two temporary summer analysts. One of our full-time analysts left in December 2011; we recently hired one of our former summer analysts, bringing us back to 3 full-time analysts.

The analysts have been with us different levels of time, but as they have been at GiveWell longer, each has continued to take on more responsibility.

Where we stand as of February 2012

We currently have 3 full-time analysts, and have made an offer to an analyst who will start in July, which would bring GiveWell to 4 full-time analysts. We continue to focus on recruiting and hope to reach 6 full-time analysts (8 total employees) summer 2012.

Analysts take the lead on most charity investigations; co-founders may provide basic guidance and sign off on work before it is published. GiveWell Labs, because of its experimental nature, will be led for the time being by co-founders.

What we can do to improve

  • We continue to experience turnover. Two analysts who worked with us in 2011 have since left GiveWell. We don’t believe that turnover is indicative of a problem – GiveWell is a unique environment that fits some people and not others and we would not be surprised if turnover remains relatively high in the near future. We have tried to address turnover by improving our hiring process to select for the characteristics most likely to lead to success at GiveWell.
  • In the past, we have exclusively hired “generalists” i.e., young, largely college-graduates who work on a little bit of everything at GiveWell – e.g., charity reviews, speaking with donors, answering emails, in-depth literature reviews. In the coming year, we plan on putting more thought into hiring people for more specialized roles, such as administration (we now have enough administrative work for a full-time hire) and literature reviews.

Does GiveWell present its research in a way that is likely to be persuasive and impactful (i.e., is GiveWell succeeding at “packaging” its research)?

Where we stood as of Feb 2011

We wrote, “We’re currently satisfied with the presentation of our content and don’t plan on emphasizing this goal in the near future.”

Progress since Feb 2011

None. This was not a priority over the past 12 months.

Where we stand as of February 2012

As traffic to our website has increased over the past 12 months, we would guess that the importance of better packaging our research has risen. In particular, we feel our site is poorly suited to donors who want to spend more than a few minutes but less than an hour on our site. (We have designed the site to make quick action easy and to provide significant depth, but we have no “middle level” of depth for gaining some information relatively quickly.)

What we can do to improve

We have a list of ideas for how to better package our research, and we may prioritize this in 2012. In particular, we are considering putting more work into a “middle level of detail” as described above.

Review of Due Diligence, by David Roodman

(updated on: ) | by Holden

Due Diligence is a new book on microfinance by David Roodman. We are fans of Mr. Roodman’s work in general (we’ve previously interviewed him for our blog, discussed his research and quoted him for a testimonial), so we were eager to read this book. We weren’t disappointed: it’s thorough, it examines the case for microfinance from multiple different angles, and (in our view) it is consistently – and refreshingly – driven by an evenhanded search for the full and complex truth of the matter, rather than by a particular agenda.

I found the strongest part of the book to be from Chapter 6 onward, where Mr. Roodman reviews the case for (or against) microfinance based on different conceptions of “development.”

  • In Chapter 6, he reviews the literature on whether microfinance directly reduces poverty. His discussion is similar to the one in More than Good Intentions (which we reviewed previously), though it covers earlier non-randomized studies in more detail; as an aside, we’ve found his work on one of the better-known early microfinance studies to be an extremely interesting case study in the problems that can arise with complex studies. Mr. Roodman concludes that (as we have written previously, largely relying in his work) any direct poverty-reducing effect of microfinance remains undemonstrated, even after substantial attempts to demonstrate it.
  • In Chapter 7, he examines the question of whether (and to what extent) microfinance “empowers” clients, giving them more control over their own lives. (There is some conceptual overlap here with our key questions for microlending institutions.) Mr. Roodman discusses interest rates (particularly their transparency to borrowers), the dynamics of group lending (both positive and negative), and some of the qualitative research that attempts to look informally at how microfinance impacts people’s lives. He concludes that (a) there are valid reasons to worry about microfinance as reducing freedom (by increasing debt), particularly when it comes to the traditional “stripped-down South Asian solidarity group loan” model; but (b) microfinance can also empower people by providing an additional tool for managing their financial lives, and the latter effect should be presumed to be the most prominent and basic one.
  • In Chapter 8, he argues that the most impressive thing about microfinance may simply be the way it has proliferated, and led to the creation of self-sustaining institutions. Even without direct proof of the connection to poverty reduction or empowerment, this “industry building” effect could in itself be considered evidence that microfinance contributes to development.

Self-evaluation: GiveWell as a donor resource

(updated on: ) | by Elie

This is the third post (of six) we’re planning to make focused on our self-evaluation and future plans.

This post answers a set of critical questions about the state of GiveWell as a donor resource. The questions are the same as last year’s.

Does GiveWell provide quality research that highlights truly outstanding charities in the areas it has covered?

Where we stood as of Feb 2011

  • Internally, we were satisfied with the quality of our research as compared to other options for donors.
  • We planned to complete a new round of research focused on international aid to find additional top charities.
  • We planned to complete regular updates for VillageReach, our top-rated charity in 2009 and 2010 and the first charity to which we moved significant (now more than $2 million) funds.
  • We felt a need for more substantial external checks on our research. In the previous year, we had several reviews completed, but we believed additional reviews were necessary.
  • Our research process was constrained because of our inability to offer project-based funding.

Progress since Feb 2011

In Feb 2011, we wrote that we hoped for

More intensive examination of our top-rated charities (the ones that attract the lion’s share of our “money moved”), including in-person site visits, continual updates on room for more funding, and conversations with major funders who have agreed – or declined – to fund them.

In 2011, we did all of these things.

  • The level of vetting to which we subject our top charities has increased significantly.
    • We visited all three leading contenders for our highest ratings in October before giving them our top rating. We intend to continue this and visit charities before we give them our top ratings.
    • We conducted extensive reviews of the cost-effectiveness and evidence of effectiveness for the programs run by our top charities, which went significantly beyond our independent assessments of research in previous years. (See our writeups on ITNs and deworming.)
  • We have completed regular updates on VillageReach’s progress.
  • We have received several additional external reviews of our research, but our attempts to significantly expand this process were not successful. We intend to revisit this in 2012.
  • In addition, we launched GiveWell Labs, a new arm of our research process that will be open to any giving opportunity, no matter what form and what sector, in the hopes of improving our ability to find great giving opportunities.

As our influence has increased, our ability to get access to relevant people such as charity representatives, scholars, and major funders has improved. (For example, see our investigation of targeted vs. universal coverage for insecticide-treated nets.) This, along with growth in our team, have improved our ability to do in-depth research efficiently.

Where we stand

We feel that our current research is high-quality and up-to-date. We are not satisfied with our total “room for money moved.” We estimate that our top charities have roughly $15-20 million in available room for more funding, which is substantially more than we have directed to them, but not necessarily “enough” if our influence were to continue growing rapidly.

We also feel there are multiple areas that could offer outstanding opportunities that we have not yet researched as thoroughly as we could (particularly in the areas of nutrition, vaccinations, neglected tropical disease control, tuberculosis control, and research and development).

We also continue to see room for improvement in our coverage of top charities.

  • We’d like to continue to increase our qualitative checks on top charities, particularly conversations with those who have funded them or chosen not to.
  • We remain unsatisfied with the degree to which our research is “vetted.” It still seems to us that we could make a substantial mistake or error in judgment, with too high a probability that it would remain unnoticed. We feel that this is much less true today than ever before, because (a) our staff is larger and we subject important pages to multiple checks from different people; (b) our research draws more attention, including from donors who are giving large amounts and vetting our work fairly closely. We feel that the degree to which our work is “vetted” will grow as our overall influence and prominence grows, though putting more effort into formal external reviews help as well.

What we can do to improve

  • Revisit the goal of having our work subjected to formal, consistent, credible external review.
  • Continue to look for more outstanding giving opportunities for individual donors, particularly (a) in the areas we have identified as most promising (b) through GiveWell Labs.