This is the second post (of four) we’re planning to make over the next two weeks focused on our self-evaluation and future plans. The first post is here.

This post answers the “additional questions for stakeholders” that we posed to ourselves in January about the state of GiveWell. For each question, we discuss

• Our progress over the last year (specifically, since our last business plan in 11/2008);
• Where we stand (compared to where we eventually hope to be);
• What we can do to improve from here.

Is GiveWell’s research process “robust,” i.e., can it be continued and maintained without relying on the co-Founders?

Progress since 11/08

As of 11/08, we had only made one extended attempt to hire a third employee, and had ended the relationship because our research process was not developed enough to make it possible for us to provide the necessary oversight (more).

In early 2009, we made several more unsuccessful attempts to integrate a new hire into our research process, before again concluding that our process wasn’t developed enough. We tried again with more success in mid-2009, hiring:

• One person who stayed with us from May through December 2009, focusing on our grant application process for economic empowerment, before leaving for a job in consulting.
• Natalie Stone, who has now been with us since July 2009 and is charged with maintaining and expanding our international aid report, by reviewing charities along the criteria we have defined.

Details at our work plan review.

We feel that a key factor in being able to productively hire people has been the progress on our international aid report. We now have many available examples of charity reviews and a consistent set of questions we ask and criteria we use. We have thus been able to be more concrete with new hires about their responsibilities and the expectations on them.

Where we stand

We feel that for the cause of international aid specifically, we are approaching the point where another person can maintain and expand the report with little oversight from the co-Founders. However,

• This applies only to our international aid report. Covering more causes will require heavy involvement from the co-Founders.
• The role of maintaining the international aid report is still not well-defined enough that we feel it can be passed on from one person with a given set of credentials and qualifications to another, reliably and systematically. At this point we would stick to saying that Natalie Stone can maintain it with relatively little oversight.

In short, our research process is not at all robust.

What we can do to improve

Our experience to date has been that hiring and training people (a) involves substantial time investment by the co-Founders; (b) can easily fail to add value to the organization; (c) is more likely to succeed when we are able to be concrete rather than open-ended about employees’ responsibilities, which in turn requires that we have a well-defined research process (i.e., many available examples of charity reviews; a consistent set of questions we ask and criteria we use). With that in mind:

• We can research more causes. We expect research on new causes to be initially open-ended and improvisational (as past research has been), but we also expect to gradually produce more consistent criteria, rules of thumb, and examples, with which we can train others to do the research.
• We can continue hiring people and training them by slotting them into our research process, where we have well-defined enough responsibilities to make this work.
• We can attempt to make our process more formal, to the point where much of what people need to know is captured in writing and as little as possible relies on specific people. We feel at this point that work in this area would be premature. As we work to train new hires, we learn more about how to communicate and formalize our process.

Does GiveWell present its research in a way that is likely to be persuasive and impactful (i.e., is GiveWell succeeding at “packaging” its research)?

Progress since 11/08

A year ago, it was extremely difficult for donors even to understand our work, much less engage with it. We have put significant effort into improving the clarity, organization and credibility of our research, and our efforts/progress on this front are discussed in our previous post under the heading of “Is it practical for donors to evaluate and use GiveWell’s research in the areas it has covered?”

We have not put effort into making our research more emotionally appealing, psychologically persuasive, etc.

Where we stand

Our current content aims for clarity over persuasiveness. We feel that our clarity is now reasonably strong (as discussed in our previous post under the heading of “Is it practical for donors to evaluate and use GiveWell’s research in the areas it has covered?”) But our presentation remains dry, making next to no attempts at emotional engagement.

What we can do to improve

• We can make a concerted effort to “repackage” our existing content, presenting the same ideas in more emotionally compelling ways. Such an effort would involve thinking and working in very different ways from what we’ve spent our time on to date, and may require substantial involvement from people who are more focused on, experienced with and skilled in communications than anyone on our current staff.
• A lower-cost, “low-hanging fruit” version of the above option is to put more effort into integrating our main website with our blog – we have created a lot of content on the blog, much of which is not duplicated in any way on the main website and is interesting/engaging in ways that the main website is not.
• We can obtain and collect endorsements from people whose support donors are likely to find meaningful. Doing so would overlap heavily with the activities described in our previous post under the heading of “Is it practical for donors to evaluate and use GiveWell’s research in the areas it has covered?”

Does GiveWell reach a lot of potential customers (i.e., is GiveWell succeeding at “marketing” its research)?

Progress since 11/08

Our metrics file shows that traffic to our site has grown significantly over the past year. Several factors have likely contributed to the improvement, and it is difficult (and in our view not necessary) to break down the improvement between them:

• Peter Singer’s featuring GiveWell in The Life You Can Save and continually promoting it
• Substantial revamping of our website
• Intensified posting to our blog during giving season
• The simple value of having been visible for another year

Where we stand

Our website traffic is still a fraction of the traffic commanded by two other donor resources, Charity Navigator and GuideStar, and our research is not used in any significant “partnerships.” Our impact remains small.

What we can do to improve

There are many possible strategies for expanding our audience.

• We can focus on improving the persuasiveness and clarity of our research, as discussed in the previous section.
• We can pursue partnerships with other donor/consumer resources.
• We can pursue partnerships with donor-advised funds and wealth advisors, which might connect us to wealthy individuals seeking help with their giving decisions.
• We can pursue partnerships with people and organizations that focus on fundraising for particular causes, rather than particular organizations. GiveWell’s research could help such organizations fundraise for outstanding organizations within their causes (and outsource responsibility for finding such organizations and justifying their choices).
• We can pursue partnerships with corporate giving programs.
• We can advertise directly.
• We can make concerted efforts at “earned media,” i.e., focus our research on issues that are likely to interest the media (which are frequently the opposite of the issues we find most interesting and important) and engage in concerted public relations efforts around these issues.

Is GiveWell a healthy organization with an active Board, staff in appropriate roles, appropriate policies and procedures, etc.?

Progress since 11/08

As of 11/08, we were operating without an official Executive Director, reflecting the fact that we felt current staff did not have adequate resources in place for professional development and supervision (see our 9/08 meeting).

Our Board of Directors consisted of only five people: Bob Elliott (then the Chair, but unable due to his schedule to remain engaged in the project, which ultimately led him to resign his position in July 2009); Greg Jensen (then the Treasurer), Holden Karnofsky (Secretary), Lindy Miller and Tim Ogden.

Since then,

• We have substantially added to and changed the composition of the Board (see our current Board members and positions). The Board now has seven members, each of whom is highly accessible and engaged with the project.
• We have replaced biweekly email reports with a monthly conference call, substantially increasing engagement from the Board. Turnout at conference calls has been high.
• We have substantially increased the ongoing professional development and supervision of staff, leading the Board to confer the co-Executive Director role on the two co-Founders. (See our 7/09 meeting.)
• We have adopted key policies.

Where we stand

Unlike a year ago, we feel that our Board of Directors and our policies are appropriate for this stage of our project. We voluntarily comply with all BBB Wise Giving Alliance standards.

What we can do to improve

We are always open to acquiring more Board members and mentors with substantial experience and ability to contribute to our project.

What is GiveWell’s overall impact, particularly in terms of donations influenced? Does it justify the expense of running GiveWell?

Progress since 11/08

As discussed above, we have seen substantial improvement in our “donations influenced” figures. A couple of questions remain as to the ultimate value/impact of influencing donations:

• Where would the donors we’re influencing be giving, if not for GiveWell?
• How does the impact-per-dollar of GiveWell’s recommended charities compare to that of the charities our “customers” would otherwise be giving to?

We have only hunches regarding the first question, and at some point should do a formal survey of our donors (including factual questions about their past giving as well as hypotheticals about where they would give without our existence) to get more meaningful information.

On the second question, we have done our best to answer “What do you get for your dollar?” for our top charities – however, answering this question meaningfully for our non-recommended charities is much more difficult (non-recommended charities are generally not recommended because so little information is available about what they’re accomplishing). We have argued that differences in charities can be enormous.

It is also worth addressing our impact on general perceptions regarding, and public discussion of, giving.

• We have become members of the Alliance for Effective Social Investing as well as the GuideStar Exchange Advisory Board; both reflect the fact that other notable organizations in the nonprofit sector take our message and/or research seriously.
• Since our joint press release criticizing the overuse of the “overhead ratio” metric, discussions of giving in the media have laid noticeably less stress on the “overhead ratio” as the primary metric of interest, and more stress on alternative approaches. Many of these discussions have directly included GiveWell.
• On one hand, we were one of the issuers of this press release and we have been stressing its message for a long time. On the other hand, there are many other parties behind this message and this shift, and we feel that relatively little of the credit can fairly be attributed to GiveWell.

Where we stand

GiveWell is not having nearly the impact that we would need to see in order to consider it a success. We still consider our project an experiment at this point.

What we can do to improve

The key paths to increasing GiveWell’s impact (as we see them) are laid out in the earlier sections of this post. Working on any of our many possible areas for improvement could lead to increased impact – the challenge is prioritizing. We’ll discuss broad priorities in a future post.

This is the first post (of four) we’re planning to make over the next two weeks focused on our self-evaluation and future plans.

This post answers the first section of questions we posed to ourselves in January about the state of GiveWell as a donor resource. For each question, we discuss

• Our progress over the last year (specifically, since our last business plan in 11/2008);
• Where we stand (compared to where we eventually hope to be);
• What we can do to improve from here.

Does GiveWell provide quality research that highlights truly outstanding charities in the areas it has covered?

This is in some ways the most difficult question for us to give a meaningful answer on. “Quality” and “outstanding” are concepts that depend a lot on your worldview, so when we review the research that we ourselves have created, you can expect our opinion to be skewed toward the positive. That said, we can discuss how our research compares to what we feel it would ideally be.

At this point we are satisfied with the quality of our international aid report, but see a need to cover more causes and to be more systematic about getting meaningful critical feedback.

Progress since 11/08

As of 11/08,

• We had published research on U.S. equality of opportunity and international aid, but felt our research process had substantial room for improvement (among other things, we felt it relied too much on the arbitrariness of how charities responded to our grant application, and that it didn’t make enough use of publicly available independent analysis). We planned to “re-do” the cause of international aid, with substantial changes to the process (see our June 2008 plan).
• We had essentially nothing in the way of critical feedback on our research, aside from very high-level conversations about our conclusions at Board meetings.

Since then,

• We have published our 2008-2009 international aid report using our revised methodology. We feel that we have found an appropriate way of investigating charities without relying on grant applications and that we have substantially improved on our prior recommendations (see our top charities – the two three-star charities were found this year, while the top charities from last year’s report have two-star ratings). We feel that our revised methodology is the best we can do with the resources and information available, and do not feel a need to “re-do” our coverage of international aid (although we will need to keep it up to date).
• We also completed a grant application process for economic empowerment, whose results we will be publishing shortly.
• A small number of people have scrutinized our research in detail and had in-depth conversations with us, partly in private and partly in public. These include (a) Board member Jon Behar, with whom we publicly discussed our reasoning for our two top-rated charities; (b) Phil Steinmeyer and Ian Turner, both donors whom we had no pre-GiveWell connection to and who put substantial time of their own into this year’s giving decisions (In each case, a substantial amount was ultimately allocated to our top-rated charity, VillageReach); (c) 5-10 other individual donors who have gone over much, though not all, of our research and discussed it with us at length.
• We have also engaged in public conversation at a more general level via our blog. There have been generally favorable reactions – though these are not the same as endorsements – from people with substantial experience and/or publications on foreign aid (including David Roodman, Saundra Schimmelpfennig, Alanna Shaikh and Chris Blattman).

Where we stand

Internally we are satisfied with the quality of our research, but:

• We need to add in direct observation of charities’ observations, to the extent that we can.
  • For international aid, it isn’t practical to get both deep and representative field experience with each, or really any, of the charities we review. Gaining truly representative up-close experience with the operations of even a single small charity (let alone one like PSI) could take years; gaining representative up-close experience with several charities, for the sake of comparative analysis, is in our view entirely impractical (and so it is not surprising that no such comparative analysis appears to exist).

  • That said, at this point we have not visited a single developing-world charity, and could potentially gain a lot from a site visit.
• Our research needs more substantial external checks than it has gotten to date. The only people who have scrutinized it in detail (as opposed to discussing/linking to it at a general level) are, like ourselves, outsiders in the field of international aid. Though people aware of our research (including ourselves) generally seem to consider it high-quality, it is important that we eventually subject it to strong, critical scrutiny from people with substantial relevant experience and credentials.

What we can do to improve

• We can conduct site visits to one or more of our reviewed charities. I am currently planning a trip to Africa from 2/10-2/23 during which I will visit both VillageReach and the Small Enterprise Foundation.
• We can make a concerted effort to subject our research to strong, critical scrutiny from people with substantial relevant experience and credentials. The first basic step – which we haven’t yet taken – is simply to systematically identify the people who are most likely to be both qualified and willing to review our research, then make persistent contact with them (both personally and through contacts of ours) asking them specifically for in-depth reviews of our research. If they decline, we’ll hopefully learn more about why they are declining and what we can do to make reviewing our research more worth their while.

Is it practical for donors to evaluate and use GiveWell’s research in the areas it has covered?

We believe that our work has become substantially more usable and practical to engage with. The limited external checks on our research (discussed above) are a major potential obstacle for donors to evaluate it.

Progress since 11/08

A year ago, the poor design of our website was a major potential obstacle for donors seeking to understand our work. Phil Steinmeyer summarized the problems in an email earlier this year. A donor who had the time and inclination could follow every step of our process, but otherwise had very little to go on besides a list of top charities and stories of our project from us and the media. Since then, we have:

• Substantially revamped our website (not just its design but its organization).
• Made a concerted effort to “repackage” much of our research in an extended series of blog posts. While our website focuses on presenting our recommended charities and the process we followed to identify them, these blog posts present (a subset of) our research such that particular general-interest points can be followed without having to read through a lot of other context. Comments on the blog posts also can give context on how others view our research.

We have also made some progress in terms of giving donors ways to evaluate our credibility:

• We have made increased efforts to keep records of our conversations with people who have substantial experience/credentials in international aid, although this page is still rudimentary and not enough output from the conversations is public.
• We have been featured in The Life You Can Save by Peter Singer, and discussed our research extensively with Prof. Singer. As shown in our metrics report, many of our donors come via referrals from Prof. Singer, and they presumably see him as a trusted source; this is relevant to their assessment of GiveWell’s credibility.

Where we stand

A donor with the time and inclination can follow our full process, reasoning, and sources, and (as discussed above) a few that we know of have actually done so. A more casual donor has some, though limited, ways of assessing our research:

• Coming to us from a source they trust, such as Peter Singer.
• Spot-checking a particular claim we make on the website.
• Reading a sampling of our blog posts and the comments on them, particularly the various responses from charities we have criticized.
• Viewing the limited records of our conversations with people who have substantial experience/credentials in international aid.

We feel these options are insufficient on the whole, given our goal of influencing as many donors as possible. We feel that we should have a single, easy-to-find roundup of available information on the reliability of credibility of our research.

What we can do to improve

• We can make a concerted effort to subject our research to strong, critical scrutiny from people with substantial relevant experience and credentials, and to create public records of such scrutiny. (This is also a key area for improvement in the previous section, “quality of our research.” We feel that strong critical scrutiny would improve both the actual quality of our research and donors’ abilities to gauge its credibility.)
• We can put together the best possible consolidated case for the credibility of our research.

Has GiveWell covered enough areas to be useful?

Our original business plan states:

We believe that pitting charities against each other using concrete, consistent criteria is the best way to evaluate them in a way that is thorough and understandable to others. At the same time, we do not want to compare all charities in the same terms, because this would involve making philosophical decisions that should be made explicit rather than buried in conversion factors. We’ve therefore taken the approach of dividing charities into causes by broad philosophical goal … Charities are pitted directly against each other within causes, but not between them.

In other words, while we sometimes argue for one broad cause over another, we ultimately seek to serve donors with many different values. We are a long way from accomplishing this goal.

Progress since 11/08

The focus of the last year has been a report on international aid. The intent was to improve both our depth (i.e., quality of the research and top-rated charities) and our breadth (i.e., the number of “causes,” or different value sets, we can make recommendations for). Our breadth did not improve as much as we had hoped, because we concluded that health has by far the strongest options for donors (and the charities we had previously recommended were in the area of health). We have been able to make only weak recommendations for education and economic empowerment (although better recommendations for the latter are forthcoming as a result of our recently completed grant application process ).

Where we stand

Currently, the only charities we recommend with two-star ratings or higher are in the areas of U.S. education (K-12), U.S. early childhood care, and global health. This is enough to serve some donors, but overall gives quite a narrow set of options.

What we can do to improve

We are considering research on areas including:

• More sub-causes within international aid – including disaster relief/recovery, charities aiming to help orphans and vulnerable children, and more attempts to find a strong microfinance option.
• U.S. equality of opportunity – our current research on this cause comes from our original (2007-2008) research process. It should be re-done with the revised research process we used to create our current international aid report.
• Disease research funding (e.g., American Cancer Society).
• Environmental issues, particularly global warming mitigation.

There are many more charitable causes, but these are the ones we feel it would be most productive to research in the relatively short term (more in a future post).

Bottom line on GiveWell as a donor resource

• Our international aid report meets our internal quality standards, but should be subjected to more systematic, in-depth, public scrutiny to improve both its quality and the ability of casual donors to judge its credibility.
• We feel that both our international aid report and our report on U.S. equality of opportunity (which does not meet our internal quality standards) are valuable resources for donors interested in those causes, and more directly useful for impact-focused donors than any other available resources.
• We offer a very narrow set of causes for donors and should research many more areas.