The GiveWell Blog

Guest post from Eric Friedman

(updated on: ) | by Guest

This is a guest post from Eric Friedman about how he decided what charity to support for his most recent donation. We requested this post along the lines of earlier posts by Jason Fehr, Ian Turner and Dario Amodei.

In 2003, I decided that I wanted to increase the amount of money I gave away, shortly after a two-week trip to India. It was there that I saw a level of poverty beyond the scope of anything I had ever seen in America, and I privately vowed that I could not stand by idly.

When I returned home to Chicago, I tried to figure out the most effective way to give and the best organizations to support. Before my trip to India, I had made a few $100-$200 gifts that I later regretted, and this time around I was not going to give away a dime until I was convinced that my donation would be used well. Unfortunately, I was not able to find the information I needed to be comfortable making a donation. Despite the promise I had made to myself, I gave nothing.

Fast forward a year, and the big tsunami hit east Asia in December 2004. The images on tv motivated me to give $1,000 to an organization well-known for disaster relief, and that reminded me that I had not done what I planned on doing after returning from India. Although I restarted my research to find an organization I wanted to give more to, I couldn’t figure out which organizations were strong. Everything seemed like fluff, and there was no “Consumer Reports” for nonprofits. I barely gave anything in 2005.

By 2006, I came to realize that there wasn’t much high-quality information on which nonprofits performed best, so if I wanted to give, I’d have to make do without it. Inaction was unacceptable, so I developed a plan. I started with the American Institute of Philanthropy, which rates organizations on financial efficiency metrics such as percentage of costs that go towards fundraising and overhead. This provided a starting point to identify organizations that might be good, then I reviewed their websites to pick three that I liked. I knew that this was not a particularly rigorous screen, but it was the best I knew how to do at the time. In 2006, I gave $2,500 to CARE, $2,500 to Africare, and $1,000 to Freedom from Hunger.

I figured that donations of this size might draw enough attention to have a serious discussion with their staff. I spoke with each organization, but was unsatisfied with all of the conversations. Whenever I asked them about how to provide the most help for people or evaluate them against other organizations, their responses were inadequate. They were filled with anecdotes about individuals they’ve helped and inspirational stories, but not much information that would genuinely help answer my questions. I was quite surprised when some of them actually asked what I liked most among their set of programs (e.g. education, clean water, healthcare, emergency relief, etc). Weren’t they supposed to be more knowledgeable than me about which of these is most effective?

I asked if other donors asked these types of questions, and they said that it was rare. I asked if big, sophisticated foundations ask these types of questions. They didn’t either. I found that to be exceptionally odd. (Ed note: this was similar to GiveWell staff’s experience before starting GiveWell. See Elie’s blog post from February 2007) (Since then, I’ve asked several different nonprofits about the grant-making process and post-grant relationships they have with big foundations, and there appears to be a surprisingly small amount of value-added in the process. In some cases, the foundations are actually requiring the nonprofits to spend the grants on things the nonprofits don’t think will best help the intended recipients.)

I also spoke with a couple of philanthropy consultants, who I expected to be better at evaluating organizations and selecting priorities. I was disappointed. Usually they would turn the question on me and ask what types of programs and organizations interested me. I explained that I wanted to support programs that were most effective at helping people and organizations that were best at executing those programs, and I was looking for information on how to do that. Other than that, I didn’t really care what type of program or organization it was. While that seemed relatively basic to me, it appeared as if I was speaking a different language. One told me that I needed to figure out what my objectives were, though I thought I had stated them clearly. A couple were somewhat condescending—implying that they were the experts and I was the one who needed help. Their responses to my questions didn’t give me much confidence in their expertise. I wondered if I was the only donor trying to structure my giving around what the world needed rather than my personal interests. (Ed note: for more, see this Tactical Philanthropy post on the rarity of issue agnostic giving.)

Maybe I was being too critical. I knew that I wasn’t asking easy questions, but they also didn’t seem unfair. I wasn’t expecting an objectively perfect answer, but just something better than I got.

A turning point happened a few weeks later. My contact at Freedom from Hunger called me to see if I wanted to meet their CEO, who was going to be in my hometown for a conference. We had a great discussion for about an hour and a half in a hotel lobby, and it became clear to me that he understood what I was trying to do and thought about many of the same issues when running Freedom from Hunger. While I still didn’t know how to evaluate the quality of programs at different organizations, I found one that had shared many aspects of my philosophy. In 2007, I donated $25,000.

Based on the information I had at the time, there was no organization I believed in more than Freedom from Hunger.

In early 2007, my life took a turn for the better: I met the woman who is now my wife. In 2008, Freedom from Hunger offered us the opportunity to join them (at our expense) on a site visit to some of their programs in Ghana. During our time there, we spent a significant amount of time with some of their senior staff (including the CEO), three board members, program staff, and their clients. We saw the programs in action, which increased our conviction in what they do.

We gave another $28,000 to Freedom from Hunger in 2008.

I started following GiveWell in 2009. It was clear from their blog that we shared similar values, and I loved what they were trying to do. But I disagreed with some aspects of their approach. Their emphasis on measurement seemed excessive. This approach had a built-in bias towards smaller, single-program organizations that could measure their impact more precisely. I wasn’t convinced that there weren’t economies of scale in international development. And their focus on scaling up existing solutions and excluding funding unproven innovations seemed incomplete. While I liked what they were doing, I still had more conviction in my own ability to pick organizations.

We gave Freedom from Hunger another $27,000 in 2009.

As I continued to follow GiveWell’s blog in 2010, I became more persuaded toward their views on areas where I previously disagreed. There were still differences of opinion, but I was also coming to realize that their skill in selecting organizations far exceeded mine. It was humbling to realize that someone else is better at something I had put so much thought into.

At that time, GiveWell had not evaluated Freedom from Hunger. For the first time in several years, I wasn’t sure where to donate to. I had a very candid conversation with Freedom from Hunger about this predicament, and they offered to have GiveWell evaluate them. GiveWell was also willing to do the evaluation—they already wanted to learn more about Freedom from Hunger independent of me.

The evaluation resulted in a “notable” rating, which is better than the vast majority of the organizations GiveWell considers, but also not nearly as strong as Gold.

My wife and I liked the people at Freedom from Hunger and had become personally connected with them, especially with the site visit to Ghana. They are extraordinary people who have devoted their lives to helping others, and they are really good at what they do. They might be the best at their specific niche in the nonprofit world. Despite this, GiveWell’s review suggested that there might be organizations in different niches that have a greater possibility of generating results. If helping others was a sport, Freedom from Hunger is good enough to qualify for the Olympics, but it didn’t win the Gold.

My wife and I had several conversations about what to do. Freedom from Hunger did nothing wrong and we had no regrets about our prior donations. They were our friends, and we had enough of a relationship with them that if we shifted our donations elsewhere, we’d have to explain why.

Eventually, we decided that there was one fundamental principle we should apply: giving was primarily about helping the less fortunate, not our friendships or personal interests. Breaking up with Freedom from Hunger would be hard. I explained our reasoning and they took it in stride, demonstrating that they care more about the less fortunate than their own institutional growth. They are a good group. But in 2010, we gave about $31,000 to GiveWell’s donor advised fund to ultimately be distributed as they recommended.

I imagine that there are other donors who read this blog, but donate to many organizations that are not recommended by GiveWell. While I don’t want to oversimplify the decision-making involved with large charitable gifts or pretend that I have all the answers, I will offer two pieces of advice.

First, know what you’re trying to do. I’ve heard many people say that philanthropy is very personal. I understand that view, and my own giving is close to my heart. But if giving is primarily about helping others, then that the most important component of giving should be about other people. That is, the donor’s personal friendships, interests, and passions should take a back seat. Although you may feel a close connection to a school you attended, an illness that affected a family member, or a community you live in, those may not be the areas positioned to provide the most help for others. Instead, donors primarily focused on helping others should identify the greatest areas of need and the most effective solutions. It can be tough to put other people’s needs over ours, but, ironically, it makes most donors feel better about giving in the end. I certainly do.

Second, know when someone else has more expertise than you. I originally viewed Freedom from Hunger as the best organization I could identify based on the information available to me at the time. And I had thought about it a lot. So it was personally challenging for me to acknowledge that GiveWell is better at evaluating charitable organizations. Neither my wife nor I agree with every aspect of GiveWell’s philosophy and approach—I doubt there is anyone who does—but the strengths they have seemed more than enough to outweigh any weaknesses we perceived. There is a certain pride of ownership many donors (including me) have as they develop their own philanthropic paths, and I’d encourage them to critically self-evaluate to make sure pride of ownership doesn’t get in the way of incorporating the expertise of others.

I am extremely appreciative for the work GiveWell has done to provide resources that were not available at the time I started giving. I get more personal satisfaction from knowing that my giving is doing more to help others, and I will have fewer reservations about opening my wallet wider in the future. To be completely frank, one thing that confuses me is why foundations and mega-donors making million-plus dollar gifts apparently make little use of GiveWell. I hope and expect this to change over time.

GAVI appears to be out of room for more funding (good news)

(updated on: ) | by Holden

We’ve always been interested in GAVI, a large funding vehicle for immunizations (which we consider to be one of the best interventions out there for accomplishing good).

Until recently, GAVI projected a need for $3.7 billion between 2011-2015 (archived). However, yesterday there was an announcement that GAVI had raised $4.3 billion, more than enough to cover this need (archived).

In the past, we’ve refrained from recommending GAVI because we have trouble fully understanding its activities, but we’ve been continuing to revisit it and think about how we might gain better understanding. Now we are fairly confident in not recommending GAVI because it appears to have all the funding it needs (which, given its area of focus, we consider very good news).

This situation illustrates one of the trickier room for more funding-related challenges of individual giving. Even if a nonprofit has significant funding needs today, are there big donors right around the corner about to swoop in and render your donation irrelevant?

Before we recommend a charity, we seek as good an understanding as possible of its room for more funding, and this includes asking it about what revenue it expects in the future. This is the best way we know to avoid recommending a charity just before its funding gap is closed by megadonors, but we don’t think this approach is foolproof. We continue to find the issue of room for more funding – and in particular, the possibility of a GAVI-like situation – to be very difficult to deal with.

Why we should expect good giving to be hard

(updated on: ) | by Holden

We’ve written before about a couple of consistent worldview differences we encounter:

When discussing any specific charity, I can usually think of specific reasons that the charity’s mission is difficult, and specific ways that it might be failing. Here I’m going to try to give a more general argument for why it’s hard to accomplish a lot of good with your donation. I’m not presenting this as a rigorous, evidence-backed argument; I’m just further clarifying my worldview and where I’m coming from.

When you want to help people as a donor, you have to get in line behind all of the groups below:

  • For-profit companies. I believe that most of the things you can do that make strangers’ lives better are things you can get paid for. Every day people help each other send packages, prepare food, recover from illness, etc. via market transactions. This may seem like a trivial and obvious point, but it’s the reason we are so focused on helping the very poor. When you’re trying to help people who aren’t poor, you’re competing with for-profit enterprises.

    And even the very poor get a lot of help from for-profit services. For example, when people started realizing that cellphones could be useful to the very poor, the result was expansion of for-profit cellphone service into the developing world. There were some nonprofit attempts to contribute to this dynamic, but we’re skeptical that they added much value on top of the profit-driven ones.

    I am certainly not saying that all profit-making enterprises are helpful, nor that all forms of help are profitable. But a lot of the easiest help to provide – even for the poorest – is already being provided by people who are doing it to make money.

  • Governments. When a market failure is clear and severe, the government often steps in. Many feel it does not step in enough or that it does more harm than good, but the fact remains that much of the “lowest-hanging fruit” for helping people where markets won’t is covered by governments. Low-income people in the U.S. get free education with high teacher attendance rates, free emergency medical care, and cash among other things. People in the developing-world get far less from their governments, but most governments still provide a good deal of free medical care.
  • Local philanthropy and community. When it comes to market failures that the government has failed to address, there are still often local nonprofits – and just local people – who are well placed to step in quickly and effectively. This is not an endorsement of small community-based organizations as giving opportunities for individual donors outside the community. If you’re outside the community you’re trying to help, you’re going to have trouble figuring out what the real problems are and who ought to be funded to address them; the people in the community will often be better placed to help, by donating and otherwise, than you are.
  • Big foundations. There are opportunities to help that are missed by for-profits, governments, and locals. There are many extremely well-funded and well-staffed foundations looking for just these opportunities.
  • Other donors. If you want your donation to have an impact, you need to find opportunities that have been missed not only by all the groups above, but by other individual donors. Our focus on room for more funding is an attempt to deal with this situation.

In my view, the wealthier the community, the more effective the first three items above (for-profits, government and locals) will be in addressing their problems. Therefore, if you want to find opportunities to provide help that isn’t already being given, you probably need to look at the world’s poorest communities – but doing that probably means helping people who are very far away and very culturally different from yourself, and you have to find opportunities that haven’t already been found by the big foundations or other donors.

When a donor says, “I have $1000 that I’d like to use to help someone,” it may not sound like they’re asking for much. But on reflection, I think they’re really saying, “I’m looking for someone who needs help that they can’t get from a company, their government, their community, or any other donor big or small – and I expect to provide this help just by sending a $1000 check, despite having very little experience or knowledge of the situation.”

Put this way, the donor’s request sounds somewhat exorbitant, and it seems that we shouldn’t expect them to be able to accomplish much with their $1000. Yet as it turns out, I believe that (if they take the rare opportunities that we highlight at GiveWell) they can often use that money to save a life. I think this is a somewhat shocking observation and that it reflects serious problems with the nonprofit ecosystem.

I also think we shouldn’t expect this to be the situation indefinitely. I hope that as the world gets better at providing help to those who need it, all the opportunities to save a life for $1000 will be snapped up more quickly. That will leave GiveWell customers – individual donors looking to help people they’ve never met and know little about – with much less exciting options, and that’s how it should be.

Profile of a GiveWell customer

(updated on: ) | by Holden

The Money for Good study examined the size of the potential audience for work like GiveWell’s. What we’d like to see next would be a study on the nature of this audience: what sort of donor is open to giving based on third-party research? How do they think, what sorts of causes are they interested in, and where can they be found?

We don’t have the resources for a large-scale study on this topic, but in the absence of this, we thought it would be worthwhile to share our impressions from interacting with our own “customer donors,” i.e., people who use GiveWell to decide where to give. Because our money moved is somewhat “top heavy” (the bulk of the money comes from a relatively small number of relatively large donors), we have had in-depth conversations and formed informal relationships with people accounting for a large chunk of our influence. Some of the things we’ve found out about GiveWell customers have surprised us and led to changes in strategy; below we discuss the general impressions we’ve gotten.

  • Attitudes toward evidence seem less key than we would have guessed. When we started GiveWell, we and most of our supporters imagined that new customers could be found in certain industries where people are accustomed to using measurement to evaluate and learn from their decisions. We hoped these people would resonate with our desire to bring feedback loops into areas where feedback loops don’t naturally exist. But we’ve found that a lot of them don’t, largely because impact isn’t the main thing they’re aiming for when they give. People give for many reasons – to maintain friendships, to overcome guilt and cognitive dissonance, to achieve recognition – and a given donor is unlikely to be interested in GiveWell unless achieving impact is at the top of his/her list.
  • Moral and spiritual values seem more key than we would have guessed. Several of our major “customer donors” see their giving as fulfilling religious imperatives to help others. Many others have backgrounds in philosophy and deep interests in secular moral systems. What these two groups have in common, in my view, is serious investment in abstract principles of morality – principles that tell them it is as worthwhile to help people they’ll never meet as it is to help those in their community.
  • GiveWell customers tend to be cause-agnostic and/or aligned with our views on the most promising philanthropic causes. We discussed this phenomenon in early 2011; it’s been one of the major surprises for us. We initially anticipated that many donors would agree with our basic approach to research, but would have strong disagreements with us on which causes to support, and that we would thus have to research a lot of different causes to serve them. That hasn’t turned out to be the case: most people tend to be either fully aligned with our approach to giving (including issue-agnosticism and a preference to help the poorest people, even if they’re overseas) or not aligned at all.
  • GiveWell customers tend to be busy with non-philanthropic pursuits. People who spend all their time on philanthropy generally have their own relationships, convictions based on personal observation, etc. The people most likely to use our research are the ones who don’t have time to do their own. While not surprising, this phenomenon has made it somewhat difficult for us to get thoughtful, meaningful feedback on our research despite our efforts to make it transparent. That’s led us to experiment with more ways to solicit feedback, including formal evaluations, our public email group, summaries of our views on key topics posted to this blog, and (in progress) a possible in-person meeting to discuss our research with any “customer donors” who want to attend.
  • GiveWell customers don’t tend to be friends with each other; it’s been very hard to find “clusters” of them. Most of our customers have had little success interesting their friends in us. We’ve found very little in the way of particular industries or social groups filled with GiveWell customers; instead, it seems that for every twenty people we talk to in any setting, one or two will become a customer. It’s thus not surprising that one of our top sources of referrals is Google: people tend to find us more than we find them.The major exception is that we seem to get disproportionate interest from fans of Peter Singer.
  • Some of GiveWell’s donors are very young (considering how much they give) and very few are retired. This has implications for the future of GiveWell’s influence: the money we’re moving now is mostly coming from people whose peak giving years are yet to come.
  • GiveWell customers never seem interested in public recognition. In our first year, we considered posting acknowledgements to major supporters on our website, but there was no interest. Since then, we have had many customers who require anonymity (even when we ask them to take public credit for our sake) and no customers who’ve requested that we publicly thank them or otherwise help them get recognition.

In my view, the last point is most consistent and most interesting. GiveWell customers have many reasons for giving – some religious, some secular – but they all revolve around achieving an impact for others rather than around getting any direct personal benefit (whether social, reputational or emotional). GiveWell customers are the kind of people who give when no one is watching, and for whom giving is not about the giver. One of the highlights of running GiveWell has been coming into contact with the people who share this value with us.

In defense of the streetlight effect

(updated on: ) | by Holden

In a recent guest post for Development Impact, Martin Ravallion writes:

The current fashion [for evaluating aid projects] is for randomized control trials (RCTs) and social experiments more generally … The problem is that the interventions for which currently favored methods are feasible constitute a non-random subset of the things that are done by donors and governments in the name of development. It is unlikely that we will be able to randomize road building to any reasonable scale, or dam construction, or poor-area development programs, or public-sector reforms, or trade and industrial policies—all of which are commonly found in development portfolios. One often hears these days that opportunities for evaluation are being turned down by analysts when they find that randomization is not feasible. Similarly, we appear to invest too little in evaluating projects that are likely to have longer-term impacts; standard methods are not well suited to such projects … (Emphasis mine)

He concludes with a call for “‘central planning’ in terms of what gets evaluated” to ensure that evaluation doesn’t become concentrated among the projects that are easy to evaluate. His post could be seen as a direct retort to the kind of work emphasized in the recent books Poor Economics and More Than Good Intentions (our review). These books present ideas and evidence that is mostly drawn from high-quality studies, and have little to say on questions that high-quality studies cannot help answer.

My instinct is the opposite of Dr. Ravallion’s: I feel that the move toward high-quality evaluations is a good thing, even if it starts to cause bias in what sorts of programs are evaluated – and carried out. What follows is an attempt to explain my feeling on this. My feeling is a function of my worldview and biases, and this post should be taken less as a “rebuttal” than as an opportunity to explicate my worldview and biases.

My disagreement with Dr. Ravallion has to do with my experience as a “customer” of social science. The vast majority of studies I’ve come across have been seemed so methodologically suspect to me that I’ve ended up not feeling they shed much light on anything at all; and many (not all) exceptions are studies that have come out of the “randomista” movement. (Another particularly helpful source of evidence has been Millions Saved, which focused on global health.) Given this situation, I’m not excited about using “central planning” to make sure that researchers continue to try answering questions that they simply don’t have the methods to answer well. I’d rather see them stick to areas where they can be helpful.

What does it look like when we build knowledge only where we’re best at building knowledge, rather than building knowledge on the “most important problems?” A few thoughts jump to mind:

  • Over the last several decades, I am not sure whether we’ve generated any useful and general knowledge about how to promote women’s empowerment and equality – from the outside – in developing-world countries. But we’ve generated a lot of knowledge about how to produce affordable, convenient birth control in a variety of forms. I would guess (though this is just a guess, as empowerment itself is so hard to measure) that the latter kind of knowledge generation has done much more for empowerment and equality than attempts to study empowerment/equality directly.
  • Similarly, what has done more for political engagement in the U.S.: studying how to improve political engagement, or studying the technology that led to the development of the Internet, the World Wide Web, and ultimately to sites like Change.org (as well as new campaign methods)?
  • More broadly, studying areas we’re good at studying and generating knowledge we’re good at generating has led to a lot of wealth generation and poverty reduction. I feel poverty reduction brings a lot of benefits that would be hard to bring about (or even fully understand) directly.

Bottom line – researching topics we’re good at researching can have a lot of benefits, some unexpected, some pertaining to problems we never expected such research to address. Researching topics we’re bad at researching doesn’t seem like a good idea no matter how important the topics are. Of course I’m in favor of thinking about how to develop new research methods to make research good at what it was formerly bad at, but I’m against applying current problematic research methods to current projects just because they’re the best methods available.

If we focus evaluations on what can be evaluated well, is there a risk that we’ll also focus on executing programs that can be evaluated well? Yes and no.

  • Some programs may be so obviously beneficial that they are good investments even without high-quality evaluations available; in these cases we should execute such programs and not evaluate them.
  • But when it comes to programs that where evaluation seems both necessary and infeasible, I think it’s fair to simply de-emphasize these sorts of programs, even if they might be helpful and even if they address important problems. This reflects my basic attitude toward aid as “supplementing people’s efforts to address their own problems” rather than “taking responsibility for every problem clients face, whether or not such problems are tractable to outside donors.” I think there are some problems that outside donors can be very helpful on and others that they’re not well suited to helping on; thus, “helping with the most important problem” and “helping as much as possible” are not at all the same to me.

It’s common in our sphere to warn against the “streetlight effect,” i.e., “looking for your keys where there’s light, rather than where the keys are most likely to be.” In the context of aid, this means executing – and studying – the programs that are easiest to evaluate rather than the programs that are most likely to do good. (Chris Blattman uses this analogy in the context of Dr. Ravallion’s post.)

But for the aid world, the right analogy would acknowledge that there are a lot of keys to be found, and a lot of unexplored territory both in and outside the light. In that context, the “streetlight effect” seems like a good thing to me.

Suggestions for the social sciences

(updated on: ) | by Holden

Chris Blattman cites our advice on using academic research and asks for “Other suggestions for the profession.” We have several and this seemed like a good time to share them.

Our suggestions should be taken in context, of course. On one hand, we do not have staff with backgrounds in academia; there’s a lot we don’t know about how the academic ecosystem works and what’s feasible. On the other hand, through our work researching charities, we do have an unusual amount of experience trying to use academic research to make concrete decisions. In a sense we’re a “customer” of academia; think of this as customer feedback, which has both strengths and weaknesses compared to internal suggestions.

The #1 change we’d like to see: pre-registration of studies
Our single biggest concern when examining research is publication bias, broadly construed. We wonder both (a) how many studies are done, but never published because people don’t find the results interesting or in line with what they had hoped; (b) for a given paper, how many different interpretations of the data were assembled before picking the ones that make it into the final version.

The best antidote we can think of is pre-registration of studies along the lines of ClinicalTrials.gov, a service of the U.S. National Institutes of Health. On that site, medical researchers announce their questions, hypotheses, and plans for collecting and analyzing data, and these are published before the data is collected and analyzed. If the results come out differently from what the researchers hope for, there’s then no way to hide this from a motivated investigator.

In addition, in some ways we think that a debate over a study would more honest before its results are known.

  • When we see a charity evaluation, we generally assume that the researchers did what they could to portray the evaluation as positive for the charity; we therefore tend to demand that the study has a very high level of rigor (maximal attempt to eliminate selection bias, full and transparent presentation of methodology and results) before putting any credence in it. We want to feel that the analysis was done the way we would have done it, and not the way the researcher felt would produce the “right answer.”
  • But if someone showed us an evaluation plan before either of us knew the results, we’d be much more inclined to accept that shortcuts and methodological weaknesses were there for the purpose of saving time, saving money, ethical considerations, etc. instead of the purpose of producing the “right answer.” If the plan sounded reasonable to us, and the study then produced a favorable outcome, we’d find it much less necessary to pick apart the study’s methodology.

Bottom line – we’d potentially lower our bar for study design quality in exchange for a chance to discuss and consider the design before anyone knows the results.

The basic principle of pre-registration is a broad one, and will apply differently in different scenarios, but here are a few additional thoughts on applications:

  • What we’re describing wouldn’t be feasible with studies that retrospectively analyze old data sets – there’s no way to know whether someone has done the analysis before registering the design and hypothesis. What we’re describing would be feasible with field experiments, which seem to be growing in popularity thanks partly to the excellent work of Poverty Action Lab and Innovations for Poverty Action. It could also be feasible with non-experiments: researchers could pre-register their plans for analyzing any data sets that aren’t yet available.
  • Poverty Action Lab (example) and Innovations for Poverty Action (example) do publicly publish brief descriptions of studies in progress. These are helpful in that they are a safeguard against burying/canceling studies that aren’t going as hoped, but they would be more helpful if they included information about data to be collected, planned methods of analysis, and hypotheses (for contrast, see this example from ClinicalTrials.gov).
  • There can be legitimate reasons to change aspects of the study design and analysis methods as the study is ongoing. So failure to conform to the pre-registered version wouldn’t necessarily cause us to dismiss a study. But seeing the original intent, the final study, and a list of changes with reasoning for each change would greatly increase transparency; we would consider whether the reasoning sounded more motivated by the desire to get accurate results or a desire to get specific results.
  • There can be legitimate reasons to carry out a study without a strong hypothesis (for example, trying a health intervention and examining its impact on a variety of quality-of-life measures without a strong prior view on which would be most likely to show results). However, we still advocate explicitly declaring the hypothesis or lack thereof beforehand. Just seeing that there was no initial strong hypothesis would help us understand a study. Studies with no strong hypothesis are arguably best viewed as “exploratory,” generating hypotheses for future “confirmatory” studies.
  • As food for thought, imagine a journal that accepted only studies for which results were not yet known. Arguably this journal would be more credible as a source of “well-designed studies addressing worthwhile questions, regardless of their results” as opposed to “studies whose results make the journal editors happy.” For our part, we’d probably watch this sort of journal eagerly while on the watch for top charities.

We think the “randomista” movement has done a lot of good for the credibility and usefulness of research. We’d love to see a “preregistrista” movement.

Some other suggestions

Chris Blattman’s suggestions. We liked Chris’s suggestions:

1. Journals should require submission of replication data and code files with final paper submissions, for posting on the journal site. (The Journal of Conflict Resolution is one of the few major political science or economics journals I know that does so faithfully.)

2. PhD field and method courses ought to encourage replication projects as term assignments. (Along with encouragements to diplomacy–something new scholars are slow to learn, to their detriment.)

Tougher norms regarding the use of footnotes to support strong claims. It is frequent for a paper to make a strong claim, with a footnote leading to a citation; it’s then up to us to find the paper cited (not always possible) and figure out in what way and to what degree it supports the claim. The paper may not be a literature review or other broad overview; it’s often a single study, which we don’t consider sufficient evidence for a strong claim (see our previous post).

On the GiveWell site we use a different convention, one that grew out of our own wish to facilitate easy checking and updating of footnotes by different staff members and volunteers. A GiveWell footnote will either (a) lead to another GiveWell page (these tend to be concise, up-to-date overviews of a topic); or (b) include enough quotation and/or analysis to make it clear what the heart of the support for the claim is, along with page numbers for any quotation. It’s therefore easy to quickly see the degree to which our footnotes support our claims (though one may still wish to vet the papers that we cite and see whether their analysis supports their own claims).

If space limitations are an issue, footnotes could be published online, as they are for some books.

Online tools for viewing the full debate/discussion around a paper at a glance. We’d like to see websites organizing the relationships between different papers and comments, such that we could look up a paper and see the whole debate/discussion around it at a glance – including

  • Followups
  • Published responses and challenges
  • Comments from other scholars, without these comments’ having to get published separately
  • Literature reviews discussing the paper
  • Related (but chronologically later) papers
  • Perhaps a summary of the overall discussion, strengths, weaknesses, etc. (when someone wanted to submit one and it passed the website’s quality criteria)