• Casey, Glennester, and Miguel pre-registered a study from Sierra Leone that’s been in working paper stage since 2011: the paper is here (PDF); the pre-registration is here (PDF)
• Humphreys, Sanchez de la Sierra, and van der Windt pre-registered a study of community-driven reconstruction programs in the Congo: a paper about their experience with pre-registration is here (PDF); the pre-registration is here.

Of course, I agree with Holden’s point that talk about these proposals has outstripped implementation to date.

This is a very exciting and promising area for GiveWell to explore, and I’m ecstatic that you guys have chosen to pursue it as a cause.

It’s always seemed to me to be a “solvable problem”; compared to, say, averting genocide or eradicating AIDS, changing the culture and practices of the academic community has always seemed to me to be a lower hanging fruit.

I heard a talk from a chemist at Harvard named George Whitesides last year about this topic. The take-home point for me was made in a single 2×2 table he drew: one axis was importance of the research, and the other was the likelihood of success of the research (“success” in this context simply referred to the production of a result that would lead to a publication, which of course is the currency for all academics.)

He said that a graduate student/professor/researcher always has an incentive to gravitate towards work in the top right quadrant of the chart (unimportant work that will probably lead to a result and therefore a publication) rather than the bottom left of the chart (work that is very important, but has a low probability of producing publishable results anytime soon).

I’ll be excited to hear what avenues you find for donors looking to fund meta-research.

• Re: systematic reviews. The Campbell Collaboration focuses on “education, crime and justice, and social welfare.” I’m not aware of any work it’s done within development economics. While there are others (including Copenhagen Consensus) that do literature reviews with development economics, I’m not aware of any systematic reviews that are comparable in quality (in terms of transparency of protocol, detailed discussion of the merits and drawbacks of each included study, etc.) to the Cochrane reviews I’ve examined; relatedly, I’m not aware of any group that is dedicated specifically to systematic reviews in this field.
• I’m aware that there are pre-registration systems in progress within development economics, but I have never seen one in use: the only cases I’m aware of in which a development economics study pre-declared its analysis plan are for (a) studies that include health outcomes and are registered on clinicaltrials.gov; (b) GiveDirectly.
• I agree that IPA emphasizes the importance of replication and is making some progress in making it happen, but we think progress could be faster and that there is room for a funder to help (and I would guess that IPA would agree with this). As of today I’m not aware of any development RCTs (except perhaps on cash transfers) that have what I consider enough replication behind them to be robust.

- Normally the area of ‘meta-research’ would include doing systematic reviews (as Cochrane does), i.e., it’s not limited solely to changing the incentives on academics

- Groups already exist which do systematic reviews in development economics, e.g., Copenhagen Consensus Centre and the Campbell Collaboration. DFID has a good intro paper. They’re rather harder in social sciences than medicine because of the external validity problem (drugs work basically the same in all bodies, but teachers don’t work basically the same in all countries) so it’s harder (often impossible) to aggregate the results of trials as the medics do.

- There are already pre-registration systems in development economics, precisely to counter publication bias. 3ie has one, as do JPAL & IPA (I think). I’ve no clue whether any of them is complete

- One of the reasons behind IPA’s formation is the problem of academics having little incentive to do replication studies. Once somebody’s found that Intervention X works amazingly in North Kenya, it’s not academically rewarding to investigate it in South Kenya or East Kenya let alone India or wherever – so IPA deliberately trains non-academics to do those studies, because the data are useful and non-academics can be incentivised differently. IPA’s current multi-country replication of an intervention found successful in East India is led by Annie Duflo, for instance.

- We have our work cut out to build demand for this kind of data. The UK’s Guardian ran a piece recently about how we need LESS uniformity between measurement systems: systematic reviews and all the learning they produce are ONLY possible if we have a lot more. I suspect that this, like most things, would magically appear if donors demanded it.