To date, most of GiveWell’s research capacity has focused on finding the most impactful programs among those whose results can be rigorously measured. This work has led us to recommend, and direct several hundred million dollars to, charities improving health, saving lives, and increasing income in low-income countries.
One of the most important reasons we have focused on programs where robust measurement is possible is because this approach largely does not rely on subject-matter expertise. When Holden and I started GiveWell, neither of us had any experience in philanthropy, so we looked for charities that we could evaluate through data and evidence that we could analyze, to make recommendations that we could fully explain. This led us to focus on organizations that had impacts that were relatively easy to measure.
The output of this process is reflected in our current top charities and the programs they run, which are analyzed in our intervention reports.
GiveWell has now been doing research to find the best giving opportunities in global health and development for 11 years, and we plan to increase the scope of giving opportunities we consider. We plan to expand our research team and scope in order to determine whether there are giving opportunities in global health and development that are more cost-effective than those we have identified to date.
We expect this expansion of our work to take us in a number of new directions, some of which we have begun to explore over the past few years. We have considered, in a few cases, the impact our top and standout charities have through providing technical assistance (for example, Deworm the World and Project Healthy Children), supported work to change government policies through our Incubation Grants program (for example, grants to the Centre for Pesticide Suicide Prevention and Innovation in Government Initiative), and begun to explore areas like tobacco policy and lead paint elimination.
Over the next several years, we plan to consider everything that we believe could be among the most cost-effective (broadly defined) giving opportunities in global health and development. This includes more comprehensively reviewing direct interventions in sectors where impacts are more difficult to measure, investigating opportunities to influence government policy, as well as other areas.
Making progress in areas where it is harder to determine causality will be challenging. In my opinion, we are excellent evaluators of empirical research, but we have yet to demonstrate the ability to make good judgments about giving opportunities when less empirical information is available. Our values, intellectual framework, culture, and the quality of our staff make me optimistic about our chances, but all of us at GiveWell recognize the difficulty of the project we are embarking on.
Our staff does not currently have the capacity or the capabilities to make enough progress in this direction, so we are planning to significantly increase the size of our staff. We have a research team of ten people, and we are planning to more than double in size over the next three years. We are planning to add some junior staff but are primarily aiming to hire people with relevant experience who can contribute as researchers and/or managers on our team.
GiveWell’s top charities list is not going to change dramatically in the near future, and it may always include the charities we recommend today. Our top charities achieve outstanding, cost-effective results, and we believe they are some of the best giving opportunities in global health and development. We expect to conclude that many of the opportunities we consider in areas that are new for us are less cost-effective than those we currently recommend, but we also think it is possible that we will identify some opportunities that are much more cost-effective. We believe it is worth a major effort to find out.
What areas will we look into?
As with any exploration into a new area, we expect the specifics of the work we will undertake to shift as we learn more. Below we discuss two major areas of work we are embarking on and building our team for currently. In the long term, we are open to considering making grants or recommendations in all areas of global health and development. We have not yet comprehensively considered what those areas might be, but they could include (for example) research and development, or social entrepreneurship.
Using reasoned judgment and less robust evidence to come to conclusions about additional direct-delivery interventions
In the past, we have often asked, “does this intervention meet our criteria?” rather than “what is our best guess about how promising this intervention is relative to our top charities?” Our intervention report on education is a good example of asking the question, “does this meet our criteria?” It reviews all randomized controlled trials of education programs that measure long-term outcomes, but it does not attempt to reach a bottom line about how cost-effective education in developing countries is.
We plan to more deeply explore how we can reach conclusions about how areas such as nutrition, agriculture, education, reproductive health, surgical interventions, mental health, and non-communicable diseases compare to our current top charities.
Investigating opportunities to improve government spending and influence government policy
Some of the areas we will consider exploring to leverage government resources and affect government policy are:
| Broad thematic area | Examples | Brief rationale |
|---|---|---|
| Public health regulation | Tobacco control; lead paint regulation; road traffic safety; air pollution regulation; micronutrient fortification and biofortification; sugar control; salt control; trans-fats control; legislation to reduce counterfeit drugs; soil pollution; pesticide regulation; occupational safety laws | Some regulatory interventions to improve public health have had a large impact in high-income countries. Low-income countries can lack the government capacity or political will to implement these regulations. Charities can advocate or provide technical assistance to accelerate regulation and improve implementation. |
| Improving government program selection | Innovation in Government Initiative; Innovations for Poverty Action; IDInsight; Center for Effective Global Action | Low-income country governments may not have the capabilities to select good programs to support with their limited budgets. Charities can directly assist governments to make better decisions in the short term, or help improve their capabilities to do so independently over the longer term. |
| Improving government implementation | Results for Development; Deworm the World in India | Low-income countries may not have the capabilities to implement programs effectively. Charities can directly assist governments to improve the reach or quality of programs in the short term, or help improve their capabilities to do so independently over the longer term. |
| Improving non-programmatic government capabilities | Building State Capability | Improving the administrative capabilities of a government can result in broad improvements in the way countries function. |
| Improved or increased aid spending | Center for Global Development; ONE Campaign; Overseas Development Institute; Brookings Institution | Spending by high-income countries on global health and development accounts for a large portion of total spending in this area. There are groups who advocate for, and provide technical assistance to improve aid spending. |
| Advocating for increased spending on highly cost-effective, direct-delivery programs | Malaria No More; Uniting to Combat Neglected Tropical Diseases | GiveWell’s money moved is a small proportion of total global spending on aid. We believe these dollars would go further if a portion were redirected to the highly cost-effective, direct-delivery programs we recommend. |
| Increasing economic growth and redistribution | Charter cities; infrastructure programs; trade liberalization; macroeconomic policy; International Growth Centre; tax reform | Economic growth is an important driver of economic well-being over the long term. Government policies can be an important determinant of the rate of economic growth and the degree to which growth translates into well-being for the population. There may be opportunities for charities to assist in promoting growth and better distributional outcomes. |
| Negative externalities of high-income country policies | Immigration reform; trade liberalization; reducing carbon emissions | Governments of high-income countries are incentivized to select policies which are popular with their own voters. These policies can impose substantial costs on low-income countries. Charities can advocate for these policies to be changed. |
| Improving governance | Election monitoring; anti-corruption; good governance awards; term limits; peace programs | There are particular characteristics of the governance of a country (e.g. democratic accountability, stability, human rights, lack of corruption) which are strongly associated with the well-being of its people. Charities can advocate for these characteristics to be adopted or strengthened. |
| Reducing the cost of health commodities | Clinton Health Access Initiative | Reductions in the cost of medical commodities can result in improved coverage and improved economic well-being for low-income households. |
| Improving data collection | Institute for Health Metrics and Evaluation | Improved data can be used by a variety of actors to make better decisions. |
| One-off big bets | Mosquito gene drives advocacy and research | We may come across promising projects that do not fit neatly into one of the above categories. |
How will our analysis change? How will it be the same?
Writing up and publishing the details of the reasoning behind the recommendations we make is a core part of GiveWell. We will remain fully transparent about our research.
Judgment calls that are not easily grounded in empirical data have long been a part of GiveWell’s research. For example, we make difficult, decision-relevant judgment calls about moral weights, interpreting conflicting evidence about deworming, and estimating the crowding-out and crowding-in effects of our donations on other actors (what we call leverage and funging).
As we move into areas where measuring outcomes and attributing causal impact is more difficult, we expect subjective judgments to play a larger role in our decision making. For examples of the approach we have taken to date, see our writeup of our recent recommendation for a grant to the Innovation in Government Initiative, a grantmaking entity within the Abdul Latif Jameel Poverty Action Lab (J-PAL) or our page evaluating phase I of our 2016 grant to Results for Development (R4D). While writing about such judgments will be a challenge of this work, we are fully committed to sharing what has led us to our decisions, with only limited exceptions due to confidential or sensitive information.
What does this mean for staffing and organizational growth?
We need to grow our team to achieve our goals. Repeatedly this past year, we had to make the difficult choice to not take on a research project or investigate a grant opportunity that seemed promising because we did not have the capacity.
We are planning to roughly double our research team over the next few years, primarily by adding researchers who have experience and/or an academic background in global health and development. We are looking to add both individual contributors and research managers to the team. We expect that the people we hire in the next few years will play a critical role in shaping GiveWell’s future research agenda and will be some of the leaders of GiveWell in the future.
For more information about the research roles we’re hiring for, see our jobs.
Comments
Could you talk about the consequences of this change on GiveWell’s budget? Do you expect to do more/different fundraising to pay for the increased staffing?
I am so excited to see how this work takes shape. I love the work you have heretofore done, and to hear how your team starts tackling the gray areas of research will be incredibly helpful. How do you plan to identify your new expert team members–those who can accurately predict and estimate impact without experimental data? Have you read “Superforecasting: The Art and Science of Prediction”? In it, the authors embark on a similar challenge. I thought the book shared some interesting ideas. For example, since one of our major challenges in the impact world is the difficulty of identifying impactful programs at a systems-change level (since there are often no counterfactuals for large systems), they instead identified individuals who have fairly accurately and frequently predicted unknowable outcomes before they came to fruition. That might be a nice skill to bring to Givewell.
Let me know if you need charter cities experts.
Hi Ian,
We included our projected increase in budget in the materials prepared for our June 2018 board meeting, available here.
We discuss our budget for 2018-2020 and approach on slides 9-12. We project that our total operating expenses will increase significantly during this period, from $4.9 M in 2018 to $9.0 M in 2020. Some of this is driven by expected growth in the number of research staff. You’ll see that we project increasing the number of research staff at GiveWell from 10 today to around 20 in 2020. You’ll also see that we plan to significantly grow the size of the outreach team alongside the research team (from 8 today to around 18 in 2020).
Outreach staff at GiveWell are responsible for a variety of functions, including increasing the amount of funding we direct to GiveWell’s operations over time. We project that we will increase our revenue sufficiently via these outreach activities to support the growth of the research team. Our current best guess is that the approach to fundraising that we’ve taken historically will lead to our operating budget continuing to grow, but we are open to changing our approach if necessary.
Hi Brittany,
Thanks for your kind words. We discuss what we’re looking for in our new research team members here.
Among other things, we’re aiming to hire people who are passionate about helping the global poor as much as possible, who are skilled at expressing what they believe and why, and who can interpret and critique statistical analysis and understand causal inference. Generally, we would expect that most research analysts will have quantitative training, such as a degree in math, economics, or statistics. However, we note on the page that “we believe a truth-seeking attitude (objectivity, curiosity, open-mindedness, humility) can be as valuable as formal statistical training.”
Many individuals at GiveWell are familiar with “Superforecasting.” In fact, the forecasts we make on our GiveWell Incubation Grants (examples here and here) are influenced by the type of thinking laid out in that book.
We are unsure precisely how we will proceed with the thorniest questions of causality that will arise as we embark on this new chapter for GiveWell.
These are excellent news! Very happy to see these new efforts take shape and hope they’ll bring a ton of good to the world’s poor.
I have a concern about branding. Some of these thematic areas, such as anti-corruption or trade policy, though I think very promising, are somewhat controversial. I think GiveWell has a very strong brand with its top charities, and is known for its rigor and objectivity. I’m concerned that involvement in these more contentious issues might hurt that reputation in some people’s eyes and hurt the stature of GiveWell’s current recommendations. Maybe it’s best to undertake these efforts under a different name, or somehow clearly distinguish them? In the same way that OpenPhil deals with more controversial issues, but that can be isolated from GiveWell’s brand.
Something to think about. Good luck!
Hi Ernest,
Thanks for raising your concern. In general, our impression in speaking with individuals who use our research has been that they are generally excited for us to explore new areas that might be more cost-effective than our current top charities. That said, we agree that broadly, there may be pitfalls for our brand as we evolve in this new direction, either by generating confusion about our top charity criteria or by our involvement in more contentious issues.
We thus, as you suggest, may want to distinguish the newer endeavor from our traditional top charity recommendations, although we are unsure exactly what that distinction will look like at this time.
I am very excited too to see the results of this scope expansion! I am working at WHO on guidelines development, and one of my key concerns is efficiency of interventions. It is usually areas where evidence are lacking, so guidelines are not considering enough criteria like cost-efficiency, feasability or equity. I think it would be great if guideline’s developers could access your research to inform their recommendations! I also would be very interested to see if you are developing a logic framework to guide your analysis, and if so, if we can replicate it at WHO to inform our recommendations.
Anyway, good luck, and I stay tuned!
Hi Olympe,
Thanks for your note! We plan to continue to share publicly the details of our research and the full rationale for our recommendations (even when it involves subjective judgment calls) going forward. We also expect to write more about our process for assessing opportunities as it evolves over time. So please stay tuned—more information is forthcoming.
This is fantastic. We (Atlas Network) make our grants to local research and advocacy organizations because their tacit knowledge is what should be driving the public policy change in their countries. We are very concerned about what we call “the outsider’s dilemma” in international giving where the intervention itself can sometimes be a net harm. And one of the ways we mitigate the “who gets the credit” problem for successful change is the degree to which grantees must articulate the reforms they are working towards ex ante. We have found our portfolio is achieving at least one measurable reform (using reputable global indices) across five continents with the majority of those tied to poverty alleviation. Three cheers to GiveWell for this important evolution. Prosperity is a function of institutions and institutions are idiosyncratic as a function of local culture so outsiders must take great care not to interject themselves too much in the solving of others’ problems but we can all do our part to support their work!
*one measurable reform per month
Thanks for this. Are you familiar with Kevin Drum’s reporting at Mother Jones regarding lead? Very scary stuff – I’m super glad you’re looking into this. (As someone who grew up at the height of leaded gas in the US)
Hi Matt,
Thanks for asking. We are familiar with Kevin Drum’s reporting on this topic.
Comments are closed.