The GiveWell Blog

Cost-effectiveness is in the eye of the beholder

Broadly-speaking, we think of “cost-effectiveness” as referring to how much of value is accomplished for a given amount of money. If two interventions are both proven and scalable to similar degrees, the more cost-effective one is a better investment because it allows the same donation to accomplish more of value. However, “value” means different things to different donors: to some, $200 to save a child’s life might be a great deal, while others may prefer $450 to repair a fistula.

We prefer to leave major judgment calls to our donors when practical. This means that rather adopt a single definition of value (such as Disability-Adjusted Life-Years), we hope to find the most cost-effective interventions for several different definitions of value. We can’t cover every possible notion of what’s worth funding, but we hope to be of use to as many donors as possible within the cause of developing-world health (our current focus – we will be moving on to economic empowerment later).

The following are several philosophical goals – i.e., definitions of what results make an intervention “valuable” – that we think will appeal to many donors.

  1. DALYs averted. Although it isn’t our favorite measure of value, the Disability-Adjusted Life-Year is a widely used metric that considers all forms of mortality and morbidity. Some donors may feel most comfortable aiming to avert as many DALYs as possible for their donation.
  2. Economic benefits. Health problems impose an economic burden, not just a moral one. There are sometimes attempts (such as the “benefit:cost ratio” used by the Copenhagen Consensus and some versions of the social return on investment metric) to combine economic and moral benefits into a single figure, measured in dollars.
  3. Life-years saved – for those who put a lot of weight on being alive vs. not alive (and less weight on quality of life). Interventions that focus on infant mortality are likely to be cost-effective in terms of saving life-years.
  4. Lives saved.
  5. Adult lives saved. It is common to value adult lives more than children’s lives. In addition, adults are more likely to have dependents, making their deaths arguably more tragic (in a way that DALYs could capture in theory, but that DALY estimates generally don’t capture in practice).
  6. Cases of extreme suffering prevented/rectified. This goal has several subcategories, for different conceptions of what constitutes extreme suffering. One that jumps to mind is fistula (and other deformities associated with ostracization).
  7. People brought to a normal standard of health and potential productivity. In some ways the opposite of the cause immediately above, in that it focuses on helping those with high potential rather than helping those with high need. (There are different places one could draw the line for “normal health and potential productivity” – is it enough to prevent/cure someone’s blindness, or is it also important that they be adequately nourished and have reasonable job opportunities?)
  8. Unwanted pregnancies averted / population growth slowed. Some donors might see births averted as a negative; others might feel that it is the key to better quality of life and sustainability.

Some of these metrics are highly well-defined and relatively easy to find or form cost-effectiveness estimates on. Others are far less so. We aren’t necessarily going to be conducting a separate search on interventions for each, but we think it’s productive to get as many appealing “goals” on the table as possible before beginning to narrow down the programs we’re focusing on. We hope that we won’t leave out programs whose potential effects are highly appealing to a large/important set of donors, even if they fail a “cost-effectiveness test” based on other metrics.

If you feel that we’re leaving out any important ones, please let us know. (We will also be consulting with our supporters and Advisory Board on this question.)

Donated technological equipment

Business Week writes:

According to the World Health Organization, about half of the imaging equipment sent to developing countries goes unused because local technicians aren’t trained to operate it or lack the necessary spare parts.

(H/t Aman at THDBlog)

Is this possible?

(Note: I tried to find the original WHO source and all I could find was this article stating that “more than half of the medical equipment in developing countries is left unused or broken because it is too complicated or expensive to operate and repair,” which is substantially different.)

Selecting program-based health interventions

We are doing both region-based and program-based research on developing-world health. First I will discuss our program-based research, which will focus on (a) finding particular interventions (and/or clusters of interventions) that appeal to us; (b) finding organizations that implement these interventions “vertically,” i.e., replicating the same basic program across a variety of regions.

For these purposes, we particularly value interventions that are proven, scalable, and cost-effective.

Unlike most foundations, we are seeking explicitly to serve individual donors – donors who don’t have personal connections to particular organizations or expertise in the issues, and who therefore are better suited to expanding what already works (i.e., interventions that are proven and scalable) than to exploring unproven innovations. More on this idea at our FAQ, as well as in this blog post.

We think of an intervention as proven when:

  • It has been previously carried out and carefully, publicly evaluated (often through academic research) in a way that provides strong empirical evidence for its positive impact on people’s lives. (A future post will further discuss our position on the “evaluation hierarchy” and what sorts of evidence we think are necessary under different circumstances.)
  • The conditions under which it has been evaluated match the conditions under which it is likely to be carried out again, in as many relevant ways as possible.

We think of an intervention as scalable when:

  • There is a significant amount of unmet “need” for the intervention, i.e., conditions under which the intervention would be helpful but has not yet been funded.
  • Donations can be used to replicate the intervention in a variety of areas, while recording enough information about its execution to be reasonably confident that it is working as intended.

Defining “cost-effectiveness” is significantly more complex, and will be discussed in the next post.

Region-based vs. program-based approaches to developing-world health

Deciding where to give involves making major judgment calls: decisions that rest on subjective and otherwise highly debatable claims (such as the decision of which sort of life change to aim for). We have no pretense of being able to make such judgment calls “objectively” or “perfectly. Rather, we try to:

  • Be explicit about which values we are pursuing and which judgment calls we’re making. For example, we’ve already declared our preference for funding proven and repeatable interventions, and we’ve declared our decision to focus on developing-world direct aid for the coming year.
  • When practical, leave major judgment calls to our donors rather than making them ourselves, by recommending a variety of charities that are strong according to different criteria. This approach is especially important for judgment calls that our donors haven’t agreed to in advance (unlike the two listed directly above).

We are currently focusing on health interventions for the developing world (we will research other aspects of developing-world aid later in our process). One of the major judgment calls involved in choosing a health intervention is the decision between a region-based approach and a program-based approach to giving.

Taking a region-based approach means focusing on a particular part of the world; learning as much as possible about the people who live there, the opportunities they have, and the problems they face; and then finding a program that is well-suited to addressing the particular needs of this region. For regions with many interrelated health problems, such a program will likely be one that aims to strengthen the general quality of health care in the region, which will make it possible to address many health issues at once. (One such program is Partners in Health, a recommended charity from our last round of research that focuses on bringing full-service health care systems to poor rural areas.)

Taking a program-based approach means focusing on a particular intervention (or cluster of interventions); learning as much as possible about the conditions under which this intervention has been shown to improve lives in the past; and then finding a program that replicates this intervention across many regions. (This is the approach we took in evaluating Population Services International, which markets materials such as condoms and bednets across the world.)

A region-based approach has the advantage that it is more likely to be well-fitted to the particular people it serves and their needs. A program-based approach, however, may in some cases be a simpler and/or more cost-effective way to address a particular health problem, which means that it may be a more efficient and reliable way of changing lives for the better.

We will be taking both approaches, and presenting the options for donors that correspond to each. In future posts, we will discuss the specifics of (a) our region-based approach, i.e., how we will be choosing one or more countries to focus on; (b) our program-based approach, i.e., how we will be choosing one or more interventions to focus on.

DALYs and disagreement

Elie thinks that fistula is worse than death. jsalvati disagrees.

I’d rather bring someone to full health than save an infant’s life. Ryan agrees, but Basti does not and Ron Noble emphatically does not.

It’s possible that we would all agree if we knew more about the lives of people in the developing world, or if we just had a long enough to argue about our values. It’s also possible that we wouldn’t. And as long as we disagree, we’ll have different opinions on what the most “cost-effective” interventions are. For example, if it’s true that fistulas can be repaired for $450 each, is this a better or worse use of donations than preventing children’s deaths for $200 each through vaccinations? My answer would be “It depends on the donor.”

Converting disease burdens and intervention benefits into DALYs doesn’t resolve questions like this. Rather, it obfuscates them, by converting the two interventions into the same terms using a single set of philosophical values. If the numbers above ($200/death averted for vaccinations, $450/surgery for fistula) are accurate, they allow different donors to make their own judgment calls, while being informed about their options. But these aren’t the numbers you’ll find in the Disease Control Priorities Project’s summary tables; instead, you’ll see only that surgical services cost an average of $136 per DALY averted (Jamison et al. 2006, Pg 75) and that the vaccinations interventions costs an average of $7 per DALY averted (Jamison et al. 2006, Pg 77).

Some simplification and information loss is necessary in order to compare different options, but reducing everything to a single unit means being able to serve only a single kind of donor. I’d prefer to estimate the effect of different interventions on a variety of “life outcomes” that different donors might value differently. We will discuss this variety more in a future post, but here’s a quick list:

  • Total life-years saved.
  • Adult lives saved (as it is common to value adult lives more than children’s lives).
  • Cases of extreme misery, such as fistula or perhaps severe elephantiasis, averted.
  • People brought to a “normal” level of health, i.e., without any debilitating nutritional or other conditions.

All of these things need to be separately estimated to produce DALY estimates. The DCP report did so with admirable thoroughness and far more people than we have (Jamison et al. 2006, Pgs xxiii-xxxiv). Yet because they published only their DALY estimates (not, with some exceptions, the estimates of different health problems that went into them), they buried a great deal of this work, and produce cost-effective estimates that are useful only if you’re completely on board with all of their values (from how bad each disability is to how to value different years of life). We’re currently trying to get in touch with the authors so we can get access to more of the details; if we don’t, we’ll have to repeat much of their work (with less capacity to do so).

Sources

Previously in series:

Donors don’t have to pay for their own philanthropic advice

Tactical Philanthropy:

Sooner or later, donors are going to start being willing to pay for advice on how to give. This will transform philanthropy.

I agree that donors should be willing to pay for advice on how to give. I certainly would have done so back when I was in the for-profit sector, if I could have found an advisor I had confidence in for a price I could afford.

But it’s conceivable that donors – especially small donors – will never have to pay for philanthropic advice, because someone else will pay to give them that advice. To give a simplified example: say that you care passionately about the cause of K-12 education, but know little about it. Now say that for $1,000, you can fund a philanthropic researcher to produce a report for other donors whose gifts to K-12 education charities will total $10,000. That means you have the choice of giving $1,000 directly to a K-12 charity (though you don’t have much to go on in picking one), or spending that $1,000 to “redirect” $10,000 of uninformed giving to the charities recommended by researchers.

The latter can be a pretty good deal. Unlike in investing, in philanthropy it makes perfect sense to pay for the privilege of redirecting other people’s money. (In fact, this practice is already widespread – large donors often fund fundraising campaigns, with the aim of raising money from others, and lots of people are happy to fund advocacy charities that are ultimately aiming to redirect government funding.)

Picture a world where some donors use philanthropic research for free, and other donors pay for that research with the knowledge that it’s redirecting the first group’s money. This isn’t the only, or necessarily the most aesthetically appealing, way for philanthropic research to get funded. But it’s a perfectly good deal for all parties involved (the donors that get the free research and the donors that pay to improve others’ giving). It’s a model that couldn’t work in for-profit investing, but when it comes to philanthropy where donors are seeking to create public goods rather than add to their own wealth, I see nothing unsustainable about this setup.

That’s the basic arrangement we’re currently pursuing. We are seeking GiveWell Pledges from donors who might be happy to use our research, but don’t necessarily want to pay for it. Meanwhile, a different set of donors pays our operating expenses, in the hopes that we’ll be able to move money from the first group.