“Dollars per life changed” metric: what is it good for?

Many moons ago, I listed the metrics we planned to use in evaluations. Well, here’s a shocker for you: when looking at actual charities’ activities, the reality is 10,000x as complicated as anything that can fit into these metrics.

We always knew we would run into the following problem: if Charity A saves 100 people from death and 1000 people from mild fever, while Charity B saves 150 people from death and 500 people from mild fever (assume the same cost), which is better? And we planned to tackle it in a slightly unusual way – rather than trying to “adjust” everything to the same terms (disability-adjusted life-years, or dollars of income gained, or some such nonsense), we would simply focus on how many lives were deeply changed. So in the case above, Charity B wins because it saved more people from death, and to heck with the fevers. Not perfect, but no solution to this problem is, and at least this approach is easy to understand and makes it easy for the donor to draw their own conclusions.

I reasoned that cases like this hypothetical should be rare anyway, as long as we’re separating charities into different causes (some aiming to educate children, others aiming to save lives, etc.) and only comparing charities when they share the same fundamental goal. For example, the charity that saves more lives is probably preventing more fevers too; when charities meaningfully diverge, it’s probably a sign that they belong in different causes.

Well, it hasn’t been even as simple as our messy expectation.

We initially wanted a cause to “save lives in Africa,” but we quickly realized that many charities are trying to prevent permanent blindness, debilitating skin disease, or a cleft lip that can lead to permanent malnutrition issues and ostracization. This isn’t exactly the same as saving a life, but doesn’t it seem pretty close? So we changed Cause 1 to “prevent death and extreme debilitation.”

Well, now we are in a pickle. What do you do when Charity A treats 10,000 cases of malaria and another performs 150 corrective surgeries? Now, bear in mind that malaria can be fatal; or it can lead to brain damage; or anemia; or it can just be a fever and stop there. And we don’t know how often it does each of these things (anyone have a source? Seriously, we can’t find it.) Now about those corrective surgeries. Some of them repair clefts (possibly, though not necessarily, life-ruining in the way described above). Others repair hand deformities or eyelid deformities, which we believe are usually cosmetic issues and not as bad as clefts (though how bad is the penalty for looking weird in these societies? We have no idea). And other surgeries are unclassified. Oh, and we don’t know for sure how many fall in each category – we just have to estimate based on past data.

How do you figure THAT one?

Here’s what I think. By default, I think the more comprehensive a charity is, the better. There are lots of things that make a community program – serving every need that everyone has – better than a narrower program (like running around distributing bednets). A more comprehensive program has tighter integration into the community, probably better relations with it, and better ability to observe it and make sure that people’s lives are improving on the whole (not trading some problems for others). On the other hand, there is exactly one thing that is worse about this approach – that you might help fewer people for the same funds. If malaria is cheap to prevent, running around and preventing everyone’s malaria could save many, many more lives than sitting in a village working on everything from AIDS to hangnail.

So, what I say is that the burden of proof is on the less comprehensive program to show that it’s getting much better results (same money) than the more comprehensive one. We’re still estimating our “dollars per life changed” metrics, using largely the same philosophy we started with – just focus on the really huge life changes – because we want to see if one program is obviously more cost-effective than another. If one program saves 10-100x as many lives as another (and I’d include cleft repair in “saving lives” for only this purpose), we’ll take our chances on it. But if it’s close at all, we’ll go with the communal/comprehensive program.

Since so much is unknown, that often makes things tough for distribution-type programs, but that matches with common sense too. If you’re trying to help people in a faraway land, you’d better be really confident to run around the countryside with nets, instead of sitting down in a village where you can see everything that’s going on.

So that’s what I’m thinking. Keep quantifying charity, as messily as we have to, knowing that we’re only going to use the wacky numbers we come up with if they show enormous differences in cost-effectiveness. Otherwise, we’re giving the edge to charities that work with one person/community at a time (instead of one problem at a time). What do you think?

Comments

“Dollars per life changed” metric: what is it good for? — 6 Comments

  1. The top of the Malaria wiki says that it kills 1-3M/year and that 650M are infected. Those seem very close to other numbers I have seen. Assume life expectancy in Africa is 40, average infection is at age 10 and lasts 30 years, and you are left with an estimate that it eventually kills 10% of those infected. That sounds roughly right.
    You can make another rough estimate based on the prevalence of the sickle cell trait. If it’s at equilibrium and is found in 10% of the population, and if from an expected number of children perspective it is fatal in a traditional setting, then a sickle cell gene has a 5% chance of pairing with another such gene, and the fitness gain from substantial resistance to malaria must also be about 5%. This is consistent with the above if the fitness loss from eventually fatal malaria is also 100% or if the fitness loss from eventually fatal malaria is smaller but other malaria victims also suffer a fitness loss.

  2. I should have been clearer – we do have a sense of what proportion of those infected with malaria die, but we have no sense of how often the non-death outcomes are extremely and permanently debilitating. We know that malaria can cause lasting brain damage, or it can just be a fever. That’s where we have trouble comparing X malaria cases to Y cleft surgeries.

  3. Well, the later part of my analysis based on sickle cell should probably make you fairly confident that consequences that would greatly reduce reproductive success are not much more common than death. You might safely assume that it doesn’t severely impact much more than 20% of the people infected.

  4. This is a topic close to my heart. After working my heart out for 3 years doing economic development in West Africa, I’m not sure what positive difference I made (if any). Few people want to come out and criticize development work undertaken in Africa for fear of hurting their career, or coming off negative or, racist. And we can’t just brush over the fact that the operating environment makes “success” in development so much more elusive in Africa than in places like Latin America or Eastern Europe (where I have also worked).

    Perhaps another way to get at charitable effectiveness in Africa is by process of elimination – we know when an effort stinks, and we know when one is good. We need to get that information out there, if for nothing else to begin the process of figuring out how to do more good than harm.

  5. Couldn’t agree more. If you can help us find the information you’re talking about – which efforts stink and which ones are good – please let me know. We are more than willing – in fact, determined – to put the truth out there; our challenge is finding it.

  6. Michael,

    People can be infected with malaria repeatedly over their lifetimes. Also, we have life expectancy figures for Africa.