We completed our first year of research a few weeks ago, and are now starting up our second. (Our annual review and plan discuss what we’ve learned from our first year, and the many ways in which we’re changing our approach for year 2.)

We found some strong organizations the first time around, but our “bottom-up” approach (counting on our applicants to tell us about their activities) left us with a very partial picture of things. At this point we’re basically starting over, and trying to answer the following questions:

1. What are all the obstacles faced by people in different parts of the developing world?

I want to know as much as possible about the groups of people we’re helping (not just the diseases we’re fighting), because this helps us to (a) form better guesses about the most important problems to focus on; (b) focus on areas where a little aid goes a long way; (c) paint a picture for donors of what sort of difference a good program can make, beyond the usual idealized anecdotes or “cost per life/DALY saved” figures. (For example, when you save a life from malaria, what sort of a life is it? When you help improve someone’s income, what does that mean for what they’re able to buy?)

Ideally, we would know the following about as many different parts of the developing world as possible:

• Full details of the prevalence and severity of different health problems, including diseases (HIV/AIDS, tuberculosis, malaria, diarrhea, pneumonia, and NTDs), malnutrition (vitamin A deficiency, anemia, low weight-for-height and height-for-age), vision problems (including cataracts), and deformities (including cleft palate and obstetric fistula)
• Availability, and cost, of basic quality-of-life goods and services including health care, water, sanitation, electricity, financial services (savings, loans, insurance), and basic entertainment (televisions, radios, festivals).
• Availability, quality, and content of schooling.
• Common occupations, along with necessary skills/qualifications for each. I’m particularly interested in what it would take for someone to improve their occupation and income (are there plenty of opportunities if only they had basic help, along the lines of nutrition assistance or financial services? Or is the set of possible jobs highly limited?)
• What people most want to change about their lives, and what they most want help with.

I’ve been reading academic papers that answer some of these questions for certain areas using survey data; I’ve also talked to a couple of people who’ve spent significant time in the developing world, just to get a basic picture. I’ll share what I’ve learned so far in a future post.

2. What impact do different health problems generally have?

There are some diseases, such as AIDS, that we have a fairly good picture of in terms of their impact on quality/length of life. There are others – particularly the NTDs – that we know very little about. We need to examine enough medical literature to have a good sense of what possible symptoms are associated with different diseases, as well as a basic idea of how different health problems interrelate (for example, the extent to which malaria increases susceptibility to HIV/AIDS).

3. What are the most promising interventions and charities?

Answering #1 and #2, at least for some parts of the world, should have a major effect on how we think about #3, so we are focusing on #1 and #2 for now. The goal is to start with people and places, rather than programs. However, we want to make sure we’re checking out any particularly highly recommended, or otherwise promising, programs we come across (for example, recently we took a look at the Fred Hollows Foundation on several people’s recommendation).

We don’t expect to get anywhere close to “answering” all of the above questions, but that’s our framework for learning as much as we can to frame our investigations of charities. We have a lot of work to do. We’ll be sharing our findings as we go.

Several people have recommended that we look at the Fred Hollows Foundation. We have been shown calculations implying that they are preventing or curing a person’s blindness for every $20-60 they spend. As we continue our research on developing-world aid, we checked them out a bit ourselves.

The Fred Hollows Foundation’s programs include surgeries to cure blindness caused by cataracts and trachoma. These surgeries are relatively straightforward and can therefore be performed relatively inexpensively (at less than $10 per trachoma surgery, according to the Diseases Control Priorities Project). But the cost per surgery doesn’t tell the whole story – for example, we also want to know:

• How bad would patients’ vision be without surgery? While improving someone’s sight is always valuable, “curing blindness” means something very different to me from helping someone who previously had vision in one eye, or slightly impaired vision in both.
• How old the people are who receive the surgeries? Again, curing blindness always has some value, but it means more to me when it means giving someone a full life of healthy vision (or when it helps someone to care for their dependents).

The Fred Hollows Foundation conducted a 65-person post-operative survey in Cambodia that sheds light on the above questions. (You can see the full report here; this is the only survey of its kind that I found on their website.)

• 44% of those who received surgeries had been able to work before undergoing surgery, as they were “usually only blind in one eye or had some vision in both eyes” (Pg 11).
• 77% of those who received surgeries were over the age of 60 and another 21% were over the age of 41 (pg 10).

I’m excited by the idea of vision correction surgery; it’s cheap and tangible, even considering the above. But these sorts of details about who is being helped significantly change my idea of what you get for your donation with this kind of program, and I’m far from convinced that it ultimately represents a better “value” than our current top health-related charities.

The Career Academies initiative recently released a report on academies’ impact on students eight years after graduation. I’m fascinated by this report and this initiative because:

The Career Academies initiative rejects conventional wisdom about education.

For much of my life I’ve assumed that learning math, reading, and other “liberal arts” related skills is the key to later success in life. All of the K-12 education-focused charities we’ve examined appear to have the same unspoken assumption, stressing the importance of academic success (and generally measuring success through outcomes such as graduation rates and test scores). Last year, however, we started questioning this basic logic (for which we’ve found no empirical support).

Career Academies, while not ignoring academics, explicitly focus on preparing students for specific jobs (for example, see the Introduction of the full report recently released). And according to the evaluation, they are improving students’ earnings without improving their graduation rates or test scores (more below). Rather than assuming that the academic gap is at the heart of the achievement gap, this initiative is going straight after the latter.

The evaluation design used by Career Academies causally connects education with later life outcomes.

None of the K-12 education-focused charities we’ve examined make any attempt to examine later life outcomes, particularly earnings. Only three of them use the kind of experimental design that points strongly to effects of the program, rather than to selection bias issues. The Career Academies evaluation does both – making it the only study I know of that can plausibly discuss the effects of a particular K-12 program on the outcomes we really care about (not test scores, but earnings).

Randomized lotteries were used to assign the limited number of slots at Career Academies, and the lotteried-out students were compared to lotteried-in students a full eight years after graduation. We’re still waiting for the “technical” companion to the evaluation to be published so we can fully examine the methodology, but according to Evidence-Based Programs, the study had important strengths such as low attrition and an intention-to-treat approach that imply that any differences between the two groups (lotteried-in and lotteried-out) can be attributed to the effect of the schools themselves. It found that lotteried-in students report over $200/mo more in earnings (see Pg 13 of the full report), and that Career Academies students report higher earnings whether they’re classified as high-, medium-, or low-risk students (Pg 26).

This benefit came despite no apparent impact (see Pgs 28-32) on traditional measures of educational progress, including test scores, graduation rates, and college enrollment/completion. (Also note that these measures show enormous selection bias; a less rigorous, more typical study would have erroneously concluded that Career Academies do affect academic performance.)

I wish we’d discovered Career Academies earlier, and checked it out as thoroughly as our K-12 applicants. Rather than chasing small (or perhaps illusory) improvements in test scores and/or graduation rates, in the hopes that classic unproven assumptions about the importance of a high school education are correct, we might have funded an intervention with a truly different approach and a truly thorough commitment to making sure it’s changing lives, not just grades.

The National Academy Foundation focuses on the Career Academies approach. We haven’t done a thorough investigation of it, but you might want to check it out.

We haven’t seen – either in charities’ grant application materials or in our own independent research – much discussion of how organizations decide where to focus their resources. This question seems particularly important for international aid organizations, which often work in many regions all over the world (but by necessity have to ignore many more).

When thinking about this question myself, one of the criteria that occurs to me is the triage approach: find regions where a little bit of aid goes a long way. For example, let’s say we’re trying to decide where to expand malaria-centered interventions (such as insecticide-treated bednets). Niger, Guinea, Malawi, and Zambia are all countries with relatively high malaria mortality rates (0.17-0.2%, mostly concentrated among young children). However, Malawi and Zambia also have severe problems with HIV/AIDS: approximately 15% of the adult population is HIV positive and 0.6-0.9% of the total population dies from AIDS each year in these countries. By contrast, Niger and Guinea are significantly less affected by this disease (1-1.5% HIV positive, 0.04%-0.1% mortality rate).

From these numbers alone (and of course there is much more to the story), I’d prefer to expand a malaria-focused intervention in Niger or Guinea than in Malawi or Zambia. Our aim is to help people live fully enabled lives; resolving one problem, in an area where that problem is the primary obstacle a person faces (because s/he can now live a fully enabled life), will have a greater impact than resolving one problem where it is one of many a person faces. I’d prefer to protect children who won’t have to grow up with so much to fear from HIV/AIDS, if I have to choose (and I do).

Of course, to really make this sort of analysis work you need to look at a lot more than by-country rates for two diseases; you need to look at smaller regions (which can vary wildly within a country) and get a full sense of the different problems people face, including not only mortality risks but more general health problems (such as malnutrition), access to education and economic opportunity, and political stability. The goal is to find places where a humanitarian intervention can truly make the difference in giving someone a life of opportunity (not just solve one of an overwhelming set of problems).

References:

• Mortality data (2002 estimates) comes from the WHO’s Burden of Disease Project, available here.
• HIV prevalence data comes from UNAIDS and the WHO’s Report on the Global AIDS Epidemic. We accessed it through Gapminder. You can download it here.