The GiveWell Blog

Beware just-published studies

A recent study on health care in Ghana has been making the rounds – first on Megan McArdle’s blog and then on Marginal Revolution and Overcoming Bias. McArdle says the study shows “little relationship between consumption and health outcomes”; the other two title their posts “The marginal value of health care in Ghana: is it zero?” and “Free Docs Not Help Poor Kids.” In other words, the blogosphere take is that this is a “scary study” showing that making primary care free doesn’t work (or even perhaps that primary care doesn’t work).

But wait a minute. Here’s what the study found:

  • It followed 2,592 Ghanaian children (age 6-59 months). Half were randomly selected to receive free medical care, via enrollment in a prepayment plan. The medical care included diagnosis, antimalarials and other drugs, but not deworming.
  • Children with free treatment got medical care for 12% more of their episodes (2.8 vs. 2.5 episodes per year per person).
  • Health outcomes were assessed after 6 months:
    • Moderate anemia (the main measure) afflicted 36 of the children who got free care, vs. 37 of the children who didn’t.
    • Severe anemia afflicted 2 of the children who got free care, vs. 3 of the children who didn’t.
    • There were five deaths among children who got free care, vs. 4 among children who didn’t.
    • Parasite prevalence and nutrition status were also measured but not considered to be good measures of the program’s effects (since it did not include deworming or nutrition-centered care).

Would you conclude from this that the free medical care was “ineffective?” I wouldn’t – I’d conclude that the study ended up with very low sample size and low “power” because the children it studied were much healthier than expected. The researchers predicted an anemia prevalence of 10%, but the actual prevalence was just under 3%. Severe anemia and death were even rarer, making any comparison of those numbers (2 vs. 3 and 5 vs. 4) pretty meaningless. So in the end, we’re looking at a control group of 37 kids with moderate anemia and looking for a significant difference in the other group, from a 6-month program – and one that didn’t even address all possible causes of anemia (again, there was no deworming and it doesn’t appear that there was iron supplementation – the only relevant treatment was antimalarials).

Bottom line, free medical care didn’t appear to lead to improvement, but there also didn’t appear to be much room for improvement in this particular group. A similar critique appears in the journal (and points out that we don’t even know how often anemia can be expected to be attributed to malaria vs. parasites or other factors).

Some possible explanations for the relatively low levels of anemia include:

  • The presence of observers led everyone to make more use of primary care (the “Hawthorne effect,” a possibility raised by a Marginal Revolution commenter).
  • Less healthy people (and/or people who used primary care less) were less likely to stay enrolled in the study (7-8% dropped out), so that the people who stayed in had better health.
  • Or for some other reason (selection of village?), the researchers studied an unusually or unexpectedly healthy group. Perhaps a group that already uses primary care when it’s very important to do so, such that the “extra” visits paid for by the intervention were lower-stakes ones, or just weren’t enough (again, only a 12% difference) to impact major health outcomes among the small number of afflicted children.

All of these seem like real possibilities to me, and the numerical results found don’t seem to strongly suggest much of anything because of the low power (as the critique observes).

I saw a similar dynamic play out a month ago: Marginal Revolution linked a new study claiming that vaccination progress has been overstated, but a Center for Global Development scholar raises serious methodological concerns about the study. I haven’t examined this debate enough to have a strong opinion on it, and overestimation seems like a real concern; but we want to see how the discussion and reaction plays out before jumping to conclusions from the new study.

We’re all for healthy skepticism of aid programs, and we like reading new studies. But in drawing conclusions, we try to stick to studies that are a little older and have had some chance to be reviewed and discussed (and we generally look for responses and conflicting reactions). Doing so still leaves plenty of opportunities to be skeptical, as with the thoroughly discussed New York City Voucher Experiment and other ineffective social programs.


  • Robin Hanson on January 13, 2009 at 7:16 pm said:

    Yes we’d like to see studies with more power, and when I see them I’ll report on them. But are you really suggesting that we shouldn’t report this sort of study? If these sort of studies were coming out once a week, well sure we could be more selective. But randomized trials of medical price effects are pretty rare these days.

  • Holden on January 13, 2009 at 7:22 pm said:

    I share your wish to see more studies with strong methodology. But if a study has one problem that stops it from being meaningful or informative, it doesn’t help much that it avoids other common flaws.

    I’m certainly not saying this sort of study should be ignored or “covered up,” but I think that all 3 of the blogs I linked to presented it as much more than it is.

  • Robin Hanson on January 13, 2009 at 9:15 pm said:

    Low power is far from not being “meaningful or informative.” It certainly is informative, and worth calling attention to.

  • Holden on January 13, 2009 at 9:23 pm said:

    In what way do you find the study informative?

    My read is that two groups of ~1000 relatively healthy children were examined; some received a voucher for health services that don’t address major causes of anemia (aside from malaria whose connection and relative contribution to anemia isn’t well understood); and the two groups had the same low levels of anemia after 6 months. I don’t know what to do with that information except shrug.

    It is interesting to me that children with vouchers only conducted 12% more visits.

  • Lilliput on January 15, 2009 at 8:33 am said:

    Dear Blogger

    I’m wondering if you can help. I’ve just read that President Bush has been the President that has most helped Africa. Have you heard that before? and if so, can you elaborate on exactly what money and where did he give it to?

    I’m just interested to see what the truth is.

    Many Thanks

Comments are closed.