Development Media International (DMI), an organization that “runs radio and TV campaigns to change behaviours and save lives in developing countries,” recently released preliminary midline results from a randomized controlled trial. These results imply that exposure to DMI’s content caused people to take up a variety of behaviors (including early initiation of breastfeeding and various measures for controlling diarrhea) at increased rates. We have not yet seen the full technical report and we have substantial questions about these results, but believe that the basic design of the study is a major improvement on previous evidence regarding similar programs.
If the results – and DMI’s cost-effectiveness calculations – held up, DMI could become a GiveWell top charity with substantially better estimated “cost per (equivalent) life saved” than our current top charities. Because of the nature of DMI’s intervention – attempting to change behavior through mass media – it is quite plausible to us that its cost per person reached is extremely small, and thus that the cost-effectiveness is extremely high. However, there are many questions we would need to look into before determining whether this is in fact the case.
As such, we plan to investigate DMI thoroughly this year. There is some possibility that we will decide to recommend it at the end of this calendar year, and some possibility that we will instead choose to hold off on deciding whether to recommend it until next year, when the study’s endline results (which we expect to give a more reliable picture of its impact) are released. Note that we recommend only charities that we have thoroughly vetted and spent substantial time considering alongside other contenders, and we have designed our process to be most up-to-date when it is most relevant (in December); so regardless of what we find, we are extremely unlikely to change any official recommendations ahead of that timeline. Our recommendations do not reflect real-time “best guesses,” but rather reflect the results of thorough investigation and deliberation.
However, as we state in our current writeup on the subject, we have previously found research on such programs to have important limitations and to be generally prone to selection bias. As such, we have been keeping tabs on DMI’s ongoing study, which – unlike previous work – uses randomization to isolate the impacts of its programming.
Preliminary, public discussion of DMI’s midline results became available recently, and we conducted a conversation with DMI representatives about the results (notes from this conversation are forthcoming). We note that:
- The preliminary results seem to indicate that DMI’s intervention had substantial impacts on behavior change (the easiest way to see this is to scan the rightmost column, “differences in differences (unadjusted)” in DMI’s table, though there is some other evidence available as well). We do not yet have complete information about the statistical significance of these findings (the public report gives p-values for four of ten behaviors).
- It appears that the treatment and control groups differed substantially at baseline (in terms of their behaviors and other indicators), making the comparison less than ideal. In addition, the control group (which did not receive DMI’s intervention) saw substantial improvement on the key indicators, which we don’t have an explanation for at this time. With that said, skimming the summary table provided by DMI, it appears that in many cases, the treatment group started with lower rates of a given behavior, and then caught or surpassed the control group, which (when viewed in the context of a randomized selection process) does seem to indicate a positive effect to us, though the magnitude may prove hard to estimate reliably.
- DMI provides an estimate that its intervention costs ~$60-300 per life saved, which would make it much stronger (around an order of magnitude stronger) than the strongest of our top charities. DMI also stated to us in conversation that the midline results are reasonably consistent with the assumptions that went into the estimate, though it appears to us that there will be substantial judgment calls in how to interpret the results in light of the estimate. We have not yet seen the technical details of DMI’s estimate. We expect our final estimate of DMI’s “cost per life saved” to be substantially less optimistic than DMI’s current estimate, but at this time it is hard to say by how much.
- DMI’s midline results come entirely from self-reported data on behavior, which is potentially a highly problematic measure for assessing the impact of a mass media campaign (since the media campaign may impact people’s perceptions of how they should respond to surveys more than it affects their actual behavior). It appears to us that DMI is aware of this issue, and states that its endline results (slated for 2015) will measure mortality.
- Based on DMI’s website and our discussion, we preliminarily perceive DMI so far to be unusually thoughtful and analytical in its self-evaluation and -analysis.
- Obtain the technical report behind the midline results and examine it closely. DMI has offered to share this with us in the next few weeks.
- Examine the technical details behind DMI’s “cost per life saved” estimate and assess them with our analysis of the midline results in mind. DMI has shared its model with us, though we have not yet examined it.
- Get more information about DMI’s plans and room for more funding and consider how our cost-effectiveness estimate might change in light of these.
- If warranted, further examine DMI via a site visit and create a full charity review.
Because of the strong study design, the degree of thoughtfulness and analysis we’ve seen so far from DMI, and the strong claimed cost-effectiveness and basic plausibility of the claim, we believe there is a substantial chance that DMI will eventually become a recommended charity, and that there is also a (smaller) chance that we will end up estimating substantially higher “bang for the buck” for DMI compared to our current top charities. If this turns out to be the case, we will hope to help close any funding gap of DMI’s.