Development Media International (DMI) produces radio and television programs in developing countries that encourage people to adopt improved health practices, such as exclusive breastfeeding of infants and seeking treatment for symptoms associated with fatal diseases. The program aims to reduce mortality of children under five years old.

In May, we wrote that we were considering DMI for a 2014 top charity recommendation. We’ve now spent a considerable amount of time talking to DMI and analyzing documents DMI shared with us. This post shares what we’ve learned so far and what questions we’re planning to focus on throughout the rest of our investigation. (For more, see our detailed interim review.)

DMI has successfully completed the first phase of our investigation process and we view it as a contender for a recommendation this year. We now plan to (a) make a $100,000 grant to DMI (as part of our “top charity participation grants,” funded by Good Ventures) and (b) continue our analysis to determine whether or not we should recommend DMI to donors at the end of the year, including conducting a site visit to Burkina Faso.

Reasons we prioritized DMI

We’ve long been interested in programs that aim to use mass media (e.g., radio or television programming) to promote and disseminate messages on potential life-saving practices. It’s quite plausible to us that messages on TV or the radio could influence behavior, and could reach large numbers of people for relatively low costs, leading to high cost-effectiveness in terms of lives saved or improved per dollar spent, but we previously deprioritized our work in this area due to limitations in the available evidence of effectiveness.

DMI is currently conducting a randomized controlled trial (RCT) of its program, preliminary results from which became available in April.

Questions we hope to answer in our ongoing analysis

How robust are the midline results from the RCT?

Our level of confidence in the success of DMI’s program rests heavily on the midline results from the RCT, but there are reasons these results should be interpreted with caution. In particular:

• The treatment group (i.e., the regions that were randomly selected to hear DMI’s broadcasts) had noticeably higher levels of child mortality and less access to healthcare at baseline than the control group. Details in our interim review.
• While DMI plans to collect data on mortality, the only results reported thus far are based on self-reported behavior change, the reliability of which is questionable.

Many details of the RCT are not yet available publicly as the study is ongoing, and we have a number of questions about it that could affect our view of DMI’s impact. In particular, we would like to know more about the activities of other health programs in Burkina Faso during the trial period, and the extent to which the midline results are driven by certain villages or regions versus consistent behavior change across all participating areas.

It’s also important to note that the evidence for DMI’s program relies heavily on a single unpublished RCT; interventions such as bednets and cash transfers are supported by multiple peer-reviewed RCTs and other evidence.

How representative is DMI’s impact in Burkina Faso of its likely impact in other countries?

There are some reasons to expect that DMI’s future results will vary from the RCT. For example, much of DMI’s expected impact comes from behavior changes that require access to health services or products to be effective, such as seeking treatment when a child displays symptoms of malaria. DMI’s ability to predict access in other countries is critical to predicting impact, and may be limited. Details in our interim review.

How cost effective is DMI’s program?

We have not yet completed a full cost effectiveness estimate for DMI’s work but plan to do so for our final review of DMI.

DMI estimates that the cost effectiveness of its intervention is extremely strong relative to other cost effective interventions (for example, more than 10x stronger than our estimate of our strongest top charities). We expect our final estimate of DMI’s “cost per life saved” to be substantially less optimistic, though still within the range of our current priority programs. Details in our interim review.

Will future work be as impactful as past work, and how will we know?

We do not know how DMI will design its attempts to measure its future programs’ impact on behavior change.