The GiveWell Blog

Our ongoing review of Development Media International

Development Media International (DMI) produces radio and television programs in developing countries that encourage people to adopt improved health practices, such as exclusive breastfeeding of infants and seeking treatment for symptoms associated with fatal diseases. The program aims to reduce mortality of children under five years old.

In May, we wrote that we were considering DMI for a 2014 top charity recommendation. We’ve now spent a considerable amount of time talking to DMI and analyzing documents DMI shared with us. This post shares what we’ve learned so far and what questions we’re planning to focus on throughout the rest of our investigation. (For more, see our detailed interim review.)

DMI has successfully completed the first phase of our investigation process and we view it as a contender for a recommendation this year. We now plan to (a) make a $100,000 grant to DMI (as part of our “top charity participation grants,” funded by Good Ventures) and (b) continue our analysis to determine whether or not we should recommend DMI to donors at the end of the year, including conducting a site visit to Burkina Faso.

Reasons we prioritized DMI

We’ve long been interested in programs that aim to use mass media (e.g., radio or television programming) to promote and disseminate messages on potential life-saving practices. It’s quite plausible to us that messages on TV or the radio could influence behavior, and could reach large numbers of people for relatively low costs, leading to high cost-effectiveness in terms of lives saved or improved per dollar spent, but we previously deprioritized our work in this area due to limitations in the available evidence of effectiveness.

DMI is currently conducting a randomized controlled trial (RCT) of its program, preliminary results from which became available in April.

Questions we hope to answer in our ongoing analysis

How robust are the midline results from the RCT?

Our level of confidence in the success of DMI’s program rests heavily on the midline results from the RCT, but there are reasons these results should be interpreted with caution. In particular:

  • The treatment group (i.e., the regions that were randomly selected to hear DMI’s broadcasts) had noticeably higher levels of child mortality and less access to healthcare at baseline than the control group. Details in our interim review.
  • While DMI plans to collect data on mortality, the only results reported thus far are based on self-reported behavior change, the reliability of which is questionable.

Many details of the RCT are not yet available publicly as the study is ongoing, and we have a number of questions about it that could affect our view of DMI’s impact. In particular, we would like to know more about the activities of other health programs in Burkina Faso during the trial period, and the extent to which the midline results are driven by certain villages or regions versus consistent behavior change across all participating areas.

It’s also important to note that the evidence for DMI’s program relies heavily on a single unpublished RCT; interventions such as bednets and cash transfers are supported by multiple peer-reviewed RCTs and other evidence.

How representative is DMI’s impact in Burkina Faso of its likely impact in other countries?

There are some reasons to expect that DMI’s future results will vary from the RCT. For example, much of DMI’s expected impact comes from behavior changes that require access to health services or products to be effective, such as seeking treatment when a child displays symptoms of malaria. DMI’s ability to predict access in other countries is critical to predicting impact, and may be limited. Details in our interim review.

How cost effective is DMI’s program?

We have not yet completed a full cost effectiveness estimate for DMI’s work but plan to do so for our final review of DMI.

DMI estimates that the cost effectiveness of its intervention is extremely strong relative to other cost effective interventions (for example, more than 10x stronger than our estimate of our strongest top charities). We expect our final estimate of DMI’s “cost per life saved” to be substantially less optimistic, though still within the range of our current priority programs. Details in our interim review.

Will future work be as impactful as past work, and how will we know?

We do not know how DMI will design its attempts to measure its future programs’ impact on behavior change.

 

Comments

  • KOUSSE Sylvain on October 10, 2014 at 5:35 am said:

    C’est vraiment encourageant de voir que ce que nous faisons comme travail a un impact vérifiable. Et en plus cela nous rejouit de savoir qu’il y a des gens qui voient les efforts que nous fournissons et qui les apprécient à sa juste valeur. Merci à la fondation Give Well .

  • Ian Turner on October 12, 2014 at 11:50 am said:

    Thanks for this. It certainly seems like an out-of-the-box approach to development.

    One question that comes to mind for me is how DMI maintains control over its translations. I assume that there are too many local languages for DMI staff to speak them all, so it seems possible that a translation could be inaccurate, inauthentic, or even fraudulent. This seems like a potential problem with the live shows also.

  • Timothy Telleen-Lawton on October 20, 2014 at 5:12 pm said:

    Thanks for the comments.

    Ian – Yes, good translation is a potential concern as DMI broadcasts in 6 languages in Burkina Faso. DMI scriptwriters in Ouagadougou actually speak each of the 6 languages, and the minute-long spots are recorded in that office. The issue may be greater for the longer drama modules which are outlined and distributed in french and improvised live in the local language at each station. I’d hope that any consistent message failures at translation would come through in the feedback monitoring that the DMI Research Team does with radio listeners. They have not picked up on this being an issue so far; I believe it is unlikely to be a significant problem.

Comments are closed.