This post is more than 9 years old

Recently, we discovered a few errors in our cost-effectiveness analysis of Development Media International (DMI). After correcting these errors, our best guess of DMI’s cost per life saved has increased from $5,236 per life saved to $7,264 per life saved. Additionally, we discovered some errors in our analysis of DMI’s finances. The corrected cost-effectiveness analysis is here.

These changes do not affect our bottom line about DMI, and we continue to consider it a standout charity.

What were the errors?
Crediting DMI with changes in antimalarial compliance. DMI broadcasts voice-acted stories embedded with health advice over radio into areas with high childhood mortalities. Among other advice, the messages encourage families to seek treatment for malaria when their child has a fever. However, the messages do not specifically address what is called “compliance”: completing the full course of malaria treatment, rather than treating the child only until symptoms stop.

DMI’s midline results found that antimalarial compliance had increased more in intervention areas than in control areas (the difference was not statistically significant). In our original analysis, we gave the option of crediting or not crediting DMI’s intervention with the increased compliance (with the default set to “yes, give credit”). We originally assumed that DMI’s campaign included messages specifically about complying with antimalarial treatment. Recently, we learned that it did not. While it’s possible that the DMI campaign had an effect on compliance without messaging on it, knowing that antimalarial compliance messages were not broadcast leads us to change our best guess. In our updated estimate, we have set the default compliance option to “no, don’t credit DMI for the increased compliance.” The option to credit DMI for the increase is still available in our model. (Note 1)

Not crediting DMI with increases in antimalarial compliance increased the cost per life saved by 38.7% (from $5,236 per life saved to $7,264 per life saved). This change accounts for the entire increase in headline cost per life saved, as the errors below are contained within the antimalarial compliance calculation, and thus only affect the headline cost per life saved if DMI is credited with improving antimalarial compliance.

Other errors in our cost-effectiveness analysis. In addition to mistakenly crediting DMI with the changes in antimalarial compliance, we discovered several other errors in our analysis. These errors did not cause any change in our cost per life saved estimate.

• Antimalarial compliance calculation: Two formulas in our compliance calculation used incorrect inputs. If we credited DMI for increasing antimalarial compliance, and did not fix other errors, these errors caused a 20.7% deflation in our cost per life saved (from $6,607 per life saved to $5,236 per life saved). (Note 2)
• Size of malaria mortality burden: We incorrectly used the upper bound of a mortality estimate instead of the point estimate. If we credited DMI for increasing antimalarial compliance, and did not fix other errors, this error caused an 11.2% deflation in our cost per life saved (from $5,899 per life saved to $5,236 per life saved). (Note 3)
• Cameroon data used in Burkina Faso calculation: We used data from Cameroon in our analysis of Burkina Faso, which we calculated as a comparison to the Cameroon cost per life saved. Holding other errors constant, this error caused a 125.5% inflation in our estimate of cost per life saved in Burkina Faso (from $446 per life saved to $1,006 per life saved). (Note 4)

Categorization of past expenditures. In our review of DMI, we included a categorization of DMI’s spending for 2011 to 2014. This categorization contained some errors, which caused our calculation of DMI’s total 2011-2014 spending to be $212,650 higher than its actual total spending (an inflation of 2.5%). Since we based our estimate of DMI’s costs in Cameroon on its projection of those costs rather than on past spending in Burkina Faso, these errors did not affect our final cost-effectiveness estimate for DMI. (Note 5)

How did we discover these errors?
We discovered these errors in two ways:

First, when revisiting our cost-effectiveness analyses (as part of our broader effort to improve our cost-effectiveness analyses this year), one of our research analysts discovered two of the errors (the antimalarial compliance calculation mistake and the size of malaria mortality burden mistake). As we were correcting the analysis, we discovered the Cameroon data in the Burkina Faso analysis, and realized that we weren’t certain if the DMI campaign messaged on antimalarial compliance. DMI clarified that its campaign did not message on antimalarial compliance.

Second, as part of our standard process, an analyst (who did not conduct the original work) carefully reviews a page before we publish it. We call this process a vet. While vetting our review of DMI, one of our research analysts discovered the expenditure categorization errors. This vet occurred after the page had been published. Our standard process is to vet pages before they are published, but in this case we published the page without a vet in order to meet our December 1st deadline for publishing our new recommendations last year.

We have added these errors to our mistakes page.

How do these corrections affect GiveWell’s view of DMI?
As noted above, these changes do not affect our bottom line about DMI, and we continue to consider it a standout charity.

In particular, the change as a result of our error is small relative to our uncertainty about other inputs into our model. Specifically:

• Our estimate of $7,264 per life saved relies solely on data from Cameroon because we guessed that Cameroon was the country where DMI was most likely to spend additional funds. We remain uncertain about where DMI will spend additional funds, and a more robust estimate of its cost-effectiveness would also incorporate estimates from other countries.
• Our estimate credits DMI with affecting behavior for pneumonia and diarrhea but not malaria because DMI’s midline results only measured a 0.1% increase in treatment seeking for malaria in the intervention group compared to the control group. It is arguably unlikely that DMI would cause behavior change for pneumonia and diarrhea treatment-seeking, but not malaria treatment-seeking, given that the promoted behaviors are relatively similar.
• As we wrote last December, we are uncertain about whether we should put more credence in our estimate of DMI’s cost-effectiveness based on available data about behavior change, or its own projection. Our cost-effectiveness analysis predicts a 3.2% decline in child mortality, but DMI’s, estimated by the people carrying out a study and paying the considerable expenses associated with it, predicts 10-20%. More in our December 2014 post.

We have not incorporated the above considerations into our cost-effectiveness analysis, but we would guess that incorporating the above could cause changes in our estimate of DMI’s cost-effectiveness significantly larger than the 38% change due to the error discussed in this post.

---

Footnotes

Note 1: See Cell D76.

Note 2: We are not sure how often ceasing antimalarial treatment prematurely is as bad (for the survival of the child) as not giving antimalarials at all; without an authoritative source we guessed that this is true 25% of the time.

One formula in our spreadsheet left this 25% figure out of the calculation, effectively assuming that 100% of non-compliance cases were as bad as not giving any antimalarials at all. Because the estimate now defaults to not crediting for compliance (see previous error), this error does not affect our updated headline figure for cost per life saved.

In our original cost-effectiveness estimate, Cell D88 (effective coverage before the campaign) erroneously incorporated Cell D75 (raw compliance before the campaign) as an input. In the updated cost-effectiveness estimate, Cell D88 incorporates Cell D79 (effective compliance accounting for the benefit from non-compliance).

In the original cost-effectiveness estimate, Cell D92 (effective coverage after the campaign) erroneously incorporated Cell D77 (raw compliance after the campaign) as an input. In the updated cost-effectiveness estimate, Cell D92 incorporates Cell D80 (effective compliance accounting for the benefit from non-compliance).

Our estimate of lives saved by pneumonia treatment did not contain an equivalent error, and we did not include an equivalent compliance factor for diarrhea since treatment is only needed for as long as symptoms persist. Our model still defaults to crediting DMI with an increase in pneumonia compliance, because DMI’s campaign messaged specifically on completing courses of pneumonia treatment.

Note 3: We use the Institute for Health Metrics and Evaluation’s data visualization tool to estimate the number of deaths from specific causes in target countries. For malaria deaths, ages 1-4, in Cameroon, we incorrectly used the upper bound of the estimate (18,724.2 deaths), rather than the point estimate (9,213.71 deaths). The RCT midline results did not report an increase in malaria treatment coverage, though antimalarial compliance did increase. Because the estimate now defaults to not crediting for compliance (see above), this error does not affect our updated headline figure for cost per life saved.

In the original cost-effectiveness estimate, Cell D106 erroneously included the upper bound of age 1-4 deaths from malaria (see Cell E106 for search parameters and calculation). In the updated cost-effectiveness estimate, Cell D106 includes the point estimate for age 1-4 deaths from malaria (see Cell E106 for search parameters and calculation).

Note 4: This comparison did not affect our headline cost per life saved, because we think a campaign in a country similar to Cameroon is a more likely use of marginal unrestricted funding directed to DMI. The Burkina Faso analysis was structurally the same as the Cameroon analysis, and included the compliance calculation error described above. In addition, the Burkina Faso analysis incorrectly used information about Cameroon, rather than Burkina Faso (specifically the number of under-5 deaths from malaria, pneumonia, and diarrhea; and the campaign cost estimate).

See columns G to I in the cost-effectiveness spreadsheet for the model of the Burkina Faso campaign. See cells G105, G106, and G107 for the data on deaths from pneumonia, malaria, and diarrhea. See cell G117 for the Burkina Faso campaign cost. In the original cost-effectiveness estimate, all of these cells duplicated the data for Cameroon (see D105, D106, D107, and D117). In the updated cost-effectiveness analysis, these cells have been updated with data pertaining to Burkina Faso.

Note 5: Our categorization process involved assigning a category code to each line item of DMI’s budget, then aggregating the subtotals for each category. Two types of errors occurred during this process:

• A line item was coded to an incorrect category that wasn’t aggregated, causing the item to not be counted in the subtotals.
• Some formulas for aggregating category subtotals drew inputs from incorrect ranges, causing some items to be double-counted.

DMI has requested that its budget be kept private. Because our categorization process involved coding the line items of DMI’s budget, we are unable to share our categorization files and the specific details about these errors.