The GiveWell Blog

Corrections in our review of Development Media International

(updated on: ) | by Milan Griffes

Recently, we discovered a few errors in our cost-effectiveness analysis of Development Media International (DMI). After correcting these errors, our best guess of DMI’s cost per life saved has increased from $5,236 per life saved to $7,264 per life saved. Additionally, we discovered some errors in our analysis of DMI’s finances. The corrected cost-effectiveness analysis is here.

These changes do not affect our bottom line about DMI, and we continue to consider it a standout charity.

What were the errors?

Crediting DMI with changes in antimalarial compliance. DMI broadcasts voice-acted stories embedded with health advice over radio into areas with high childhood mortalities. Among other advice, the messages encourage families to seek treatment for malaria when their child has a fever. However, the messages do not specifically address what is called “compliance”: completing the full course of malaria treatment, rather than treating the child only until symptoms stop.

DMI’s midline results found that antimalarial compliance had increased more in intervention areas than in control areas (the difference was not statistically significant). In our original analysis, we gave the option of crediting or not crediting DMI’s intervention with the increased compliance (with the default set to “yes, give credit”). We originally assumed that DMI’s campaign included messages specifically about complying with antimalarial treatment. Recently, we learned that it did not. While it’s possible that the DMI campaign had an effect on compliance without messaging on it, knowing that antimalarial compliance messages were not broadcast leads us to change our best guess. In our updated estimate, we have set the default compliance option to “no, don’t credit DMI for the increased compliance.” The option to credit DMI for the increase is still available in our model. (Note 1)

Not crediting DMI with increases in antimalarial compliance increased the cost per life saved by 38.7% (from $5,236 per life saved to $7,264 per life saved). This change accounts for the entire increase in headline cost per life saved, as the errors below are contained within the antimalarial compliance calculation, and thus only affect the headline cost per life saved if DMI is credited with improving antimalarial compliance.

Other errors in our cost-effectiveness analysis. In addition to mistakenly crediting DMI with the changes in antimalarial compliance, we discovered several other errors in our analysis. These errors did not cause any change in our cost per life saved estimate.

  • Antimalarial compliance calculation: Two formulas in our compliance calculation used incorrect inputs. If we credited DMI for increasing antimalarial compliance, and did not fix other errors, these errors caused a 20.7% deflation in our cost per life saved (from $6,607 per life saved to $5,236 per life saved). (Note 2)
  • Size of malaria mortality burden: We incorrectly used the upper bound of a mortality estimate instead of the point estimate. If we credited DMI for increasing antimalarial compliance, and did not fix other errors, this error caused an 11.2% deflation in our cost per life saved (from $5,899 per life saved to $5,236 per life saved). (Note 3)
  • Cameroon data used in Burkina Faso calculation: We used data from Cameroon in our analysis of Burkina Faso, which we calculated as a comparison to the Cameroon cost per life saved. Holding other errors constant, this error caused a 125.5% inflation in our estimate of cost per life saved in Burkina Faso (from $446 per life saved to $1,006 per life saved). (Note 4)

Categorization of past expenditures. In our review of DMI, we included a categorization of DMI’s spending for 2011 to 2014. This categorization contained some errors, which caused our calculation of DMI’s total 2011-2014 spending to be $212,650 higher than its actual total spending (an inflation of 2.5%). Since we based our estimate of DMI’s costs in Cameroon on its projection of those costs rather than on past spending in Burkina Faso, these errors did not affect our final cost-effectiveness estimate for DMI. (Note 5)

How did we discover these errors?

We discovered these errors in two ways:

First, when revisiting our cost-effectiveness analyses (as part of our broader effort to improve our cost-effectiveness analyses this year), one of our research analysts discovered two of the errors (the antimalarial compliance calculation mistake and the size of malaria mortality burden mistake). As we were correcting the analysis, we discovered the Cameroon data in the Burkina Faso analysis, and realized that we weren’t certain if the DMI campaign messaged on antimalarial compliance. DMI clarified that its campaign did not message on antimalarial compliance.

Second, as part of our standard process, an analyst (who did not conduct the original work) carefully reviews a page before we publish it. We call this process a vet. While vetting our review of DMI, one of our research analysts discovered the expenditure categorization errors. This vet occurred after the page had been published. Our standard process is to vet pages before they are published, but in this case we published the page without a vet in order to meet our December 1st deadline for publishing our new recommendations last year.

We have added these errors to our mistakes page.

How do these corrections affect GiveWell’s view of DMI?

As noted above, these changes do not affect our bottom line about DMI, and we continue to consider it a standout charity.

In particular, the change as a result of our error is small relative to our uncertainty about other inputs into our model. Specifically:

  • Our estimate of $7,264 per life saved relies solely on data from Cameroon because we guessed that Cameroon was the country where DMI was most likely to spend additional funds. We remain uncertain about where DMI will spend additional funds, and a more robust estimate of its cost-effectiveness would also incorporate estimates from other countries.
  • Our estimate credits DMI with affecting behavior for pneumonia and diarrhea but not malaria because DMI’s midline results only measured a 0.1% increase in treatment seeking for malaria in the intervention group compared to the control group. It is arguably unlikely that DMI would cause behavior change for pneumonia and diarrhea treatment-seeking, but not malaria treatment-seeking, given that the promoted behaviors are relatively similar.
  • As we wrote last December, we are uncertain about whether we should put more credence in our estimate of DMI’s cost-effectiveness based on available data about behavior change, or its own projection. Our cost-effectiveness analysis predicts a 3.2% decline in child mortality, but DMI’s, estimated by the people carrying out a study and paying the considerable expenses associated with it, predicts 10-20%. More in our December 2014 post.

We have not incorporated the above considerations into our cost-effectiveness analysis, but we would guess that incorporating the above could cause changes in our estimate of DMI’s cost-effectiveness significantly larger than the 38% change due to the error discussed in this post.

Footnotes

Note 1: See Cell D76.

Note 2: We are not sure how often ceasing antimalarial treatment prematurely is as bad (for the survival of the child) as not giving antimalarials at all; without an authoritative source we guessed that this is true 25% of the time.

One formula in our spreadsheet left this 25% figure out of the calculation, effectively assuming that 100% of non-compliance cases were as bad as not giving any antimalarials at all. Because the estimate now defaults to not crediting for compliance (see previous error), this error does not affect our updated headline figure for cost per life saved.

In our original cost-effectiveness estimate, Cell D88 (effective coverage before the campaign) erroneously incorporated Cell D75 (raw compliance before the campaign) as an input. In the updated cost-effectiveness estimate, Cell D88 incorporates Cell D79 (effective compliance accounting for the benefit from non-compliance).

In the original cost-effectiveness estimate, Cell D92 (effective coverage after the campaign) erroneously incorporated Cell D77 (raw compliance after the campaign) as an input. In the updated cost-effectiveness estimate, Cell D92 incorporates Cell D80 (effective compliance accounting for the benefit from non-compliance).

Our estimate of lives saved by pneumonia treatment did not contain an equivalent error, and we did not include an equivalent compliance factor for diarrhea since treatment is only needed for as long as symptoms persist. Our model still defaults to crediting DMI with an increase in pneumonia compliance, because DMI’s campaign messaged specifically on completing courses of pneumonia treatment.

Note 3: We use the Institute for Health Metrics and Evaluation’s data visualization tool to estimate the number of deaths from specific causes in target countries. For malaria deaths, ages 1-4, in Cameroon, we incorrectly used the upper bound of the estimate (18,724.2 deaths), rather than the point estimate (9,213.71 deaths). The RCT midline results did not report an increase in malaria treatment coverage, though antimalarial compliance did increase. Because the estimate now defaults to not crediting for compliance (see above), this error does not affect our updated headline figure for cost per life saved.

In the original cost-effectiveness estimate, Cell D106 erroneously included the upper bound of age 1-4 deaths from malaria (see Cell E106 for search parameters and calculation). In the updated cost-effectiveness estimate, Cell D106 includes the point estimate for age 1-4 deaths from malaria (see Cell E106 for search parameters and calculation).

Note 4: This comparison did not affect our headline cost per life saved, because we think a campaign in a country similar to Cameroon is a more likely use of marginal unrestricted funding directed to DMI. The Burkina Faso analysis was structurally the same as the Cameroon analysis, and included the compliance calculation error described above. In addition, the Burkina Faso analysis incorrectly used information about Cameroon, rather than Burkina Faso (specifically the number of under-5 deaths from malaria, pneumonia, and diarrhea; and the campaign cost estimate).

See columns G to I in the cost-effectiveness spreadsheet for the model of the Burkina Faso campaign. See cells G105, G106, and G107 for the data on deaths from pneumonia, malaria, and diarrhea. See cell G117 for the Burkina Faso campaign cost. In the original cost-effectiveness estimate, all of these cells duplicated the data for Cameroon (see D105, D106, D107, and D117). In the updated cost-effectiveness analysis, these cells have been updated with data pertaining to Burkina Faso.

Note 5: Our categorization process involved assigning a category code to each line item of DMI’s budget, then aggregating the subtotals for each category. Two types of errors occurred during this process:

  • A line item was coded to an incorrect category that wasn’t aggregated, causing the item to not be counted in the subtotals.
  • Some formulas for aggregating category subtotals drew inputs from incorrect ranges, causing some items to be double-counted.

DMI has requested that its budget be kept private. Because our categorization process involved coding the line items of DMI’s budget, we are unable to share our categorization files and the specific details about these errors.

Update on GiveWell’s web traffic / money moved: Q1 2015

(updated on: ) | by Tyler Heishman

In addition to evaluations of other charities, GiveWell publishes substantial evaluation of itself, from the quality of its research to its impact on donations. We publish quarterly updates regarding two key metrics: (a) donations to top charities and (b) web traffic.

The tables and chart below present basic information about our growth in money moved and web traffic in the first quarter of 2015 compared to the last two years (note 1).

Money moved and donors: first quarter

Table_2015Q1MoneyMoved.png

Money moved by donors who have never given more than $5,000 in a year increased 78% to about $760,000. The total number of donors in the first quarter increased to about 3,400. This was up 70% compared to last year, roughly consistent with the last year’s growth.

Most of our money moved is donated near the end of the year (we tracked about 70% of the total in the fourth quarter each of the last two years) and is driven by a relatively small number of large donors. Because of this, our year-to-date total money moved provides relatively limited information, and we don’t think we can reliably predict our year-end money moved (note 2). Mid-year we primarily use data on donations from smaller donors, rather than total money moved, to give a rough indication of how our influence on donations is growing.

Web traffic through April 2015

Table_2015Q1WebTraffic.png

Growth in web traffic excluding Google AdWords increased moderately in the first quarter. Last year, we saw a drop in total web traffic because we removed ads on searches that we determined were not driving high quality traffic to our site (i.e. searches with very high bounce rates and very low pages per visit).

GiveWell’s website receives elevated web traffic during “giving season” around December of each year. To adjust for this and emphasize the trend, the chart below shows the rolling sum of unique visitors over the previous twelve months, starting in December 2009 (the first period for which we have 12 months of reliable data due to an issue tracking visits in 2008).

Chart_2015Q1WebTraffic.png

We use web analytics data from two sources: Clicky and Google Analytics (except for those months for which we only have reliable data from one source). The data on visitors to our website differs between the two sources. We do not know the cause of discrepancy (though a volunteer with a relevant technical background looked at the data for us to try to find the cause; he didn’t find any obvious problems with the data). (Note on how we count unique visitors.)

The raw data we used to generate the chart and table above (as well as notes on the issues we’ve had and adjustments we’ve made) is in this spreadsheet.


Note 1: Since our 2012 annual metrics report we have shifted to a reporting year that starts on February 1, rather than January 1, in order to better capture year-on-year growth in the peak giving months of December and January. Therefore, metrics for the “first quarter” reported here are for February through April.

Note 2: In total, GiveWell donors directed $1.76 million to our top charities in the first quarter of this year, compared with $1.45 million that we had tracked in the first quarter of 2014. For the reason described above, we don’t find this number to be particularly meaningful at this time of year.

Note 3: We count unique visitors over a period as the sum of monthly unique visitors. In other words, if the same person visits the site multiple times in a calendar month, they are counted once. If they visit in multiple months, they are counted once per month. Google Analytics provides ‘unique visitors by traffic source’ while Clicky provides only ‘visitors by traffic source.’ For that reason, we primarily use Google Analytics data in the calculations to exclude AdWords visitors.

GiveWell summer fellowship

| by Elie

We’re planning to host a one week fellowship this summer at our office in San Francisco for students more than a year away from graduation (e.g., first years or sophomores in college), who would be ineligible for our Summer Research Analyst position.

We expect to have 4-8 fellows, who will spend a week doing standard GiveWell work, getting to know staff, and getting a feel for GiveWell and Open Philanthropy research.

For more, see our page with details and application instructions.

History of philanthropy case study: Pew and drug safety legislation

(updated on: ) | by Elie

Tamara Mann Tweel, who has been working for us on our history of philanthropy project, has completed a case study of a Pew Charitable Trusts (“Pew”) program focused on drug supply chain safety legislation in 2012.

The report concludes:

Pew put drug supply chain safety concerns on the legislative agenda in 2011 and actively built the coalition that ensured its passage in 2012. The team also assisted with and vetted the language of the 2012 bill and made sure that weak policy proposals did not supplant strong ones. Pew accomplished these goals largely by capitalizing on their expertise and by deploying the multi-pronged strategy [explained in this report]. Pew did not act alone. It had the assistance of strong industry partners, FDA officials, and key congressmen and senators… [Pew] became the single most important non-government and non-industry player in the field. They brought the stakeholders together, gave them the necessary information to pursue the topic, and demonstrated the viability of actual policy.

The full case study is available here.

Our impression is that Pew has had fairly concrete impact on policy in a variety of areas (note that we’ve separately investigated its work on public safety performance, in the context of a $3 million grant we made to Pew). While its model is not the only or necessarily best one for policy-oriented philanthropy, we believe it is one example of a generally well-executed and impactful model, and that Pew is a group we can learn from. We chose to do a case study on its work in order to examine our beliefs on this point, and we believe that the case study has generally been consistent with our views on Pew’s work.

Read the full case study here

Funder-initiated startups

(updated on: ) | by Holden

We’ve come across many cases where a funder took a leading role in creating a now-major nonprofit. This has been surprising to us: it intuitively seems that the people best suited to initiate new organizations are the people who can work full-time on conceiving an organization, fundraising for it, and doing the legwork to create it. Most successful companies seem to have been created by entrepreneurs rather than investors, and the idea that a philanthropist can “create” a successful organization (largely through concept development, recruiting and funding, without full-time operational involvement) seems strange. Yet we’ve seen many strong examples:

This is not anything approaching a comprehensive list. It’s a set of organizations we’ve come across in our work, many of which we perceive as prominent and important. I would struggle to think of many analogous cases of for-profit companies for which the original concept, recruitment, etc. came from investors rather than full-time founding employees.

Assuming this difference is real, what might explain it? While I’m not sure, I’ll list a few speculative possibilities:

  • A nonprofit startup must raise funds from a relatively thin and fragmented market. Investors ultimately all want the same thing (returns); philanthropists want very different things, and a nonprofit won’t be able to get off the ground if it can’t find a match. One symptom of “philanthropists want different things” is that nonprofit proposals are generally highly tailored to the values of funders. Thus, people with ideas may choose not to write up and shop proposals until they’ve identified a highly interested funder.
  • A nonprofit startup also doesn’t have an analogous option to bootstrapping to prove its value and raise its negotiating power. It can hope eventually to reach the point where its donor base is highly diversified, but early on nonprofits will very often live or die by major funders’ preferences.
  • Starting a new company is generally associated with high (financial) risk and high potential reward. But without a solid source of funding, starting a nonprofit means taking high financial risk without high potential reward. Furthermore, some nonprofits (like some for-profits) are best suited to be started by people relatively late in their careers; the difference is that late-career people in the for-profit sector seem more likely to have built up significant savings that they can use as a cushion. This is another reason that funder interest can be the key factor in what nonprofits get started.
  • The dynamics of competition may be different. If someone sees a for-profit with a good concept and poor execution, s/he might start a competitor. Someone who sees a nonprofit with a good concept and poor execution (and a solid funding situation) might be more likely to try to improve the nonprofit, e.g. by working for it. If true, this might make funder-initiated organizations – which, it seems, would be hard to find the right leadership match for – more viable on the nonprofit side than the for-profit side.

Our tentative view is that funders should think of “creating an organization” as a viable possibility, though as something of a last resort, since it is likely to be a much more intensive project than supporting an existing organization.

Our updated agenda for science philanthropy

(updated on: ) | by Holden

We’re hoping to set the Open Philanthropy Project’s initial priorities within scientific research this year. That means being in a place roughly comparable to where we currently are on U.S. policy and global catastrophic risks: having a ranked list of focus areas and goals for hiring and grantmaking.

The process is going to have to look very different. For both U.S. policy and global catastrophic risks, we were able to do a relatively large number of “shallow investigations,” in which we quickly got a sense for the importance, tractability, and crowdedness of a cause. By contrast, it seems to us that investigating even a single cause within scientific research – to the level of understanding we achieved with shallow investigations in other areas – is a major project.

Our neglected goal investigations have been proceeding slowly. We’ve been working with scientific advisors who have limited time available, and it’s taken them significant effort to (a) get up to speed on a given area of research (e.g., R&D targeting tuberculosis control); (b) have initial conversations about the most promising paths within the area; (c) begin thinking about how to assess which of these directions seem most promising. One of our major bottlenecks is scientific advisory capacity, and that’s something we hope to change. But even if we did, we wouldn’t anticipate being able to do a large number of shallow investigations of science causes. Meanwhile, investigating other possible approaches to science – such as breakthrough fundamental science or translational science – seems likely to be even more challenging than investigating neglected goals.

Our working plan for the moment aims to set priorities mostly via very high-level, comparative investigations. Our specific goals are as follows:

1. Create a prioritized list of neglected goals via conversations with unusually broad scientists as well as people in the effective altruist community. I have a draft list currently, based on suggestions I’ve picked up over the years. After getting input from 10-15 people – with a mix of junior and senior, science-focused and effective-altruism-focused – I expect to have a reasonable (though far from complete) sense for what is most worth prioritizing. As an aside, I wouldn’t expect this approach to work very well for U.S. policy, where it’s hard to find people who have a good sense both for politics and for our values. But for identifying neglected goals, I believe I can identify people who combine these qualities.

This investigation will not be specific to life sciences. I hope to speak with people who have broad interests and expertise and can identify potential technologies that would be worth more effort to develop than is currently being put in.

We will do as many cause-specific investigations as we can, prioritizing those that rank at the top of our list, in order to further inform our priorities.

For highly prioritized neglected goals, we may (after this year) move toward forming grant advisory committees and providing direct funding of relevant research, or we may think about other ways to raise the profile of the goals in question. My impression is that providing funding for a thin field can create something of a self-reinforcing dynamic, since research often raises new interesting questions and makes it easier and more desirable for other researchers to work on similar issues; I hope to investigate this impression further (more below).

2. Get a sense for potential systemic issues in fields other than life sciences. While “neglected goals” refers to cases where there isn’t enough investment in a particular social problem, “systemic issues” refers to cases where the system for supporting scientific research seems to be falling short on its own terms. I’ve written at length about two potential examples in life sciences: Translational science and the valley of death, Breakthrough fundamental science. (Some other systemic issues are discussed in Science policy and infrastructure as well as previous posts on reproducibility-related issues and open science.)

If we decided to prioritize addressing systemic issues in a particular field, we’re not sure exactly how we’d do it. We might focus on supporting work that directly proposes, and advocates for, improvements to the system rather than on directly funding research that is undervalued due to systemic issues. But both would be strong possibilities.

My understanding of systemic issues in life sciences is high-level and quite limited, but it is sufficient to have a basic sense for the size and shape of potential improvements, and I feel that I hit diminishing returns on understanding these issues after a relatively contained number of conversations. In addition, I felt that the opinions of junior scientists I spoke to early in the process were fairly predictive of what I heard from more prominent scientists later in the process.

We’ve set a goal of coming to a similar level of understanding of systemic issues in fields other than life sciences. Doing so will be a highly informal process, following referrals from contacts I believe understand both science and our values well.

3. Build scientific advisory capacity. We’ve found strong scientific advisors, but their time availability is limited. We’re hoping to find people who can work for us on closer to a full-time basis (ideally full-time). In the short run, such people would help us investigate potential neglected goals. In the long run, they might help us build further capacity after we set our priorities – finding the appropriate hires, constructing the relevant advisory boards, and otherwise finding the best contacts for executing on the science-related objectives we choose.

At this time, we believe our ideal candidate would: (a) have a strong background in life sciences or another scientific area of interest; (b) be available for full-time work; (c) be a generalist, willing and able to put significant effort into networking and recruiting as well as investigation. We are currently informally seeking such people, and may soon develop a job posting and a more formal process.

4. Other projects. We hope to complete a few other cross-cutting projects this year:

  • Investigating the question of “differential technological development”: the question of whether it’s desirable to develop some scientific and technological innovations sooner than others, in light of the fact that many of the most dangerous global catastrophic risks seem to hinge on the develop of new technologies (and may be mitigated by the development of other technologies).
  • Compiling a rough list of major historical breakthroughs in life sciences. We would then investigate the origins of some such breakthroughs, trying to get more basic context on the roles of different kinds of research – and different kinds of funding – in past breakthroughs.
  • Investigating historical cases where a funder took up a “neglected goal” that was getting little attention, and tried to bring it more attention from scientists. This would inform our likely paths forward on top-priority neglected goals.