This post is more than 4 years old

Charity evaluation is rarely straightforward. Many factors, within a charity’s control or outside of it, can influence the impact a charity has.

This blog post will highlight a case that illustrates how thinking through these factors can lead to surprising information that changes our understanding of a charity’s impact.

Summary

GiveWell recommended a grant to Results for Development (R4D) in May 2016 for its recently-launched program to increase access to pneumonia treatments for children in Tanzania. We thought this program was promising enough to potentially join our short list of GiveWell top charities once we had more information on its impact.

Expanded access to treatments is a factor in reducing child mortality from pneumonia, but not the only factor. We ultimately want to know not just whether more pneumonia treatments are available in Tanzania, but whether fewer children die of pneumonia as a result of R4D’s work. We expect the program to best achieve this impact if pneumonia patients visit health clinics with treatments in stock and are diagnosed and treated correctly.

We learned as we followed R4D’s work that there was limited information available on the accuracy of clinicians’ pneumonia diagnoses. We initially guessed that clinicians were diagnosing pneumonia accurately around 80 percent of the time. R4D collected data on diagnostic accuracy and we learned that the rate of accurate pneumonia diagnosis was actually 18 percent. This caused our estimate of the program’s impact to fall, though it remains in the range that we look for in potential top charities.

This finding highlights why it’s important to think through all of the factors along the path from a charity’s activities to its ultimate impact; if we had just considered whether more treatments were available, we would have missed this part of the story. We’re excited to continue following R4D’s work because of the role it has played in collecting this information to date and our expectation that it will continue collecting information that allows us to estimate its impact on the availability of pneumonia treatments across Tanzania. We expect to consider R4D as a potential future top charity.

In this post, we discuss:

• The background for GiveWell’s grant to R4D (More)
• Our plans for assessing the impact of R4D’s program (More)
• Approaches to measuring R4D’s impact (More)
• Lessons from this work (More)

Grant background

Pneumonia is a leading cause of children’s death worldwide.[1] R4D approached us in 2015 and told us that Tanzania did not have sufficient funding to maintain an adequate supply of pneumonia treatments in the country’s public sector health system.[2] R4D was interested in providing market-shaping technical assistance and catalytic, time-limited funding for pneumonia drug supplies, with the goal of improving the availability of drugs in order to avert more deaths.[3]

We recommended a GiveWell Incubation Grant in May 2016 of $6.4 million to support the first phase of R4D’s scale-up of pneumonia treatments in Tanzania. We thought R4D might meet our top charity criteria once we had more information with which to assess its impact.[4]

How will we know if R4D is reducing deaths from pneumonia?

Funding the purchase of additional pneumonia treatments would seem a simple solution to the inadequate supply of the drugs. But to truly assess the impact of the program on reducing child mortality from pneumonia, we wanted to understand:[5]

1. Would R4D increase the availability of pneumonia treatments?
2. Would clinicians diagnose pneumonia accurately? (We initially estimated 80 percent accuracy in diagnoses in the public and private sectors.[6])
3. Would clinicians prescribe pneumonia treatments to people who needed them?

The second and third questions relate to factors outside of the scope of R4D’s program, which aimed to increase the availability of treatments. However, they play an important role in R4D’s success in reducing deaths from pneumonia.

We were surprised by how difficult it was to answer the second and third questions. There did not appear to be existing data from Tanzania on pneumonia diagnosis and treatment and it was challenging to design effective ways to measure them.

Gathering information

A common story we hear is that many charities do not conduct surveys to verify whether they’re reaching program participants and having the hoped-for impact because:

• donors don’t want to pay for monitoring; or
• charities don’t want to implement monitoring: it’s time-consuming, expensive, and not clearly in demand from donors.

Neither was true in this case. We were interested in funding measurement of the rates of accurate diagnosis and treatment. R4D was interested not only in implementing the measurement, but in taking the lead on developing creative ways to tackle questions about the program’s impact. The latter is rare in our experience. When we have asked charities how they monitor their work, we have often been told that the charity simply knows its program works.

Initial plans

R4D initially planned to use health clinic records to see whether pneumonia treatments were increasing due to its program and whether those treatments were correctly prescribed.[7] However, R4D found in an initial investigation that these records were incomplete and thus did not indicate whether the intended impact was occurring.[8]

R4D considered and decided against a number of other means of assessing whether children who had pneumonia received treatment, such as video-recording clinicians (which was rejected due to anticipated challenges in obtaining consent for patients), surveying patients outside of health clinics (which was rejected due to its cost and anticipated challenges with patient recall), and conducting a high-quality study focused on child mortality (which was rejected due to the high cost of running a sufficiently large study).[9]

Eventual solution

R4D next partnered with IDinsight, another GiveWell Incubation Grant recipient, to develop a new approach to gathering this information.[10] Working with IDinsight, the government of Tanzania, and the Tanzanian national medical school, R4D used lung ultrasounds, which directly tested whether patients with respiratory symptoms had pneumonia, to measure the accuracy rate for clinicians’ pneumonia diagnoses—a neat solution.[11]

The lung ultrasound information yielded surprising results. The rates of accurate pneumonia diagnosis were quite low. Only 18 percent of children with pneumonia confirmed by lung ultrasound were correctly diagnosed.[12]

Getting the full picture

Even that, however, didn’t tell the full story. If we had just looked at diagnostic rates and assumed that incorrect diagnosis leads to incorrect prescription of treatment, then we would have missed another important element of the story: many children who were not diagnosed with pneumonia were still prescribed the right drug to treat pneumonia. When they had the pneumonia treatment in stock, clinicians prescribed it in 46 percent of cases in which they had incorrectly diagnosed a child as having something other than pneumonia. We are unsure why.[13]

Our estimate of the cost-effectiveness of R4D’s pneumonia program fell by 27 percent when we updated it to reflect this new information.[14]

A broader question

The importance of looking for factors that influence impact across a charity’s causal chain, whether under the charity’s control or not, is not unique to pneumonia, nor Tanzania, nor R4D. For example, when we try to understand whether GiveWell top charity Against Malaria Foundation‘s work to prevent malaria by supplying insecticide-treated nets results in fewer people dying of malaria, we think through all the parts of the process that could fail. We aim to do this for our other top charities, as well.

Our estimate of R4D’s pneumonia program’s cost-effectiveness remains in the range that we look for in potential top charities and we’re excited to continue following its work.[15] But without the new information on diagnostic accuracy, we, R4D, and the government of Tanzania might have gotten an incorrect picture of its impact.

We made another grant to R4D in January 2019 to support the second phase of the pneumonia treatment program. We forecast a 40 percent chance that R4D (as a whole) or one of its specific programs (like pneumonia treatment) is a top charity by December 2023.[16] As we move forward, we plan to continue to ask ourselves all of the ways this grant might have more or less impact, as we did before, and as we do in all cases.

Sources

Sources for this post may be found here.