At GiveWell, we’re committed to understanding the impact of our grantmaking and improving our decisions over time. That’s why we’ve begun conducting “lookbacks”—reviews of past grants, typically two to three years after making them, that assess how well they’ve met our initial expectations and what we can learn from them.

We conduct lookbacks for two main reasons: accountability and learning. By examining both the successes and challenges of past grants and publishing those findings on our website, we aim to be transparent about the impact of donor funding. Systematically reviewing past grants also helps us identify ways to improve our decision-making. When lookbacks identify challenges, lower-than-expected impact, or key questions that we think we should have an answer to, we use these findings to adjust our approach to similar grants in the future or prioritize follow-up research. When lookbacks show higher-than-expected impact, that’s valuable, too—in those cases, we made the error of underestimating impact and might be granting too little to certain programs.

Lookbacks compare what we thought would happen before making a grant to what we think happened after at least some of the grant’s activities have been completed and we’ve conducted follow-up research. While we can’t know everything about a grant’s true impact, we can learn a lot by talking to grantees and external stakeholders, reviewing program data, and updating our research. We then create a new cost-effectiveness analysis with this updated information and compare it to our original estimates. There’s a lot we can’t know with certainty—like how many lives were saved because of a grant—but we believe these lookbacks teach us a lot about the quality of our past decisions and how to improve.

Lookbacks at two different scales

We recently completed our first two lookbacks: one reviewed all 10 of the grants we have made to New Incentives for childhood vaccination incentives in northern Nigeria (totaling $120 million), and the other reviewed a single $7.5 million grant to Helen Keller Intl for vitamin A supplementation expansion in Nigeria. We reviewed all New Incentives grants together since they implement the same program model with consistent implementation and data collection across the northern Nigerian states where they work. For Helen Keller, we focused on just one expansion grant rather than the entire portfolio we support, which would have been too complex for a single lookback. We selected these grants for our initial lookbacks because enough time had passed to observe meaningful outcomes, the findings could inform upcoming grant decisions, and the grants were large or otherwise important to our portfolio.

These initial lookbacks showed different outcomes for the two programs. For New Incentives, our best guess is that our $120 million in grants averted 27,000 deaths from vaccine-preventable diseases—about 60% higher than our initial estimate of 17,000 deaths. Our estimate of cost-effectiveness increased because costs per child dropped significantly as New Incentives expanded from serving 70,000 children to serving 1.5 million children. For Helen Keller Intl, our best guess is that the $7.5 million grant will avert around 450 deaths, lower than our initial projection of 2,000 deaths, because we now think vitamin A deficiency rates in Nigeria are lower and counterfactual coverage rates are higher than we originally estimated.

What did we find?

New Incentives: Conditional cash transfers for vaccination in Nigeria

New Incentives’ conditional cash transfers program aims to increase coverage of routine childhood vaccinations in northern Nigeria by providing small cash incentives to caregivers who bring their children into clinics for vaccinations. Routine vaccination significantly reduces risk of illness and death from vaccine-preventable diseases.

GiveWell has directed over $120 million in grants to support New Incentives’ conditional cash transfer program since it became a Top Charity in 2020. This lookback looks at the full performance of that support by evaluating all 10 grants made to New Incentives from 2020 through 2024.

Here are key takeaways and next steps from our lookback on these grants to New Incentives:

• Cost-effectiveness estimate increased: We now estimate New Incentives is about twice as cost-effective as our initial assessment. When we made our initial grants, we estimated the program was about 11 times as cost-effective as cash transfers.¹To date, GiveWell has used GiveDirectly’s unconditional cash transfers as a benchmark for comparing the cost-effectiveness of funding opportunities, which we describe in multiples of “cash” (more). In 2024, we reevaluated GiveDirectly’s cash program and now estimate it is three to four times more cost-effective than we’d previously estimated (more). For now, we continue to use our existing benchmark while we evaluate what benchmark we want to use going forward. Currently, we estimate it’s approximately 22 times as cost-effective in the three Nigerian states where the program was first implemented, and 9 to 38 times as cost-effective across all the states where it is currently implemented. As a result, we estimate that our funding has or will avert around 27,000 deaths, compared to our initial projection of around 17,000.
• Substantial cost reduction: The increase in cost-effectiveness was driven by a fall in cost for child enrolled—from approximately $40 in 2020 to about $20 in 2024. We think this is due to economies of scale, efficiency efforts by New Incentives, and the devaluation of the Nigerian naira, but we haven’t prioritized a deep assessment of drivers of cost changes. We plan to look into this in 2025.

Helen Keller Intl: Vitamin A supplementation in Nigeria

Helen Keller Intl’s vitamin A supplementation program addresses vitamin A deficiency by funding government-run vitamin A supplementation campaigns that provide supplements to preschool-age children. Vitamin A deficiency can increase the risk of infection and lead to death in young children.

GiveWell has directed over $200 million in grants to support Helen Keller Intl’s vitamin A supplementation program since it became a Top Charity in 2017, and we continue to view it as one of the most outstanding and cost-effective programs we have found. This lookback focuses on a $7.5 million grant we made to support expansion of Helen Keller Intl’s program into five new Nigerian states from 2021 to 2024.

Here are the key takeaways and next steps from our lookback on this grant to Helen Keller Intl:

• Cost-effectiveness estimate decreased: We now estimate the program is significantly less cost-effective than initially projected—about 3 to 11 times as cost-effective as cash transfers, varying by state, down from our initial nationwide estimate of about 24 times as cost-effective as cash transfers. This decrease was driven by new data from Nigeria showing lower prevalence of vitamin A deficiency and higher rate of children receiving vitamin A supplementation without campaigns than we initially thought. With these updates, we think this grant has averted or will avert 450 deaths, compared to our initial prediction of 2,000. We’re highly uncertain about these estimates though, given our high level of uncertainty about vitamin A deficiency rates in Nigeria.
• Challenges with quick exits: Based on state-specific estimates we generated in 2022 that showed cost-effectiveness below GiveWell’s funding threshold, Helen Keller Intl exited two states shortly after beginning work. These quick exits may have strained their relationships with governments, and the lack of a plan for transitioning program responsibilities back to local governments may have contributed to subsequent declines in coverage. In the future, we plan to develop a stronger understanding of grantees’ plans for exits and potential challenges, with the goal of enabling smoother transitions in the future.
• Concerns about coverage data: While coverage surveys showed high levels of vitamin A supplementation after Helen Keller’s entry, we have some concerns with their survey methodology. We have shared these concerns with Helen Keller and have seen initial indicators that they have implemented changes to address them. We plan to revisit this issue in the future.

What can we apply?

These lookbacks revealed three broader takeaways for our grantmaking, beyond these grants. We’ve already started using these findings in our recent grant decisions and are investigating some of the open questions they raised.

• Cost-effectiveness estimates can change dramatically over time. Both lookbacks revealed significant changes in cost-effectiveness, though in opposite directions. New Incentives proved to be more cost-effective than expected, while Helen Keller Intl’s program was less cost-effective. These shifts highlight the inherent uncertainty in our initial estimates and the importance of continued monitoring and reassessment.
• Location-specific analysis improves resource allocation but requires caution. There is substantial geographic variation in cost-effectiveness (e.g., New Incentives’ program ranges from 9x to 38x across different states). While targeting funding to states where we estimate high cost-effectiveness aims to maximize impact, it relies on state-level data (e.g., on burden or costs) that we think are especially noisy. To guard against making decisions based on potentially misleading estimates, we plan to triangulate our state-level cost-effectiveness estimates against perspectives from implementing organizations, ministries of health, and other data sources.
• We need to engage more with local stakeholders. In both lookbacks, we found we’d underinvested in speaking directly with local stakeholders like government officials and subject matter experts in the regions where we fund programs. We think these conversations can help us better understand local conditions, potential implementation challenges, and other considerations that may not be captured in our models. We’ve already begun to systematically increase the number of conversations we have with local stakeholders during each grant investigation, and we have substantially increased our site visits and networking efforts in order to be more fully grounded in the experiences and perspectives of people in the communities our work touches.

What happens next?

These first lookbacks have already informed our work with these grantees and others. As we approach renewal decisions, we are being more cautious about how we respond to fluctuations in cost-effectiveness estimates based on noisy state-level data. The looksbacks have also helped us identify important research questions to prioritize, such as digging into cost reductions over time as programs scale.

Going forward, we plan to conduct lookbacks more regularly across our grant portfolio and publish what we find. Because this was our first iteration of lookbacks, we also expect to improve our approach for doing them over time.

For more detail, including our specific modeling assumptions and calculations, check out the full New Incentives lookback and Helen Keller Intl lookback.