This is a cross-post from Good Ventures’ blog Give & Learn. It was co-authored by Cari Tuna, Good Ventures Co-Founder, and Natalie Crispin, GiveWell Research Analyst.

We recently traveled to Myanmar to visit a project Good Ventures is supporting to help prevent the spread of drug-resistant malaria. This post shares some observations from the trip and general thoughts about the value of site visits versus other ways of learning about the impact of one’s giving.

About the project

To recap, this project aims to rapidly replace one type of malaria treatment in Myanmar (AMT) with another (ACT) to reduce the risk of drug resistance, which could have a devastating effect on global malaria control efforts if left unchecked.

The project is being carried out by Population Services International (PSI) with support from the Gates Foundation, the UK Department for International Development (DFID) and Good Ventures. We contributed to the project as part of our effort to learn from other major funders through co-funding.

The project involves selling subsidized ACTs to the largest pharmaceutical distributor in Myanmar, sending “product promoters” to private drug providers to promote the appropriate use of ACTs, and piloting a project to promote the use of rapid diagnostic tests (RDTs) among such providers. It also involves encouraging the Myanmar government to follow through on its commitment to end the importation of AMTs.

The project got off to a slow start in 2012, but progress has accelerated over the course of 2013. The latest data show that the ratio of AMT to ACT in the market has decreased from about 20:1 in 2012 to about 1:2 in 2013. The subsidy has been largely passed on to patients: the price of a full course of ACT is less than or equal to the price of a typical (partial) dose of AMT in 94% of drug outlets. Due to quickly declining malaria rates in the country, PSI Myanmar estimates that the project has enough funding to continue into 2016, 18 months longer than originally planned. (A more detailed update is forthcoming.)

Observations on the site visit

We spent five days with PSI Myanmar for the annual donor review of the project, which was also attended by representatives of the Gates Foundation and DFID. We spent the first day in PSI Myanmar’s office in Yangon reviewing the project’s progress and potential risks to its continued success and sustainability. We spent the next three days traveling through Mon State and interviewing people working at various levels of the supply chain, including itinerant drug vendors, village pharmacists, drug wholesalers, and representatives of the large pharmaceutical distributor, AA Medical Products. (We’ll refer to these three days as the “field visit.”) We spent the last day in Yangon reflecting on the visit with PSI and DFID. We’re deeply grateful to PSI Myanmar for hosting us and organizing the trip.

We’ll post detailed notes from our travels soon. In the meantime, we wanted to share some miscellaneous observations and reflections:

• A major goal of ours in co-funding this project — and attending the annual review — was to learn about how funders such as the Gates Foundation and DFID operate.
  • These funders took a similar approach to the site visit as we would have taken on our own. They asked everyone we interviewed copious questions, both directly and indirectly related to the project. They tended to avoid leading questions and tried to ask the same questions in multiple ways, in order to increase their odds of getting an unbiased view of the situation.
  • We were impressed by Louise Mellor from DFID and her efforts to establish a positive rapport with the people we interviewed. She encouraged everyone in our entourage to introduce themselves by name at each stop. After we had asked our questions, she often offered the interviewees words of encouragement or thanks for the role they were playing to help prevent drug resistance. (Previously, Louise told me that establishing a positive rapport is important for helping people feel comfortable speaking freely.) We plan to be more intentional about establishing a positive rapport with interviewees on future site visits.
  • We were also impressed by the Gates Foundation’s Tom Kanyok, who volunteered to take a rapid diagnostic test (RDT) at a general store we visited in Mon State. The test involved the general store owner pricking his finger for a blood sample. It was helpful to see an RDT performed live.
  • DFID’s process for evaluating the project’s progress involved both a formal review of pre-determined, quantified indicators and unstructured reflection on what could be improved. At the end of our visit, DFID staff provided PSI with both overall feedback on how they believed the project was going (“a very positive review”) and some recommendations for PSI to consider over the next few months, such as studying the feasibility of reducing the ACT subsidy, exploring the use of local languages on drug packages, and providing more information to donors on issues raised during the review. DFID will also produce a formal report on the review (update: recently published here). In addition to DFID staff who are based in Myanmar, a DFID economic advisor, who is based in London and not normally involved in the project, attended the annual review, in order to provide an outside perspective.
  • DFID staff noted that while field observations cannot be treated as representative evidence, site visits help to provide a reality check on one’s assumptions, surface potential problems, and allow for relationship building and evaluation of project management. They said DFID is trying to incorporate more site visits into its project monitoring.
  • We were struck by how often Tom Kanyok from the Gates Foundation raised the topic of malaria elimination and eradication. (Representatives of the Gates Foundation previously told us that malaria eradication is a focus of the foundation. PSI told us that governments in the region, with support from the World Health Organization, have adopted the strategy of eliminating of P. falciparum malaria, and believe it’s the only long-term solution for antimalarial drug resistance.)
• ACTs were available in all but one of the 10 drug outlets we visited. The shopkeeper at the outlet that was out of stock out told us that she had sold the last pack the previous day. At each outlet, we asked the provider whether he or she sold other antimalarial drugs, and we looked for such drugs on display. No provider said they carried oral AMTs. At one wholesale outlet, the shop owner told us that he did not sell oral AMTs, but we subsequently found them on display. The shop owner then became worried that he was in trouble. (It’s not against the law to sell oral AMTs, only to import them, but there seemed to be rumors that selling them was banned as well. What’s more, we were traveling with a representative of the Myanmar Ministry of Health, which could have increased the shop owner’s concern.)
• From what we saw, the pilot program to encourage use of rapid diagnostic tests (RDTs) appeared to be going well. All of the pilot participants with whom we spoke reacted positively to questions about the program, and the shopkeeper who performed a demonstration test for us did so competently. She had completed only a few tests before our visit.
• One issue that emerged as a concern to donors during the field visit — but wasn’t emphasized as a major risk to the project’s success during PSI’s earlier presentation — was the upcoming expiration of ACTs in the market. At the start of the project, PSI had to place its first order of ACTs with limited information about the size of the market. As a result, and because of declining malaria rates in Myanmar, PSI ordered more ACTs than have been needed. It had stock that would expire, unused, in March and April 2014. At the time of the site visit, PSI was waiting to begin selling a new batch of drugs, and replacing expired drugs in the market with new drugs, to minimize wastage. The issue surfaced during the field visit when a DFID participant noticed that all the ACTs we encountered in the market were set to expire in March or April. Donors raised the concern that PSI may be waiting too long to replace the drugs and that expired drugs may trickle down to harder-to-reach areas as a result. This prompted a conversation that led PSI to begin the replacement process slightly earlier than planned, after obtaining approval from the donors. PSI notes that it’s not alone in having expired drug stock — all agencies in Myanmar, including the Ministry of Health, are currently seeing rapidly decreasing disease transmission resulting from aggressive control efforts, which has led them to need fewer drugs than originally anticipated. PSI also notes that overstock is greatly preferable to understock, which could lead to market demand for sub-standard products.
• Because we were traveling with a large entourage (around 10 people) and making mostly pre-scheduled stops, it was difficult to know whether we were seeing an unbiased picture of circumstances on the ground. We do note that the purpose of the field visit was not to assess the project’s overall success, which is more appropriately done by looking at representative monitoring data rather than a small sample of anecdotal observations.
  • For instance, we were struck by the abundance of promotional materials for the PSI-subsidized ACTs on display at the outlets we visited, including at outlets where providers reported relatively low malaria caseloads. Many of the materials looked relatively new. A PSI representative assured us that extra materials were not hung up in anticipation of our visit. This may well have been the case; PSI staff said they had recently undertaken a big marketing push.
• We were struck by the complexity of the operating environment in which the project is taking place. This served as a general reminder of the large number of ways in which a project can fall short of its objectives, including ways that would be difficult for a donor who is not highly informed to predict. In this case, complexities of the operating environment include violent conflict in some parts of the country where drug resistance is of particular concern, conflicting policies in different government departments, dissatisfaction with the goals of the project among some government physicians, a second common malaria parasite that is treated with a different drug regimen, and working conditions and vector behavior that make certain interventions, such as bednets, somewhat less effective for at-risk populations.

General thoughts on the value of site visits

As we research potential focus areas, we’re often advised to “get out into the field” in order to understand the work we’re funding, or considering funding, better. We agree with the notion that site visits can be valuable for learning. That said, we’ve found that such visits are helpful for some — but not all — types of learning. They are not a substitute for desk research, though they can complement such research in important ways. Site visits also take a great deal of time to conduct, and hosting them requires a significant investment on the part of the nonprofit. These are trade-offs we take into account when deciding how to prioritize our time to maximize our learning.

What are site visits good for?

• Field observations can provide a valuable reality check on our assumptions.
• They can be helpful for raising questions and surfacing potential problems that may not have appeared important based on reviewing formal reports and monitoring data. In this case, for example, the issue of expiring drugs became a greater concern for donors because we encountered them in the market.
• Field visits often involve spending an extended amount of time with project managers and staff — and in this case, other donors — including unstructured time on the road and eating meals, in addition to scheduled stops. This time is not only helpful for relationship building; it also allows us to ask many more questions than we could over the phone or by e-mail. We’ve consistently found this to be one of the biggest benefits to participating in such visits.
• Field visits help us to develop a fuller picture of the context in which a nonprofit is working, including complexities that may be hard to appreciate from afar.
• We’ve found that communicating about a project is easier after spending a lot of time immersed in the details of the work.
• Lastly, visiting with beneficiaries of a project, as we did in our 2012 visit to GiveDirectly in Western Kenya, can lead to a greater emotional connection to the work.

What aren’t site visits good for?

In most cases, site visits do not seem to be an appropriate tool for learning how a project is going in general, because they only allow for a small number of anecdotal observations, which are not necessarily representative of the situation overall. Furthermore, despite our best efforts not to ask leading questions or prime interviewees to respond in certain ways, we’ve found it hard to know whether we’re getting a fully accurate view of circumstances on the ground in the places we’ve visited. This is to be expected and doesn’t diminish the other benefits of conducting site visits. But it does point to the importance of representative monitoring data and rigorous, independent evaluation in learning about whether the work we’re funding is succeeding in meeting its goals.