We recently published an updated review of PSI based on conversations over the past year with PSI staff and additional analysis. We now rate PSI as a “Notable” organization, and no longer list it among our top-rated charities. (Note that PSI is not the only organization whose rating changed in 2010: we similarly “downgraded” both Partners in Health and the Global Fund to “Notable” and the Stop TB Partnership to “Silver” from “Gold.”)

PSI is the organization GiveWell has followed the longest:

• I chose PSI as the recipient of my first substantial charitable donation in December 2006 (the outcome of my GiveWell research when GiveWell was just a part-time project).
• In GiveWell’s first round of full-time research in the fall of 2007, PSI was our top-rated charity in our “Saving Lives in Africa” cause, and GiveWell sent PSI a $25,000 grant. We believed that PSI’s cost-effectiveness was roughly in-line with its own $/DALY estimates.
• In mid-2009, we conducted a more thorough and in-depth analysis of international aid charities reviewing hundreds of organizations (instead of merely reviewing the ones that applied for a grant as we had in 2007). In that process, PSI maintained its “Recommended” status and we ranked it fourth out of the hundreds of charities we had considered. In that report’s review of PSI’s impact, we questioned whether PSI’s programs cause behavior change (such as condom use by individuals who did not previously use condoms).

What changed?

In 2010, we returned to all our previously completed charity reviews to reconsider them and update them with current information. In that process, our conclusion about PSI changed, due specifically, to (a) our view of the way in which PSI uses and evaluates the monitoring data that they collect and (b) new data we found that provided estimates for changes in health behaviors across all of Sub-Saharan Africa.

In our July 2009 review of PSI, we had concluded that

Data from [PSI’s monitoring and evaluation] surveys indicates that PSI’s products are likely being used (more below). It is unclear, however, whether the availability of PSI’s products causes behavioral change (such as condom use by individuals who did not previously use condoms).

While we weren’t sure that PSI’s programs were effectively creating behavior change, we felt that PSI was carefully evaluating its programs in a way that would identify failing programs and lead to improvements. We were ambivalent about PSI’s impact but felt good enough about its monitoring and evaluation process to recommend it. We believe that a charity (a) focused on programs with strong track records (as PSI is by promoting condom and bednet use) that (b) has an unusual degree of transparency while (c) carefully evaluating its programs to determine whether they are working is worth recommending to donors.

Earlier this year, PSI questioned our conclusion that its programs don’t cause behavior change. PSI made its case for impact by stating that “out of 163 available multi-round studies of behavior change, 82 have shown statistically significant impact on at least one behavior change indicator.” We didn’t find this case compelling as we see many ways PSI could measure these results without having significant impact. Specifically:

• PSI’s test for “impact” – a single indicator showing a statistically significant change in behavior – seems unreasonably generous. PSI didn’t offer a compelling justification for the test it used. This led us to question the quality of process PSI uses to monitor and evaluate whether its programs are largely succeeding or failing.
• We also found data providing estimates for bednet use across Sub-Saharan Africa and compared this data to measured changes in PSI’s focus areas to broader changes in bednet and condom use across Sub-Saharan Africa. Our analysis of this data shows that changes of health product use in PSI programs seems in line with changes in non-PSI areas across Sub-Saharan Africa (details in our review of PSI’s impact): PSI’s average change was 3-4% for condoms (compared to 2-3% across Sub-Saharan Africa) and 12-14% for bednets (compared to 23-28% across Sub-Saharan Africa). While we do not believe this demonstrates that PSI is not having impact, this data does shift our assessment of the likelihood PSI’s programs are effective and increases the burden of proof on PSI to demonstrate its programs’ effectiveness.

In part, the downgrade is a reflection of the fact that we are continually digging deeper and raising the bar higher for our recommendations. When we first encountered PSI, it was the only nonprofit we could find that seemed to have any meaningful data on its impact; the limited slice of data we looked at seemed to indicate impact and to be consistent with its own cost-effectiveness estimates; and that was enough for us. By now, we are more careful about demanding data that is broadly representative of the full organization’s activities, and in our fuller analysis of PSI’s data we were less compelled. We also place high value on admitting and responding to shortcomings. If PSI agreed with our view that its data does not convincingly demonstrate impact, we would consider a higher rating based on discussions of its ongoing learning and adjustment.

We still believe that PSI stands out from the vast majority of international aid charities because of its focus on programs with strong track record and its unusual and commendable degree of transparency. (If I had to guess I’d say that PSI is probably in the 99th percentile of all international aid charities in terms of its impact per dollar.)

Nevertheless, our focus at GiveWell is identifying the very best charities and for the reasons above, we no longer feel comfortable placing PSI in that group. We plan to continue to watch PSI carefully and reevaluate its status as new information becomes available.