The GiveWell Blog

Elie’s general notes from living in India

(updated on: ) | by Elie

A previous post shared Holden’s general thoughts from last year’s several-month staff trip to India. This post shares my thoughts; the next will share Natalie’s.


Site visits are short and it’s impossible get a full picture of an organization, its work, and its likely impact. But, little things about how programs look on the ground and how they compare to what the marketing materials say can be telling.

  • “We give shoes to poor children.” I visited a soccer practice for kids. It had rained the day before I went, so the field was soaked. Because the field was wet, the kids were allowed to play barefoot, the coach told me. The organization provides the kids with shoes, which the kids must wear. The coach said that the kids were happy because they prefer to play barefoot, but the organization only lets them do so when it’s wet out. Later that day, I went back to the organization’s office and spoke to the development director. As part of his presentation, he noted that the organization provides children with shoes, and the kids love coming to the field and putting on their shoes.
  • “We teach English.” English appears to be an extremely valuable skill, allowing speakers to obtain relatively high paying jobs such as working in a call center. One organization we visited taught English classes to teenagers, which seems like a very promising program to run in Mumbai. But, after visiting the classroom, we wondered how effective the class actually was. In the class we visited, the teacher stood at the front of the class writing English sentences on the board and the students copied down the sentences into their notebooks. When we were there, we saw approximately 5 sentences on the board, one of which was “We shall make your tongue right again.”
  • “We teach computer classes.” Several organizations we visited offered computer classes for kids. The classes were held in rooms with a few computers. A couple of times, I clicked on the Start menu of computers not currently in use to take a look at recently accessed documents. I wanted to know what types of work the kids did. When I loaded Microsoft Word or Microsoft Excel – two of the programs the organization said they used in their classes – it was clear that this was the first time these programs had been loaded. They asked me to register my version of Excel and enter a username.
  • “We educate children on health behaviors.” We visited an organization offering health classes for children. The topic of the session I attended was “healthy eating” and the teachers talked about not eating too much junk food which is high in calories. While it makes sense that a child should eat fruits and vegetables and not too much candy and fried food, it didn’t seem like the most pressing message for the children I saw. They were all frail, looked a lot younger than they were, and may have been malnourished. I felt like I was attending a class that would fit better at school in the United States than a classroom in a Mumbai slum.

In Mumbai, we also had the chance to visit several organizations we had already reviewed on our website and get an on-the-ground perspective. I visited Mumbai’s TB Control Program, guided by a WHO consultant stationed in Mumbai. This wasn’t a visit, specifically, to the Stop TB Partnership, but because Stop TB supports government-run TB programs, this visit was to the type of program Stop TB supports. Each person I met with from the WHO consultant to the director of Mumbai’s TB Control Program to directors of hospitals and clinics were (a) aware of monitoring procedures, (b) answered all my questions about how they monitor, and (c) often showed me examples of documents and data as support.

For example, a major focus of TB control programs is making sure that patients maintain the 6-month drug regimen, taking all prescribed doses and don’t stop taking them early (which often happens after a couple months as symptoms begin to subside). When I visited a local TB treatment center, I asked them how they monitor patients to ensure that they take all doses. The treatment center director told his staff to go into the back room and bring back a patient’s box of drugs along with that patient’s paper records. My guide showed me how he would check this particular patient’s records. The treatment center keeps each patient’s box of pills, the patient’s treatment schedule, and all used pill wrappers. At any point, an auditor can come and check the box and the paperwork to make sure everything is in order.

Seeing the monitoring and data-collection operation up close – and the ease with which the people I met with discussed the data and issues associated with collecting it – gave me more confidence in the monitoring data we rely in our report.

Holden’s general notes from living in India

(updated on: ) | by Holden

Last year, the staff of GiveWell moved to Mumbai for several months. We did this largely for personal reasons (our staff is small enough that we were able to coordinate it, and we all thought it would be interesting) but also took advantage of our time there to visit 20 nonprofits (some of them more than once). This post contains my general impressions; future posts will contain impressions from other GiveWell staff members as well as our thoughts on specific nonprofits. Our full set of notes and pictures from site visits to local charities in India is also available.

Limits to learning
The first time we saw the metro, we observed the people hanging out of the doorways and concluded that it was amazingly crowded. The second time we saw it, we noticed that the insides of the cars were actually empty at that time, but people were still hanging out of the doorways. By the time I was actually riding the metro, I realized that hanging out of the doorway was a privilege, not a necessity (and I did it whenever I could).

A lot of the observations I hear people make about their travels – and perhaps some of the observations I’ll make below – seem to me to be along the lines of “The subway is so crowded that people have to hang out of the doorways.” It’s very easy to look around and feel that you’ve seen something significant and meaningful, but very hard to get the full context needed to really understand what it is you’re looking at. Although we were in India for several months, there was never any real possibility of become fully integrated into society or blending in with the crowd. Our ethnicity alone marked us as clear outsiders, targets for panhandling to some and photo opportunities for others.

So take the rest of my observations – and, I’d argue, the observations you hear from anyone’s travels in foreign countries – with a grain of salt.

Absolute poverty

Across the board – with the exception of the very upper end of the income range (which included us) – poverty levels and living conditions seemed substantially and visibly worse than what one sees in the U.S. We saw people living in slums – and on the street – with much less space and much worse sanitation than any regular living conditions I’m aware of (including public housing) in the U.S.

I was also disturbed by the state of many animals I saw, particularly dogs and cattle – they consistently looked more emaciated and sickly than anything I’m used to seeing in the U.S.

We’ve argued, mostly based on data, that what counts as “poor” in the U.S. would be seen as “rich” in much of the rest of the world. Spending time in India reinforced this idea in a tangible way.

Relative poverty
I found that looks were often deceiving when it came to guessing which people were lowest-income and/or most appropriate as recipients of aid.

  • As discussed previously, there are certainly cases of relatively wealthy people living in the slums. I ran across anecdotes of people who owned motorcycles, people who had college degrees and promising careers, and even one woman who owned a flat outside the slum but continued to spend most of her time in it. My point isn’t that any of these people are wealthy in U.S. terms; it’s that I wouldn’t expect a donation from an American to go very far toward helping people who have substantial assets, education, etc.
  • In conversations, we got the impression that panhandling is a particular money-making strategy and is not necessarily the exclusive domain of the poorest. In fact, there may often be property rights around certain prime spots for panhandling, which have to be rented. We tended to encounter more panhandlers in areas that were wealthier and had higher concentrations of foreigners – on the occasions when we went into more remote, poorer areas (including a couple of accidental trips through slums), we were rarely if ever asked for money. I think there is a strong case to be made that the people asking for money are not the ones who would most benefit from it.
  • It seemed to be conventional wisdom among the people we talked to, particularly at NGOs, that Mumbai was a very wealthy city and that no one there was really poor compared to people from rural Bihar, Andhra Pradesh, etc. Here our instincts diverged heavily from those of locals: we thought many of those we saw in Mumbai seemed like appropriate aid recipients, but multi-city NGOs we talked to expressed strong preferences for working in poorer areas. One stated that it was planning on expanding to Mumbai primarily for fundraising purposes (since it would then have better access to donors living there).

The economy
One major difference between my time in India (especially Mumbai) and my time in Africa was the sense I got of the local economy. In the conversations I had during my trip to Africa, many spoke of the difficulty of getting jobs and seemed to see government and NGO jobs as most desirable. Most of activity I saw involved people providing redundant services (for example, when we drove through a village our car was surrounded by children trying to sell us water – I saw similar redundancy among people selling SIM card recharges and souvenirs). By contrast, in India many of the people I spoke to were adamant that (a) just about anyone can get a job in Mumbai; (b) anyone with valuable skills (such as English speaking) can get a good job in Mumbai.

Just by walking around Mumbai, we saw people making a living in a huge variety of ways. Our tour of the Dharavi slum was certainly consistent with Reality Tours’s statement that

Dharavi’s industries have an annual turnover of approximately US$ 665 million. Through our tour visitors experience a wide range of these activities: recycling, pottery-making, embroidery, bakery, soap factory, leather tanning, poppadom-making and many more. Most of these things are created in innovative ways and in very small spaces!

One of the highlights of the trip for me was visiting Ebizon Netinfo, a website development team that had been working with us remotely for over a year, outside of Delhi. We enjoyed their enthusiasm and optimism; like many we spoke to in India, they believe that with hard work they are headed for serious financial advancement.

The overall picture I got was of an economy that’s extremely poor compared to the U.S. but has a lot more prospects for growth and optimism than what I saw on my trip to Africa. This would make it a very promising place for the kind of giving that I, as opposed to Natalie, prefer.

Site visits
A donor of ours earmarked $10,000 for regranting to a local charity in India, and in deciding how to give this away (and for general learning) we conducted 20+ site visits to small NGOs during our travels. In a sense, this was a chance for us to try out a more traditional method of giving: heavily based on referrals, site visits, and informal impressions rather than desk research.

Future posts will discuss much of what we saw on these site visits and what we came away with. Here I just want to note that

  • Small NGOs have the advantage that a single site visit can give one a look at a sizable portion of what they’re doing, and that they are (at least potentially) in close and constant contact with their clients. But they also raise problems of their own for a donor.
    • The room for more funding question can be particularly difficult with small organizations not built to scale.
    • Many small organizations simply seemed to have issues with professionalism and competence, as well as with listening to clients and putting their needs first. (Specific examples are forthcoming.)
  • A site visit can give a general impression of what staff are like, how they think, and how they interact with clients. However, many NGOs’ goals are very long term (for example, improving education), and in most cases a site visit isn’t nearly enough to form confidence in an NGO’s effect on clients’ lives.
  • As we usually do, we sought NGOs we could be confident in with the information we had. We did run across a couple that I feel good about supporting, despite the fact that our existing criteria are not a fit for them; thinking about these cases has been useful in re-examining our criteria.

Internal debate on the goals of giving to charity

| by Natalie Crispin

A donor wrote to us recently with a question (paraphrased):

Is it better to help people who cannot currently live a decent life or to help those who might be better off to start with but have more potential to live enjoyable and meaningful lives?

Ideally the answer to this question would be “let’s help everyone get to the point where they can live enjoyable and meaningful lives.” Realistically, though, resources available for charity are not unlimited and there are trade-offs involved in choosing where to give.

Recently, we’ve been internally debating the question above, which said another way is: “If you had similarly strong opportunities to help different groups, who do you help to do the most good?” and we have some differences of opinion. For example:

  • Holden believes that the ultimate goal, the highest moral imperative, is to achieve and help others achieve their full potential, i.e. help them do something great with their lives.
  • I believe that the ultimate goal is to reduce suffering and create happiness.

Holden and I agree, though, that these goals often lead to similar intermediate goals:

  1. Fulfill people’s basic needs
  2. Create environments in which there are plenty of good opportunities and people can choose for themselves what is best.

Where we tend to differ is the populations we are most concerned with helping. Given the choice, Holden would prefer to help those with high potential to do great things with their lives (one frequent example that comes up is young people in fast-growing, democratic India), whereas I most want to help those in abject poverty who suffer from hunger, disease, and uncertainty (this tends to be people in isolated, rural areas in developing countries).

That being said, Holden and I agree that within our current known charitable options, we choose to give to VillageReach. VillageReach’s case for impact and ability to use additional funds to have additional impact is significantly more compelling to us than that of any other organization we’ve found. If Holden had a similarly strong option to help, say, Indian youth, he would likely opt for that, but as it stands, VillageReach continues to be the best option we know of for helping any population.

We will continue to search for strong options for donors who most want to help different populations. In the meantime, we’d love to hear your thoughts on this debate in the comments section below.

More Than Good Intentions by Dean Karlan and Jacob Appel: Our review

(updated on: ) | by Holden

A few disclosures regarding this book:

  • Innovations for Poverty Action is a client of Sona Partners, and Tim Ogden of Sona Partners sits on our Board.
  • We are currently considering Innovations for Poverty Action for a recommendation.
  • We sent the authors of the book an early draft of this review and have made modifications after some back-and-forth.

I recently read More than Good Intentions, a book by Dean Karlan and Jacob Appel of Innovations for Poverty Action (which we’ve written positive things about before).

  • I thought the book was excellent, and I highly recommend it to anyone interested in effective giving and in development economics. I especially recommend it to anyone interested in microfinance – it has the best extended discussion of microfinance that I’ve seen.
  • I am a bit concerned that the book doesn’t adequately capture the distinction between “exciting ideas with some preliminary evidence behind them” and “the most proven interventions in aid.” I feel that programs falling in the latter category are largely not discussed in the book, and that this fact may be unclear to donors unfamiliar with the area.
  • The closing chapter of the book points readers to the Proven Impact Fund, which we understand is still a work in progress. After some discussion with Innovations for Poverty Action, we have determined that the Proven Impact Fund is unlikely to receive a GiveWell recommendation until and unless it becomes more established and has a track record. (We will be publishing the details of our views on the Proven Impact Fund at another time.) We are, however, evaluating Innovations for Poverty Action itself as a vehicle for donors to contribute to more high-quality research on aid.

Excellent discussion of microfinance and other topics

The book opens with a multi-chapter discussion of microfinance. Dean Karlan has been a leader in high-quality studies of microfinance, many of which have debunked the myths surrounding it, and I find his discussion of it to be the best I know of: engaging, educational, and sensible in its interpretation of the information we have.

The discussion centers around the idea – often lost in the stylized stories around microfinance – that different people have different needs, and that loans can be helpful to some and harmful to others. Karlan and Zinman’s study on consumer loans in the Philippines makes a strong case that when the timing is right, a loan can be very helpful even at a high interest rate. But the book also discusses people like Erlyn, whose business had a limited need for capital, such that the large long-term loans associated with her microfinance institution largely served to put her deeper into debt. The work of Karlan and others has shown how harmful many of the restrictions and rules of traditional microfinance – often based on romanticized notions of poor entrepreneurs – can be.

After discussing microfinance, More than Good Intentions goes through research on how to help the poor in other ways, from improving access to health services to raising attendance in schools. I was familiar with most of these studies prior to reading the book, but the book adds helpful context on the background and motivation for the various research projects.

One of the consistent themes of the research is the tradeoff between (a) giving clients the freedom and power to make their own choices; (b) finding “nudges” that can help people act more strongly in their self-interests. Much of the microlending research suggests that more freedom and discretion is better, but in other areas, clients seem to benefit from some restrictions – and in some cases the restrictions are even self-imposed (as with the SEED program, in which clients open savings accounts that can’t be withdrawn from until they save a certain amount).

It does seem to me that the set of investigations discussed in the book is missing an important piece. The studies cited generally compare one version of a charity’s program to another, or they compare it to “no intervention”, rather than compare a program to simply handing out cash equivalent to the program’s costs. To us this seems the most appropriate way to ethically and reasonably evaluate the value of programs: put them up against the simplest, most direct, and most empowering/discretionary of all interventions, a cash gift.

We understand that there have been studies in this field that do investigate unconditional cash transfers vs. other interventions; we also understand that funders are not always willing to finance a comparison to cash transfers as opposed to “no treatment.” Still, the fact that microfinance programs, free school uniform programs, etc. have always been compared to “no treatment” rather than “cash” seems suboptimal to us.

I also want to highlight the following anecdote from the beginning of the book, as a reminder of what low-quality evaluation (even at the largest and best-known organizations) can look like:

The first “impact evaluation” I ever saw of microcredit made my stomach ache. It was clearly intended to generate pretty numbers for a brochure to donors – not to determine whether something was really working. It asked clients something like, ‘You are eating better now, compared to before you joined FINCA, yes?’ … What I have since learned is that FINCA was doing just as much to measure its impact as anyone else. Which is to say, very little.

“Proven” vs. “exciting”

The book overall covers a lot of ground but doesn’t cover all the ground we would have liked it to, given its ambitions to encourage people to give to proven, effective interventions. This is best illustrated by the final section of the book, which aims to “leave you with seven ideas that have me [authors] particularly excited.”

The seven ideas are microsavings, reminders to save, prepaid fertilizer sales, deworming, remedial education in small groups, chlorine dispensers for clean water, and commitment devices. (The list of highlighted ideas has since been updated, as noted below.) I feel very strongly that this list (and its update) do not include the programs with the best track records in international aid. In my view, several proven health interventions – most notably immunization campaigns, distribution of insecticide-treated nets, and tuberculosis case finding and treatment – all have far better and more robust track records than any of the seven. The reasons I feel this way have to do with the distinction between micro evidence and macro evidence: the ideas in More than Good Intentions are all primarily supported by micro evidence (programs that took place on a small scale and were monitored and evaluated by academics, with the ultimate proof of effectiveness consisting in numerical differences between treatment and control groups) whereas the ideas I have listed are supported by both (not just randomized controlled trials, but also country-level success stories where a scaled-up program achieved an impact that is easy to see outside the confines of a study). I may elaborate on why this distinction is important to me in a future post.

I’m not sure that the authors of More than Good Intentions would disagree. So why does the book focus on relatively new ideas and give less attention to programs such as traditional immunization campaigns? I think there is a perfectly reasonable explanation:the authors are particularly excited about interventions that they and their colleagues have helped prove the promise of, as opposed to interventions like immunization campaigns that have been recognized as effective for decades.

There’s nothing wrong with being particularly excited about insights that are particularly new, counterintuitive, or simply a source of personal pride. However, donors need to recognize the difference between “exciting new ideas worth learning more about” and “the programs with the best track records,” and I am not sure that they will come away from More than Good Intentions with this distinction clear to them.

The closing chapter of the book points readers to the Proven Impact Fund, which we understand is still a work in progress (and unlike the list of ideas in the book, it includes interventions related to bednets and immunizations – though not the same interventions that I feel are most strongly associated with macro-level successes). After some discussion with Innovations for Poverty Action, we have determined that the Proven Impact Fund is unlikely to receive a GiveWell recommendation until and unless it becomes more established and has a track record. (We will be publishing the details of our views on the Proven Impact Fund at another time.) We are, however, evaluating Innovations for Poverty Action itself as a vehicle for donors to contribute to more high-quality research on aid.

In the meantime, we highly recommend More than Good Intentions as one of the most sophisticated discussions we’ve seen of microfinance and other fronts in the fight against poverty.

Three Cups of Tea scandal: Why we had the right bottom line on the Central Asia Institute

(updated on: ) | by Holden

There has been a lot of coverage of the scandals around the Central Asia Institute. The founder has been accused of fabricating inspiring stories, as well as of spending less than half of the millions of dollars he’s raised on building schools.

The Central Asia Institute receives four stars from Charity Navigator (archived) and had perfect ratings from Great Nonprofits prior to the scandal (archived), but GiveWell has declined to give this organization a recommendation or distinction (note that that page was published in mid-2009).

In a sense this doesn’t indicate impressive foresight on our part: nearly all charities we examine do not receive recommendations or distinctions, so it’s not as though we spotted the fabrications and financial mismanagement ourselves. Yet in the bigger picture, I see this incident as a vindication of our approach to giving: it’s a reminder that you shouldn’t give charities the benefit of the doubt.

Central Asia Institute is an education charity. Our questions for education charities are:

  • Is the goal to improve attendance or to improve school quality?
    • If the goal is to improve school attendance, what sorts of schools will beneficiaries be attending? What information is available on teacher attendance and quality of instruction? What evidence is available regarding the program’s effect on attendance?
    • If the goal is to improve school quality, is there evidence that similar activities have causally led to improved schools (in terms of attendance, test scores, graduation rates) in the past? Are indicators of school performance tracked over time, both before and after the interventions?
  • What evidence is available regarding the likely impact of any improved performance/attendance on later life outcomes? (For example – what are the economic opportunities that will be available to students and how do education levels relate to them?)
  • How would activities change if more revenue than expected was received? Would more revenue translate into more students served, and up to what point?

 

When we looked at Central Asia Institute’s website, we saw lots of stories but not a word to answer any of these questions substantively. So, as we usually do, we declined to give it a recommendation or distinction. As it turns out, the Central Asia Institute couldn’t have answered these questions well, because (a) it does appear to focus on school building rather than school quality, thus falling prey to one of our main concerns about education charities; (b) it apparently does not have strong monitoring and evaluation of its own schools. From the CBS story:

The IRS tax return Central Asia Institute filed last year included a list of 141 schools that it claimed to have built or supported in Pakistan and Afghanistan. Over the past six months, we visited or looked into nearly 30 of them. Some were performing well, but roughly half were empty, built by somebody else, or not receiving support at all. Some were being used to store spinach, or hay for livestock; others had not received any money from Mortenson’s charity in years.

If the Central Asia Institute had had anything on its website to indicate promise (such as an example study from a single school showing high-quality education), we would have opened a more in-depth investigation, including in-depth examination of the “What do they do?” and “Is there room for more funding?” questions, which would certainly have led us to the observation that the Central Asia Institute spends less than half its money on actually building and maintaining schools. (This point, raised by the CBS story, is easy to see from page 2 of the organization’s most recent tax return (PDF).) The fact that it still technically has a low “administrative expense ratio” is just a reminder of how easily manipulated, and ultimately meaningless, this metric is.

People frequently object to the fact that we recommend so few charities, and to our equating of “not enough evidence to evaluate” with “not recommended.” But the fact is that if a charity isn’t disclosing substantive information to answer the tough questions about its activities, you have no way of knowing whether it has the same kind of problems that the Central Asia Institute has. Certainly you can’t know from looking at its rating from other agencies.

We recommend very few charities, but we are able to stand behind those we do recommend. If you’re looking to do as much good as possible with your donation, consider choosing from a smaller menu to ensure that you get quality.

PSI rating change

(updated on: ) | by Elie

We recently published an updated review of PSI based on conversations over the past year with PSI staff and additional analysis. We now rate PSI as a “Notable” organization, and no longer list it among our top-rated charities. (Note that PSI is not the only organization whose rating changed in 2010: we similarly “downgraded” both Partners in Health and the Global Fund to “Notable” and the Stop TB Partnership to “Silver” from “Gold.”)

PSI is the organization GiveWell has followed the longest:

  • I chose PSI as the recipient of my first substantial charitable donation in December 2006 (the outcome of my GiveWell research when GiveWell was just a part-time project).
  • In GiveWell’s first round of full-time research in the fall of 2007, PSI was our top-rated charity in our “Saving Lives in Africa” cause, and GiveWell sent PSI a $25,000 grant. We believed that PSI’s cost-effectiveness was roughly in-line with its own $/DALY estimates.
  • In mid-2009, we conducted a more thorough and in-depth analysis of international aid charities reviewing hundreds of organizations (instead of merely reviewing the ones that applied for a grant as we had in 2007). In that process, PSI maintained its “Recommended” status and we ranked it fourth out of the hundreds of charities we had considered. In that report’s review of PSI’s impact, we questioned whether PSI’s programs cause behavior change (such as condom use by individuals who did not previously use condoms).

What changed?

In 2010, we returned to all our previously completed charity reviews to reconsider them and update them with current information. In that process, our conclusion about PSI changed, due specifically, to (a) our view of the way in which PSI uses and evaluates the monitoring data that they collect and (b) new data we found that provided estimates for changes in health behaviors across all of Sub-Saharan Africa.

In our July 2009 review of PSI, we had concluded that

Data from [PSI’s monitoring and evaluation] surveys indicates that PSI’s products are likely being used (more below). It is unclear, however, whether the availability of PSI’s products causes behavioral change (such as condom use by individuals who did not previously use condoms).

While we weren’t sure that PSI’s programs were effectively creating behavior change, we felt that PSI was carefully evaluating its programs in a way that would identify failing programs and lead to improvements. We were ambivalent about PSI’s impact but felt good enough about its monitoring and evaluation process to recommend it. We believe that a charity (a) focused on programs with strong track records (as PSI is by promoting condom and bednet use) that (b) has an unusual degree of transparency while (c) carefully evaluating its programs to determine whether they are working is worth recommending to donors.

Earlier this year, PSI questioned our conclusion that its programs don’t cause behavior change. PSI made its case for impact by stating that “out of 163 available multi-round studies of behavior change, 82 have shown statistically significant impact on at least one behavior change indicator.” We didn’t find this case compelling as we see many ways PSI could measure these results without having significant impact. Specifically:

  • PSI’s test for “impact” – a single indicator showing a statistically significant change in behavior – seems unreasonably generous. PSI didn’t offer a compelling justification for the test it used. This led us to question the quality of process PSI uses to monitor and evaluate whether its programs are largely succeeding or failing.
  • We also found data providing estimates for bednet use across Sub-Saharan Africa and compared this data to measured changes in PSI’s focus areas to broader changes in bednet and condom use across Sub-Saharan Africa. Our analysis of this data shows that changes of health product use in PSI programs seems in line with changes in non-PSI areas across Sub-Saharan Africa (details in our review of PSI’s impact): PSI’s average change was 3-4% for condoms (compared to 2-3% across Sub-Saharan Africa) and 12-14% for bednets (compared to 23-28% across Sub-Saharan Africa). While we do not believe this demonstrates that PSI is not having impact, this data does shift our assessment of the likelihood PSI’s programs are effective and increases the burden of proof on PSI to demonstrate its programs’ effectiveness.

In part, the downgrade is a reflection of the fact that we are continually digging deeper and raising the bar higher for our recommendations. When we first encountered PSI, it was the only nonprofit we could find that seemed to have any meaningful data on its impact; the limited slice of data we looked at seemed to indicate impact and to be consistent with its own cost-effectiveness estimates; and that was enough for us. By now, we are more careful about demanding data that is broadly representative of the full organization’s activities, and in our fuller analysis of PSI’s data we were less compelled. We also place high value on admitting and responding to shortcomings. If PSI agreed with our view that its data does not convincingly demonstrate impact, we would consider a higher rating based on discussions of its ongoing learning and adjustment.

We still believe that PSI stands out from the vast majority of international aid charities because of its focus on programs with strong track record and its unusual and commendable degree of transparency. (If I had to guess I’d say that PSI is probably in the 99th percentile of all international aid charities in terms of its impact per dollar.)

Nevertheless, our focus at GiveWell is identifying the very best charities and for the reasons above, we no longer feel comfortable placing PSI in that group. We plan to continue to watch PSI carefully and reevaluate its status as new information becomes available.