A donor of ours earmarked $10,000 for regranting to a local charity in India, and in deciding how to give this away (and for general learning) we conducted 20+ site visits to small NGOs during our travels. In a sense, this was a chance for us to try out a more traditional method of giving: heavily based on referrals, site visits, and informal impressions rather than desk research.

This post, a followup to posts on our general thoughts on India (from myself, Elie and Natalie) summarizes our thoughts from these site visits and this decision. Note that we have posted detailed notes (and in some cases pictures) from the site visits at the official page of GiveWell’s trip to India.

Criteria

Our normal criteria weren’t a good fit with the organizations we visited, most of which were tiny and without the capacity for extensive monitoring, evaluation and documentation. So we followed the same basic principles that led us to our original criteria: look around, ask questions, and try to articulate our reactions in ways that lead to consistent principles.

The questions that ended up mattering most to us were:

Is the organization serving a population that is clearly in need? Is the organization run by people who seem thoughtful, competent and well-intentioned? These are the questions that seem most amenable to being answered by site visits, and need little explanation.

To what extent do the organization’s activities flow from clients’ needs, as opposed to donors’ strategies? It’s easiest to articulate the categories we mentally placed organizations in through a few examples:

• Room to Read, a large international NGO, seemed focused, to its core, on supporting and promoting libraries. I’d refer to this as a “strategic” organization, consistently pursuing a particular theory of change.
• Helping Hands, a small community NGO, seemed much more “improvisational.” When we asked the woman running it how she chose whom and what to fund, she told stories of individuals and one-off events (girls who needed money, schools that needed new supplies, etc.)
• Seva Mandir, an NGO working in many areas around the city of Udaipur, had a set of relatively large-scale programs, each of which seemed (from conversations) to have sprung from specific, consistent, repeated needs and requests of clients. Its described process for deciding which programs to execute revolved around formal village meetings. I’d term this a “systematically bottom-up” organization.

The approach we felt most comfortable with was the “systematically bottom-up” approach. “Improvisational” organizations seemed to rely too heavily on specific personalities and relationships, while “strategic” organizations left us concerned about the extent to which they were run for clients as opposed to donors, and the appropriateness of their programs for the people with whom they were working. These concerns mean not that we would never recommend such organizations, but that the need for strong monitoring and evaluation (which our normal research process emphasizes and our site visits did not) becomes more central for them.

How insulated is the organization’s management from potential problems? Some organizations are structured for infrequent, limited interactions with clients, while others are built on constant and close contact with clients. Some organizations try to make very long-term or difficult-to-observe differences in people’s lives (examples: sports for character development, advocacy for changing sexual and other private practices), while others aim for more tangible, short-term help.

Both of these distinctions relate to what we came to refer to as the “insulation level” of an organization. Some organizations seem, by their basic structure, likely to become aware of any failure to help clients; for organizations without this feature, monitoring and evaluation become more important to us.

Related to this, we often (when we felt it was appropriate) asked staff to tell us about specific clients, and preferred organizations where they seemed to know a lot about the lives, histories and particular needs of individuals.

How do staff prioritize clients vs. funders?

At one orphanage we visited, we were told that the children were sad today because they couldn’t go outside; when we asked why they couldn’t outside, we were told that they were staying in to meet us. On another visit, we joined a village meeting and were told that the people had been waiting for us for over an hour. We were embarrassed and unnerved by these situations.

We feel strongly that clients ought to come first, which for small organizations (often without specialized fundraising staff) may mean that funders and other visitors have to wait their turn. The orphanage we felt best about was the one where the children assembled for a brief greeting and then dispersed to play computer games and otherwise enjoy themselves. The director of the organization we ultimately awarded the $10,000 to was clear and unapologetic about the fact that her work with clients was more important than her time with us (more below).

Thoughts on different types of organizations

The organizations we found ourselves most interested in were the ones that focused on orphans and/or street children. It seemed to us that these were the populations with the greatest risks and needs, and that the help they needed had a “low-insulation” quality: organizations working with them aim to ensure that they are having basic needs met, are being raised in healthy environments closer to “normal” than what they would have otherwise, and end up healthy and educated.

There are concerns about orphanages and the extent to which they may be taking children away from their families. We definitely felt more comfortable with some orphanages than with others (examples above), and as mentioned below, we ultimately felt most comfortable with a center that provided services to children (including food and shelter) while not taking complete control of their lives. One thing that does appeal to me about orphanages, though, is that the children in them seem generally to have excellent English – something that I believe is a very real and strong advantage in the Mumbai economy.

We were also very interested in organizations focused on keeping children out of the sex trade, simply because the need is so extreme and the goal seems so valuable to us. However, the specific organizations we visited in this category didn’t leave us feeling confident, partly because of the greater challenges one faces in knowing whether their activities are working.

Given the striking and visible needs we saw all around us, we were less interested in organizations focused on more intangible, theoretical benefits. For example, Magic Bus facilitates soccer programs for children; it’s hard for me to justify this focus when those same children are underfed and under-sheltered. I have heard the argument that sports have a character-building quality that ultimately results in more good being done, but this sort of reasoning strikes me as theoretical and romanticized, and I wouldn’t accept this hypothesis without strong evidence (which I do not believe exists).

We have mixed feelings about education-focused programs.

• The evidence that education is causally helpful for later success in life is surprisingly weak – there isn’t strong evidence either for or against the idea. Our intuition is that education is indeed very important; based on conversations we’ve had, it seems a strong prerequisite to many of the more desirable career tracks. More at our writeup on developing-world education.
• We’ve seen no organizations that can make a compelling case, using academic performance data, for their positive impact on education. The best we’ve seen in this area is Pratham, which we visited twice while in Mumbai.
• Our site visits to education charities left us with little sense of the quality of the programs; we weren’t even sure what to look for. See our notes on Pratham for more on this.

Salaam Baalak Trust
Salaam Baalak Trust was the organization we felt performed best on the metrics above:

• It is a drop-in center for children living on the street. Unlike orphanages, it encourages children to live primarily with their existing families, but it provides shelter, meals, counseling and other services for children to use as they wish.
• This basic program is strong on the insulation- and strategy-related criteria above, and the organization is small enough that the two site visits we made gave us a look at a good proportion of what it does.
• Its director, Dinaz Stafford, left a good impression on us.
  • We had a definite sense that she was putting clients before visitors: she interrupted a meeting to speak to a child who had come in, was 45 minutes late to another meeting (explaining she had been dealing with a child who was having issues) and told us we had to leave when it was time for her to attend a meeting about children who were struggling.
  • She also invited us to examine her case files (on condition that we not record or disseminate names), which gave basic information on each child’s family situation and health over time, and when we asked her questions about specific children, she was easily able to rattle off specifics about them.

Previous posts shared Holden’s and Elie’s general thoughts from last year’s several-month staff trip to India. This post shares my thoughts; the next will discuss our thought process in deciding which local charity to support.

General notes on site visits

• I was quite surprised (pleasantly so) by how receptive the charity directors we contacted in Mumbai were to our requests for visits. Unsurprisingly, none of the small organizations we contacted had heard of GiveWell, but nearly all were willing to show us their work and to let us publish our accounts of our visits online. We are very grateful for their openness and for the time they spent with us.
• Some of the programs we saw were similar to programs found in the cities in the U.S., such as teaching parents and children about how to prevent and deal with child abuse, pre-schools, prison programs, after school tutoring, scholarships, and sports clubs. Others were very different from what I’d encountered here: helping slum dwellers respond to government agents who burned down their homes, building walls around wells, distributing goats and seeds, marketing chemicals used to clean water, providing tuberculosis treatment, orphanages, teaching elected representative about how government works, working with the daughters of prostitutes to prevent them entering the sex trade, and providing food and a place to sleep to children who spend their days on the street.
• We often felt that the time spent talking to staff members, especially program staff in management positions was more informative than seeing the charity’s programs in action. From talking to staff, we learned about the organization’s priorities, the breadth of its work, how it judged its own performance, and what challenges it had encountered. Seeing programs in action provided a sense of concreteness and an emotional connection to the organization, but due to our short stay and the fact that we often saw only a portion of the work the organization did (and this portion was chosen by the organization), this on its own did little to improve our understanding of organization. That being said, we could see program visits being informative in cases where we have basic, factual questions about an organization we are reviewing. And given the need for translators in some cases and general communication difficulties, there was definitely value in the ability to have face-to-face conversations.

Observations

• We met with an organization working to educate children and parents about child abuse. While we certainly believe this to be a worthy goal, our conversation highlighted the even greater difficulties involved in dealing with this issue in India than the U.S. The organization interviews children and when it finds instances of abuse (often by relatives), informs a non-offending parent. When made aware of abuse, parents often refused counseling for their children and caring adults were often unable to remove the child from the situation due to lack of alternatives.
• We heard both about progressive government programs to provide guaranteed work for 100 days a year, food assistance programs, and day care services, as well as cases of government agents burning down slums to make way for higher-value developments and requiring bribes to process paperwork.
• We heard from a few organizations about the need to build trust in communities before being able to fully run programs there. This process could take years. Building trust could involve providing direct services (such as day care centers) when the ultimate goal was to teach people to advocate for the government services they’re entitled to, or it could involve establishing village committees that need 75% participation by villagers before they can vote on which programs to implement.
• When we visited a village in which Seva Mandir worked, 30-40 people (almost all men) showed up to talk to us and waited over an hour for our arrival.
• Based on my experience with school libraries in the U.S., I expected the Room to Read library we visited to have hundreds of books, covering a range of reading levels and topics. I was surprised to find only a few dozen titles (with multiple copies of each title). The books we saw appeared to be for a very basic reading level (just a few words per page and lots of pictures), and the more advanced books were in a closed cabinet and appeared rather dusty.
• I was surprised by the level of interest in the PSI street performances (see picture above). I didn’t expect that there would be a high level of voluntary interest in public service messages such as ‘use a condom even when you’re drunk’ and ‘use water purification products so you don’t get diarrhea.’ The messages appeared to be presented in an entertaining way (though I couldn’t understand the dialogue) and people were gathered around in large numbers and appeared highly engaged.
• I went to a number of orphanages. These largely conformed to my expectations. As I had heard anecdotally before, few of the children seemed to actually have no parents or relatives. Instead they were mostly children whose parents could not or did not want to care for them. The orphanages for the most part seemed capable of providing for the basic needs of the children (food, water, shelter, and education), though in one case the orphanage director said his organization survived month-to-month, hoping that donations would come in time to pay the bills. Unsurprisingly, these places did not strike me as substitutes for family life, though I was somewhat surprised by a couple of cases of what I would consider minor mistreatment. One director told us that staff are not allowed to hit children because staff turnover is relatively high, but that the directors were allowed to spank them. At another orphanage, the children were kept from their naptime to sit in front of the director and I while we discussed the organization. I was also told that they are taken to sing at public events, which had led to donations in the past.
• I was told by one organization that its measure of success is ‘the percentage of children who complete the program who do not enter prostitution.’ However, “completing the program” was defined as those who did not drop out of the program before the age of 21 and were settled and married. In other words, it seems that only success cases are included in the success rate. Unsurprisingly, the organization claimed a 100% success rate on this metric.

A previous post shared Holden’s general thoughts from last year’s several-month staff trip to India. This post shares my thoughts; the next will share Natalie’s.

Site visits are short and it’s impossible get a full picture of an organization, its work, and its likely impact. But, little things about how programs look on the ground and how they compare to what the marketing materials say can be telling.

• “We give shoes to poor children.” I visited a soccer practice for kids. It had rained the day before I went, so the field was soaked. Because the field was wet, the kids were allowed to play barefoot, the coach told me. The organization provides the kids with shoes, which the kids must wear. The coach said that the kids were happy because they prefer to play barefoot, but the organization only lets them do so when it’s wet out. Later that day, I went back to the organization’s office and spoke to the development director. As part of his presentation, he noted that the organization provides children with shoes, and the kids love coming to the field and putting on their shoes.
• “We teach English.” English appears to be an extremely valuable skill, allowing speakers to obtain relatively high paying jobs such as working in a call center. One organization we visited taught English classes to teenagers, which seems like a very promising program to run in Mumbai. But, after visiting the classroom, we wondered how effective the class actually was. In the class we visited, the teacher stood at the front of the class writing English sentences on the board and the students copied down the sentences into their notebooks. When we were there, we saw approximately 5 sentences on the board, one of which was “We shall make your tongue right again.”
• “We teach computer classes.” Several organizations we visited offered computer classes for kids. The classes were held in rooms with a few computers. A couple of times, I clicked on the Start menu of computers not currently in use to take a look at recently accessed documents. I wanted to know what types of work the kids did. When I loaded Microsoft Word or Microsoft Excel – two of the programs the organization said they used in their classes – it was clear that this was the first time these programs had been loaded. They asked me to register my version of Excel and enter a username.
• “We educate children on health behaviors.” We visited an organization offering health classes for children. The topic of the session I attended was “healthy eating” and the teachers talked about not eating too much junk food which is high in calories. While it makes sense that a child should eat fruits and vegetables and not too much candy and fried food, it didn’t seem like the most pressing message for the children I saw. They were all frail, looked a lot younger than they were, and may have been malnourished. I felt like I was attending a class that would fit better at school in the United States than a classroom in a Mumbai slum.

In Mumbai, we also had the chance to visit several organizations we had already reviewed on our website and get an on-the-ground perspective. I visited Mumbai’s TB Control Program, guided by a WHO consultant stationed in Mumbai. This wasn’t a visit, specifically, to the Stop TB Partnership, but because Stop TB supports government-run TB programs, this visit was to the type of program Stop TB supports. Each person I met with from the WHO consultant to the director of Mumbai’s TB Control Program to directors of hospitals and clinics were (a) aware of monitoring procedures, (b) answered all my questions about how they monitor, and (c) often showed me examples of documents and data as support.

For example, a major focus of TB control programs is making sure that patients maintain the 6-month drug regimen, taking all prescribed doses and don’t stop taking them early (which often happens after a couple months as symptoms begin to subside). When I visited a local TB treatment center, I asked them how they monitor patients to ensure that they take all doses. The treatment center director told his staff to go into the back room and bring back a patient’s box of drugs along with that patient’s paper records. My guide showed me how he would check this particular patient’s records. The treatment center keeps each patient’s box of pills, the patient’s treatment schedule, and all used pill wrappers. At any point, an auditor can come and check the box and the paperwork to make sure everything is in order.

Seeing the monitoring and data-collection operation up close – and the ease with which the people I met with discussed the data and issues associated with collecting it – gave me more confidence in the monitoring data we rely in our report.

Last year, the staff of GiveWell moved to Mumbai for several months. We did this largely for personal reasons (our staff is small enough that we were able to coordinate it, and we all thought it would be interesting) but also took advantage of our time there to visit 20 nonprofits (some of them more than once). This post contains my general impressions; future posts will contain impressions from other GiveWell staff members as well as our thoughts on specific nonprofits. Our full set of notes and pictures from site visits to local charities in India is also available.

Limits to learningThe first time we saw the metro, we observed the people hanging out of the doorways and concluded that it was amazingly crowded. The second time we saw it, we noticed that the insides of the cars were actually empty at that time, but people were still hanging out of the doorways. By the time I was actually riding the metro, I realized that hanging out of the doorway was a privilege, not a necessity (and I did it whenever I could).

A lot of the observations I hear people make about their travels – and perhaps some of the observations I’ll make below – seem to me to be along the lines of “The subway is so crowded that people have to hang out of the doorways.” It’s very easy to look around and feel that you’ve seen something significant and meaningful, but very hard to get the full context needed to really understand what it is you’re looking at. Although we were in India for several months, there was never any real possibility of become fully integrated into society or blending in with the crowd. Our ethnicity alone marked us as clear outsiders, targets for panhandling to some and photo opportunities for others.

So take the rest of my observations – and, I’d argue, the observations you hear from anyone’s travels in foreign countries – with a grain of salt.

Absolute poverty
Across the board – with the exception of the very upper end of the income range (which included us) – poverty levels and living conditions seemed substantially and visibly worse than what one sees in the U.S. We saw people living in slums – and on the street – with much less space and much worse sanitation than any regular living conditions I’m aware of (including public housing) in the U.S.

I was also disturbed by the state of many animals I saw, particularly dogs and cattle – they consistently looked more emaciated and sickly than anything I’m used to seeing in the U.S.

We’ve argued, mostly based on data, that what counts as “poor” in the U.S. would be seen as “rich” in much of the rest of the world. Spending time in India reinforced this idea in a tangible way.

Relative povertyI found that looks were often deceiving when it came to guessing which people were lowest-income and/or most appropriate as recipients of aid.

• As discussed previously, there are certainly cases of relatively wealthy people living in the slums. I ran across anecdotes of people who owned motorcycles, people who had college degrees and promising careers, and even one woman who owned a flat outside the slum but continued to spend most of her time in it. My point isn’t that any of these people are wealthy in U.S. terms; it’s that I wouldn’t expect a donation from an American to go very far toward helping people who have substantial assets, education, etc.
• In conversations, we got the impression that panhandling is a particular money-making strategy and is not necessarily the exclusive domain of the poorest. In fact, there may often be property rights around certain prime spots for panhandling, which have to be rented. We tended to encounter more panhandlers in areas that were wealthier and had higher concentrations of foreigners – on the occasions when we went into more remote, poorer areas (including a couple of accidental trips through slums), we were rarely if ever asked for money. I think there is a strong case to be made that the people asking for money are not the ones who would most benefit from it.
• It seemed to be conventional wisdom among the people we talked to, particularly at NGOs, that Mumbai was a very wealthy city and that no one there was really poor compared to people from rural Bihar, Andhra Pradesh, etc. Here our instincts diverged heavily from those of locals: we thought many of those we saw in Mumbai seemed like appropriate aid recipients, but multi-city NGOs we talked to expressed strong preferences for working in poorer areas. One stated that it was planning on expanding to Mumbai primarily for fundraising purposes (since it would then have better access to donors living there).

The economyOne major difference between my time in India (especially Mumbai) and my time in Africa was the sense I got of the local economy. In the conversations I had during my trip to Africa, many spoke of the difficulty of getting jobs and seemed to see government and NGO jobs as most desirable. Most of activity I saw involved people providing redundant services (for example, when we drove through a village our car was surrounded by children trying to sell us water – I saw similar redundancy among people selling SIM card recharges and souvenirs). By contrast, in India many of the people I spoke to were adamant that (a) just about anyone can get a job in Mumbai; (b) anyone with valuable skills (such as English speaking) can get a good job in Mumbai.

Just by walking around Mumbai, we saw people making a living in a huge variety of ways. Our tour of the Dharavi slum was certainly consistent with Reality Tours’s statement that

Dharavi’s industries have an annual turnover of approximately US$ 665 million. Through our tour visitors experience a wide range of these activities: recycling, pottery-making, embroidery, bakery, soap factory, leather tanning, poppadom-making and many more. Most of these things are created in innovative ways and in very small spaces!

One of the highlights of the trip for me was visiting Ebizon Netinfo, a website development team that had been working with us remotely for over a year, outside of Delhi. We enjoyed their enthusiasm and optimism; like many we spoke to in India, they believe that with hard work they are headed for serious financial advancement.

The overall picture I got was of an economy that’s extremely poor compared to the U.S. but has a lot more prospects for growth and optimism than what I saw on my trip to Africa. This would make it a very promising place for the kind of giving that I, as opposed to Natalie, prefer.

Site visitsA donor of ours earmarked $10,000 for regranting to a local charity in India, and in deciding how to give this away (and for general learning) we conducted 20+ site visits to small NGOs during our travels. In a sense, this was a chance for us to try out a more traditional method of giving: heavily based on referrals, site visits, and informal impressions rather than desk research.

Future posts will discuss much of what we saw on these site visits and what we came away with. Here I just want to note that

• Small NGOs have the advantage that a single site visit can give one a look at a sizable portion of what they’re doing, and that they are (at least potentially) in close and constant contact with their clients. But they also raise problems of their own for a donor.
  • The room for more funding question can be particularly difficult with small organizations not built to scale.
  • Many small organizations simply seemed to have issues with professionalism and competence, as well as with listening to clients and putting their needs first. (Specific examples are forthcoming.)
• A site visit can give a general impression of what staff are like, how they think, and how they interact with clients. However, many NGOs’ goals are very long term (for example, improving education), and in most cases a site visit isn’t nearly enough to form confidence in an NGO’s effect on clients’ lives.
• As we usually do, we sought NGOs we could be confident in with the information we had. We did run across a couple that I feel good about supporting, despite the fact that our existing criteria are not a fit for them; thinking about these cases has been useful in re-examining our criteria.