The Harvard Legal Aid Bureau provides free legal services, via volunteer Harvard Law students, to “low-income people in civil (non-criminal) matters in order to ensure equal access to justice and to remove legal barriers to economic opportunity.”

It’s about as intuitive an intervention as any. Good legal services are expensive; the services provided by Harvard Law students are probably good; so it seems they would be beneficial to the low-income people receiving them. But a recent (unpublished) study calls this into question, implying that the benefits of the program (at least the service studied) are small and that there are possible harms as well.

Before getting into the details of the study, it’s important to credit the Harvard Legal Aid Bureau for participating in this study. Many organizations would likely shy away from a study that could find negative results. The Bureau deserves praise for participating in the study, and our opinion of them is now higher for their participation.

The study uses a randomized controlled trial methodology to evaluate the impact of offering legal services to clients seeking unemployment benefits. The paper finds that:

• An offer of services had no statistically significant effect on the likelihood of an individual’s receiving government benefits. 76% of those offered services (the treatment group) by the Harvard group received benefits; 72% of those who were not offered services (the control group) received benefits.
• An offer of services had a statistically significant effect on the delay until an individual received benefits. The treatment group received services more than 13 days later than the control group. The paper argues that those applying for unemployment benefits often have an immediate need for cash, and this delay is a significant negative impact of the program.

What explains the results? The authors offer a number of suggestions.

• The authors note that they used an intent-to-treat methodology in this paper. That is, they assessed the impact of the offer of services, not services, themselves. Ultimately, 90%+ of the treatment group received services while only 49% of the control group did. The fact that nearly half the control group received services may explain the lack of a statistically significant effect of offering services. We believe that using an intent-to-treat approach was the correct approach here, especially given the question most important to us at GiveWell: what is the marginal impact of a particular organization? In the absence of the Harvard Legal Aid Bureau, it seems that approximately half of the clients they serve would be served elsewhere.
• The Harvard Legal Aid Bureau is staffed by law students (under the supervision of a practicing attorney). The authors consider the possibility that because of their limited experience, the students are less effective than practicing attorneys would be. Nevertheless, the authors conclude — largely it seems based on their observations of the students and experience with similar organizations — that the students are likely competent and this is unlikely a cause of the lack of impact.
• The authors suggest that the additional delay experienced by students’ clients may be due to the students’ schedules and the fact that they only commit part-time to the legal aid bureau.

It’s only one study and it shouldn’t be taken too far – it addresses the impact of one service offered by one particular legal aid organization, and doesn’t show that free legal services for low-income people are unnecessary. But it does hint that

• From a donor perspective, legal aid charities may not have room for more funding. (Here it looks like the supply was enough that the existence of a particular group, Harvard’s, couldn’t be connected to strong impact.)
• A sensible-seeming program can have unintended consequences. In this case, matching low-income people with law students at a prestigious institution might also mean matching them with people who don’t have enough time to address their issues promptly.

Our new report grades major disaster relief organizations on their transparency and accountability to donors. It provides detailed reports on each charity, a summary table with our conclusions, and the full details of our process.

We conclude that Doctors Without Borders, Partners in Health and Direct Relief International stand out for the clarity with which they discuss their activities and expenditures.

Over the coming months, we’ll be adding to this report by examining information from the response to the 2004 Asian tsunami. We’ll also be providing some more discussion of non-transparency-related factors behind the donation decision. For example, Direct Relief International and Doctors Without Borders are very different in their focus – the former focuses on distributing supplies while the latter largely facilitates and supplies medical treatment. Personally, I find the latter to be a more appealing vehicle for addressing pressing and challenging needs in a relief situation. Other donors may prefer to support organizations with broader mandates, which are more likely to play direct roles in (for example) providing shelter and assisting with longer-term reconstruction.

That said, we feel that what we have now is a substantial improvement over the information and analysis previously available to donors. It consolidates the clearest and most detailed information provided by each organization, and it can help donors take a first step toward creating incentives for organizations to raise money by being more transparent (and not just more pushy in fundraising) than their competitors.

While we haven’t assessed the quality of individual organizations’ relief efforts, we have accompanied our report with a general overview of what the overall relief effort has and hasn’t accomplished in the year since the disaster, and for how much.

Over the past year, we have been examining disaster relief organizations, with particular attention to the quality of the information they provide on their work in the 2004 Asian tsunami and the 2010 Haiti earthquake.

We don’t think disaster relief is the best cause for impact-oriented donors and we are unable (details below) to apply our standard international aid criteria to disaster relief organizations. However, we think there is a lot of room for improvement in the information currently available for disaster relief donors, and a lot of room to improve incentives for the major disaster relief organizations to be as transparent and accountable as possible.

Questions we’ve focused on

In examining charities’ information, we’ve focused on the following questions:

1. Financials. Did organizations clearly and frequently disclose how much they were seeking, what they’d do if they raised more than they sought, how much they’d raised and how much they’d spent?
2. Transparency on activities. Did charities provide specific and comprehensive accounts of their activities in the affected areas? Could someone on the ground verify or refute their claims about how its money was spent?
3. Results. Did charities publish substantive information on their successes and shortcomings?
4. Everyday work. Do charities publish clear and substantive information about their non-disaster relief work? We’ve argued before that donations intended for disaster relief may effectively fund charities’ everyday operations instead (even if they are formally earmarked for and allocated to disaster relief). Therefore, we think the quality of (and transparency around) a charity’s everyday work ought to carry a heavy weight in donors’ decisions.
   

What we’ve found

1. Financials. Most major organizations have disclosed how much they’ve raised and (less frequently) how much they’ve spent. No organizations have been consistent and clear about how much money they were seeking. In cases where we have been able to find posted amounts sought, the charities posting them generally quickly raised more than they were seeking. In one case the appeal was constantly revised upwards. Despite raising much more than they sought (and than they’ve spent), we’ve only seen two groups that stopped taking donations for the disaster.
2. Transparency on activities. Few organizations are clear about what they’re doing on the ground. Most give examples and very broad budgets. But with a few exceptions, charities have not given specific enough accounts to givea sense of how they spent their money.
3. Results. No evaluations seem to be posted yet for Haiti relief. Many have been posted for work on the Asian tsunami; we haven’t yet had time to evaluate these evaluations (and don’t expect to have done so by next week).
4. Everyday work. Few of the large organizations we examined are clear about how their overall budget breaks down and what they do around the world. There are some groups that are clearer than others.

How we chose organizations to examine

We are unable to predict the location and nature of the next disaster, so it wouldn’t make sense for us to look for an organization with specific ongoing, outstanding, underfunded work (as we usually do). Instead, we wish to focus on the large, global organizations that will probably attract the lion’s share of donor attention and funding whenever, wherever and however the next disaster strikes. To create a list of these organizations, we referred to:

• The Chronicle of Philanthropy’s tally of which charities had raised how much for Haiti relief as of July 2010. The biggest numbers on this list are likely the biggest money-raisers overall; charities absent from the list likely didn’t raise enough to be of interest, or didn’t share their tallies with the Chronicle (and we feel the latter would be a significantly bad sign for accountability).
• A list we had made in the weeks immediately following the Haiti earthquake, noting which charities were advertising via Google Adwords for earthquake-related searches. This is a simple heuristic for finding charities that are likely to rely upon, and solicit, donations from the public at large.

Specifically, we rated and ranked any charity that either (a) appeared in the Chronicle tally and advertised via Google Adwords; or (b) was among the 10 biggest money-raisers in the Chronicle tally, regardless of whether we saw its ads on Google Adwords. These criteria produced a list of 19 prominent charities; we added a few more at our discretion.

More information forthcoming

Next week we will be publishing:

• Detailed writeups on each charity we examined and the information we could find on the above questions.
• A summary table showing the strength of different charities’ answers to our questions, and noting the charities we think stand out for their superior transparency so far.
• Full details of our process for creating these pages.

Added 7/15/21: This blog post reflects the personal views of GiveWell’s co-founder in 2010; it does not represent GiveWell’s organizational view on animal welfare as a cause area. For more on why we don’t currently focus on animal welfare, see here.

We’ve gotten some questions about whether we plan to research charities working on animal welfare, so I wanted to share my thoughts on the cause.

• It’s easy for me to believe that animals often are treated horribly, and live in horrible conditions, relative to people (even people living under the international poverty line). I believe this applies both to animals in factory farms (one resource on this topic is The Way We Eat, co-authored by Peter Singer, (disclosure: Prof. Singer has actively promoted GiveWell)) and to stray animals, particularly in the developing world (such as those I saw throughout my recent stay in India).
• It is unclear to me whether charities are good at improving conditions for these animals. Much of what animal-welfare charities seem to focus on is political advocacy, which introduces a substantial set of complications.
• However, it seems likely to me that there are at least some groups that radically improve the condition of strays, such that if one valued the lives of animals equally to those of humans – or even in the same ballpark – these groups might be competitive with our best charities in terms of what you accomplish for your donation.
• I do value the lives of animals somewhat. I am very disturbed by what I’ve heard of their treatment in factory farms, and I’m interested in “ethical eating,” i.e., adjustments to eating habits that could create less incentive for this treatment.
• I do not value the lives of animals equally to those of humans – not even close. I couldn’t bring myself to give money to animal welfare charities that could be spent on global health instead, given what I understand as the realistic range of cost-effectiveness for the two.
• I recognize that there is a tension between the two points above. One could argue that if I spend more money – or even more time – on my food so that I can eat more ethically, this money or time could have been redirected to helping people in the developing world, and that it’s therefore inconsistent to be interested in “ethical eating” but not animal welfare charity. This argument might be correct, though I believe it is not, and may lay out my thoughts more thoroughly at a later point.

Those are my personal thoughts. For most of GiveWell’s history, the only full-time staff have been myself and Elie Hassenfeld, and Elie values the lives of animals far less than I do. We strongly prefer to research causes that we’re personally interested in, because it’s harder to ask a charity the right questions if you can’t really get behind what it does – so we haven’t given serious consideration to animal welfare charities.

However, we now have an employee who cares more about animal welfare, enough (in our judgment) to potentially do good work researching animal welfare charities. We aren’t yet ready to commit to researching this cause – we need to draw up our plan for next year, which we will be doing soon – but a report on animal welfare charities is a possibility in the next year, and very likely to happen eventually if GiveWell stays in existence.

It’s also possible that we will (eventually) produce content on “ethical eating,” which may be a way (aside from charity) that individuals can spend more money in order to accomplish good. Whether this content fits with our core mission is debatable; it won’t be happening under the GiveWell name in the short term.