The Harvard Legal Aid Bureau provides free legal services, via volunteer Harvard Law students, to “low-income people in civil (non-criminal) matters in order to ensure equal access to justice and to remove legal barriers to economic opportunity.”
It’s about as intuitive an intervention as any. Good legal services are expensive; the services provided by Harvard Law students are probably good; so it seems they would be beneficial to the low-income people receiving them. But a recent (unpublished) study calls this into question, implying that the benefits of the program (at least the service studied) are small and that there are possible harms as well.
Before getting into the details of the study, it’s important to credit the Harvard Legal Aid Bureau for participating in this study. Many organizations would likely shy away from a study that could find negative results. The Bureau deserves praise for participating in the study, and our opinion of them is now higher for their participation.
The study uses a randomized controlled trial methodology to evaluate the impact of offering legal services to clients seeking unemployment benefits. The paper finds that:
- An offer of services had no statistically significant effect on the likelihood of an individual’s receiving government benefits. 76% of those offered services (the treatment group) by the Harvard group received benefits; 72% of those who were not offered services (the control group) received benefits.
- An offer of services had a statistically significant effect on the delay until an individual received benefits. The treatment group received services more than 13 days later than the control group. The paper argues that those applying for unemployment benefits often have an immediate need for cash, and this delay is a significant negative impact of the program.
What explains the results? The authors offer a number of suggestions.
- The authors note that they used an intent-to-treat methodology in this paper. That is, they assessed the impact of the offer of services, not services, themselves. Ultimately, 90%+ of the treatment group received services while only 49% of the control group did. The fact that nearly half the control group received services may explain the lack of a statistically significant effect of offering services. We believe that using an intent-to-treat approach was the correct approach here, especially given the question most important to us at GiveWell: what is the marginal impact of a particular organization? In the absence of the Harvard Legal Aid Bureau, it seems that approximately half of the clients they serve would be served elsewhere.
- The Harvard Legal Aid Bureau is staffed by law students (under the supervision of a practicing attorney). The authors consider the possibility that because of their limited experience, the students are less effective than practicing attorneys would be. Nevertheless, the authors conclude — largely it seems based on their observations of the students and experience with similar organizations — that the students are likely competent and this is unlikely a cause of the lack of impact.
- The authors suggest that the additional delay experienced by students’ clients may be due to the students’ schedules and the fact that they only commit part-time to the legal aid bureau.
It’s only one study and it shouldn’t be taken too far – it addresses the impact of one service offered by one particular legal aid organization, and doesn’t show that free legal services for low-income people are unnecessary. But it does hint that
- From a donor perspective, legal aid charities may not have room for more funding. (Here it looks like the supply was enough that the existence of a particular group, Harvard’s, couldn’t be connected to strong impact.)
- A sensible-seeming program can have unintended consequences. In this case, matching low-income people with law students at a prestigious institution might also mean matching them with people who don’t have enough time to address their issues promptly.