The Harvard Legal Aid Bureau provides free legal services, via volunteer Harvard Law students, to “low-income people in civil (non-criminal) matters in order to ensure equal access to justice and to remove legal barriers to economic opportunity.”
It’s about as intuitive an intervention as any. Good legal services are expensive; the services provided by Harvard Law students are probably good; so it seems they would be beneficial to the low-income people receiving them. But a recent (unpublished) study calls this into question, implying that the benefits of the program (at least the service studied) are small and that there are possible harms as well.
Before getting into the details of the study, it’s important to credit the Harvard Legal Aid Bureau for participating in this study. Many organizations would likely shy away from a study that could find negative results. The Bureau deserves praise for participating in the study, and our opinion of them is now higher for their participation.
The study uses a randomized controlled trial methodology to evaluate the impact of offering legal services to clients seeking unemployment benefits. The paper finds that:
- An offer of services had no statistically significant effect on the likelihood of an individual’s receiving government benefits. 76% of those offered services (the treatment group) by the Harvard group received benefits; 72% of those who were not offered services (the control group) received benefits.
- An offer of services had a statistically significant effect on the delay until an individual received benefits. The treatment group received services more than 13 days later than the control group. The paper argues that those applying for unemployment benefits often have an immediate need for cash, and this delay is a significant negative impact of the program.
What explains the results? The authors offer a number of suggestions.
- The authors note that they used an intent-to-treat methodology in this paper. That is, they assessed the impact of the offer of services, not services, themselves. Ultimately, 90%+ of the treatment group received services while only 49% of the control group did. The fact that nearly half the control group received services may explain the lack of a statistically significant effect of offering services. We believe that using an intent-to-treat approach was the correct approach here, especially given the question most important to us at GiveWell: what is the marginal impact of a particular organization? In the absence of the Harvard Legal Aid Bureau, it seems that approximately half of the clients they serve would be served elsewhere.
- The Harvard Legal Aid Bureau is staffed by law students (under the supervision of a practicing attorney). The authors consider the possibility that because of their limited experience, the students are less effective than practicing attorneys would be. Nevertheless, the authors conclude — largely it seems based on their observations of the students and experience with similar organizations — that the students are likely competent and this is unlikely a cause of the lack of impact.
- The authors suggest that the additional delay experienced by students’ clients may be due to the students’ schedules and the fact that they only commit part-time to the legal aid bureau.
It’s only one study and it shouldn’t be taken too far – it addresses the impact of one service offered by one particular legal aid organization, and doesn’t show that free legal services for low-income people are unnecessary. But it does hint that
- From a donor perspective, legal aid charities may not have room for more funding. (Here it looks like the supply was enough that the existence of a particular group, Harvard’s, couldn’t be connected to strong impact.)
- A sensible-seeming program can have unintended consequences. In this case, matching low-income people with law students at a prestigious institution might also mean matching them with people who don’t have enough time to address their issues promptly.
Comments
They were absolutely right to test the impact of the offer of services rather than the services themselves. It sounds like a well-designed study. They deserve credit for having conducted the study and making the negative results public. I would think that if someone really believes in his own charity, the natural response to a study that shows no impact would be to immediately blame the study as being flawed (I feel like I see this all the time in other fields, especially politics). I imagine it’s tough to accept that your work may have not been accomplishing anything!
I’ve always compared the incentive structure for charities regarding self-evaluation with that of alternative medicine providers. If a pharmaceutical company develops a cholesterol-lowering drug, they need to provide evidence that it actually lowers cholesterol in order to have it approved and marketed. However, if you’re a company selling ginseng root or multivitamins, the buyers (typically) don’t care whether it’s been shown to work, they either believe in it or they don’t (like a religion!) So, if the buyer doesn’t care about data and the study will probably not show the treatment to be effective anyway, why would the company spend money to study it?
I hope this unpublished study becomes published!
When you say that 90% of the treatment group received services versus 49% of the control group, you mean legal services, right?
How did 49% of the control group receive legal services? What did they cost? That has to be factored in some how (maybe net benefits received?).
I’d really like to see the data. Also, I’m curious to know that the participants thought about the longer time horizon and whether it was worth it or not.
James, yes, the paper reports that 49% of the control group received legal services:
The authors don’t state the portion of the control group that each of the above service providers served. Note that Greater Boston Legal Services and the Volunteer Lawyers Project both state on their websites that they offer free representation. My guess is that the clinical program at Northeastern Law School does as well, and it is common for private attorneys to offer pro bono services to clients in need.
The study examined people who were applying for unemployment benefits. Applicants for unemployment don’t need lawyers or law students to apply for benefits. They do need lawyers when they are first denied benefits and must follow an appellate procedure.
Law students primary reason for existance is to graduate from law school. Running legal aid clinics out of law schools is a teaching exercise. Non profit organizations who provide legal aid by staff attorneys or private attorneys providing those services free of charge (pro bono) do so under far better conditions for the client than do law schools.
To take this one tiny example of hgow law schools insert themselves into the legal aid process so they can secure government funding for same is simply an exercise in shaping propaganda by manipulating the research used to prove a thesis. I don’t know how it is in Massachusetts, but way down here in Texas, private attorneys of all political stripes, and the entire state judiciary fully support the provision of free civil legal aid to the poor. It is not an issue that is open to politicization. It is an issue of fairness. And all people of good will agree.
Samuel,
Note that the study considers individuals appealing a decision on unemployment benefits, not those making an initial claim.
We agree that (a) it’s possible that the explanation for the study’s findings is that the students do not provide adequate legal counsel and that (b) one should not over generalize these results, as this study considers one program offered by one legal service agency. We make both these points in the post.
I would like to see a study of the outcomes where services were provided by legal services programs vs. the outcomes obtained by pro se litigants.
Comments are closed.