A few responses to your last:

1. Few organizations want the responsibility of housing sensitive info like SS#s in house. Some fear the data could be lost or misused somehow (credit card companies lose sensitive info all the time - imagine a barebones nonprofit?) or that the government will come knocking asking for access for more nefarious purposes (like going after people using false SS#s in employment projects and free tax preparation programs, or Homeland Security coming after it to find & purge people for security reasons).

Also, what you need to understand is that most NGOs and charities strengths’ are not centered around data collection. Most DO collect data from their clients, but they are not very good at designing data systems, collecting data, storing data and developing systems to use/analyze it over time. This all costs money. If their donors really cared, they would fund rigorous evaluation - as you point out, few obviously do.

2. The info is absolutely out there on what works in improving life outcomes for the disadvantaged. It is just very scattered, and of course people have strong opinions on what aspects should take priority - juvenile justice reform or education, poverty alleviation or improved preventative health care, youth employment, etc.

Anyway, one starting point for your edification is The Annie E. Casey Foundation website. FYI - this foundation brought together experts over a period of years to discuss how to do community development best and improve family outcomes. They covered what worked and what didn’t extensively and used this information to design their huge, 10-year “Making Connections” initiative underway now which has a huge, nationwide data collection component. Check out their website to see what data they have already collected and analyzed (www.aecf.org).

Also, this foundation also funds and publishes a great resource called “Kids Count”. This annual report is data heavy and basically outlines the status of states vis-a-vis the most significant indicators that can determine whether kids succeed in life in America. Again - check out www.aecf.org.

I worked for this foundation so I know their work better than others, and how seriously they take data collection/evaluation, but all the big foundations fund data collection to some extent, especially when it comes to different pieces of poverty alleviation. Go to the Brookings Institute (a big recipient of these funds) for more great data and info on “what’s wrong”.

3. Sure, there are many NGOs that do crummy evaluation work, particularly in the U.S. But to see something impressive, check out the data component that is being designed by AECF for the Making Connections initiative - extensive & super pricey, but will ultimately be extremely informative on whether the project is accomplishing what it sets out to do.

1. I agree with you that legal and privacy issues aren’t to be taken lightly. But I don’t see why they can’t be a condition of aid. This is another way in which charities are uniquely positioned to carry out research; presumably, if what they offer is very valuable, people will be willing to participate in a study to get it.

2. I don’t agree with you that there’s already enough information on “what’s wrong.” As far as I can tell, nobody knows much of anything about what really works in improving life outcomes for the disadvantaged. If you do, could you point me to a starting point?

3. It makes perfect sense to me that people are “sick of being evaluated.” Firstly, any kind of learning and improvement, esp. in this area, is going to take a long time no matter what. But also, I feel like most of the evaluation that takes place today results in nothing - it’s done because it was part of the grant proposal, but it is rarely rigorous enough to conclude anything, nor are many interested in using it to inform their opinions. This post was about the former - my main point being that randomizing evaluation would slightly increase the cost of it while changing it from basically useless to extremely informative - and the latter issue remains, but GiveWell hopes to make a dent in it.

Having worked with many poor communities in the past, most are pretty sick of being evaluated, while nothing changes for them. And the fact is there are tons of studies out there on the problems of poverty and the poor - more information on “what’s wrong” is not lacking.

And, re: Holden’s last post, participant data and information is incredibly sensitive (things like SS numbers, legal status, income, investments). Lots of legal and privacy issues arise when dealing with this sort of information. Very complex issue, not to be taken lightly.

On a positive note, one donor that takes data and evaluation super seriously is the Baltimore-based Annie E. Casey Foundation (a UPS charity). This group is pretty cutting edge in terms of how it uses data to shape programs and inform the public about policy implications (in no small part b/c their board is made up of number-crunching UPS execs).

Somewhere in between is the approach I outlined above - just asking people to send a letter or email reporting something basic like their income or educational status. I do think the 2-hour interview approach is ideal, but what I’m describing might allow a charity to gather meaningful data more continuously, rather than saving up for years for “the big study.”

And if charities already have they systems in place to track their participants, it wouldn’t be too hard to track non-participants as well.

Everything you say makes sense to me. I do want to put it out there that a lot of charities already do the hard work of following their clients after “graduation” from their programs - in many cases the randomization is actually the only thing that is really missing (although I also recognize that following non-clients is probably much harder even than following former clients).

One other thing I wonder is how much the problems you describe could be mitigated by the way you use incentives. My basic reasoning is that if you have $X, rather than giving it to an employee to hound clients and follow up with them, it seems potentially better to give it to the clients themselves, conditional on participation. Partly because the money is presumably worth more to them; partly because if they really want to participate, they will have a much easier time doing so than your tracker has finding them; and partly because you can structure the incentive in particularly appealing ways. (For example, as a lottery with a big grand prize - this seems to have big psychological appeal, esp. to low-income people).

What incentives do people have to participate in your study? What do you think of this idea?

I agree that the information gained through a randomized design is worth the effort it takes to collect it (and the lottery seems like a good system), but I think you are underestimating the effort it takes to collect data, especially in underprivledged populations.

If you just send out a questionaire you are probably going to get a 50% response rate at best, and it is VERY LIKELY THAT THE RATE AND COMPOSITION OF RESPONDERS WILL DIFFER BETWEEN INTERVENTION AND CONTROL GROUPS. You need to have people tracking each case and getting on the phones or going out into the field to collect data and to insure that the rate of attrition is minimal and does not differ meaningly between groups.

Another issue to consider is consent to research. I don’t know how this would work in a non-university setting, but here we are mandated to obtain informed consent for research and to protect the rights of research subjects (especially if the subjects are children).

And if you are working with an underprivledged population there are barriers in tracking participants. Cell phones get disconnected, people change addresses or enter institutions (prisons, hospitals, rehab centers), and many do NOT have email and computer access.

I remember reading a previous blog entry about researchers and charities joining forces. This is what we do in our study (we work in collarboration with foster care agencies) and I think this may be the most feasable option.

Basically, it’s the usual problem that almost everyone thinks it’s legitimate to have preferences over beliefs (and in a sense they may be right, fact is, I do have such preferences even if I’m uncomfortable about it)

By the way, this is very interesting
http://www.overcomingbias.com/2007/02/unequal_inequal.html