Note: a positive effect of recent events was a set of substantive concerns about our model raised by non-profit insiders and others. We put all our reasoning and assumptions on our website precisely because we value critiques that will help us improve, and we look forward to responding to and discussing all concerns shortly. This post, however, focuses on the work we’ve been doing recently, evaluations for Cause 4 (K-12 education).

Three of our finalists focus on creating new schools. Replications, Inc. does so through the public school system, while the Knowledge is Power Program (KIPP) and Achievement First create charter schools. In evaluating these programs, we’re hesitant to focus too much on test scores – the link between test scores and useful skills, or later life outcomes, is far from clear – but test scores are a start, and both the media coverage and the organizations themselves have focused on test scores as the primary evidence of their value-added. So what does the evidence say?

So far, it’s very unclear. All three of the organizations discussed sent us data from an incomplete set of their schools; when it came to providing comparison-school data (to put it in context), what we were provided is even more incomplete. The same is true for the independent reports available on the organizations’ websites. For example, the independent reports that KIPP lists focus either on very few schools, or examine many schools but look only at one year. It’s often unclear to us why particular schools, grades, and years were examined and others weren’t, especially in light of the fact that nearly all test-score data is publicly available through state governments. (Collecting such data may be time-consuming, and we wouldn’t necessarily expect the organizations themselves to do so – but what about funders?)

In our minds, if we want to gauge these organizations’ impact, it’s necessary to collect all easily available and relatively recent data (to avoid concerns about cherry-picking particularly successful schools), and study not just the schools relative to district- or state-wide averages, but the trajectory of students’ scores. The question has been raised whether KIPP, for example, effectively selects stronger students for participation, both through its acceptance process and through attrition (students’ dropping out as time goes on). So even if their students outperform “comparison group” students, this isn’t necessarily an effect of the schools themselves; if we can establish, on the other hand, that their students enter at similar levels to their peers but improve gradually over time, the “KIPP effect” will become much clearer.

This sort of observation wouldn’t support a “KIPP effect” as strongly as a true randomized study would, but it would still be much stronger than what we’ve seen to date. We’ve seen comparisons of particular schools to nearby schools, and to statewide averages, but nothing that is either comprehensive or attempts systematically to address the issues discussed above. For example, this report, by the Education Policy Institute which looks across many KIPP schools, but only at 5th grade test scores in 2003-04. Other available reports look only at a few schools, or schools in a particular city for a one academic year.

So, at this point our plan is to bite the bullet: get all the data we can from state governments, and do our own gruntwork putting it in a form that can be systematically analyzed. We certainly don’t want to reinvent the wheel if someone has already done this work – we’d rather read research from experts that have spent much more time on this than we have – but at this point we don’t see an alternative, because we can’t find anyone who is publishing research on the effect of these programs in a way that addresses concerns about selection/attrition/publication bias.

What do you think? Are we going too far? Is there a faster and better way to answer these questions? Do you know of any research that already exists on these issues (for these charities)?

On Monday, Elie Hassenfeld communicated to the board that he posted a comment on December 31, 2007 related to GiveWell that was not under his own name. The comment was made under the name Talia and is linked here. In response to the actions by Mr. Hassenfeld which were inconsistent with GiveWell’s core values of transparency and honesty, the board has decided to impose a financial penalty of $5000. We have updated our ‘Statement from the GiveWell Board of Directors’ to reflect this situation (please note the addition of paragraph 5). We believe that Mr. Hassenfeld’s previous commitment to the goals of GiveWell demonstrates that he can continue to make a positive contribution to the work of the organization in the future in his current role as a program officer.

The following is a statement from the GiveWell Board of Directors. Audio and materials from the meeting are located here. Additional materials related to this statement will be posted soon.

————-

The board has amended its original statement as of January 11, 2008 at 830pm est. Please note the addition of paragraph 5.

———————–

The Board of Directors wishes to emphasize that GiveWell was, is, and will remain committed to the goal of helping those in need by performing and publishing high quality research on charities within selected spheres for free and discretionary use by all interested parties. We are committed to the goals of improving philanthropy and increasing transparency in the sector by publicly sharing our process, our criteria, our methodologies, our deliberations, and our findings. Furthermore, we remain committed to engaging in public dialogue about how best to achieve these goals and encourage continued comment about our organization’s work.

The Board believes that the acts of misrepresentation that were committed are indefensible and are in direct conflict with the goals of the organization, and we condemn them in the strongest possible terms.

The board apologizes for any violations of public trust resulting from Mr. Karnofsky’s actions.

Effective January 3, 2008, Holden Karnofsky has been removed from his position as Executive Director of GiveWell and from his position as Board Secretary. The Executive Director position is now vacant; Tim Ogden will serve as interim Board Secretary. In addition to being removed from his positions, a financial penalty has been imposed on Mr. Karnofsky. While we are removing him from the Executive Director position, we believe that his previous contributions outside of the acts noted above have demonstrated a commitment to the goals of the organization and have been important to accomplishing GiveWell’s work. As a result, Holden will be moved to a Program Officer position, where we believe he will be valuable in helping GiveWell meet its outstanding commitments to applicants and donors. He will also participate in a program of professional development and mentoring.

In response to the actions by Elie Hassenfeld which were inconsistent with GiveWell’s core values of transparency and honesty, the board has decided to impose a financial penalty. We believe that Mr. Hassenfeld’s previous commitment to the goals of GiveWell demonstrates that he can continue to make a positive contribution to the work of the organization in the future in his current role as a program officer.

The GiveWell Board of Directors will begin meeting bi-weekly to develop and approve operating plans for GiveWell and to discuss the organization’s strategy and future. We will institute an organization-wide communications policy that will mandate that all staff and members of the Board of Directors explicitly disclose their affiliation with the organization when communicating with others. The policy will be made public once it has been completed.

We would also like to make clear to all donors to GiveWell since December 1 that we will contact them directly about these recent changes and will provide them the opportunity to have their gift returned if they so choose.

Signed,

Bob Elliott, Chairman
Greg Jensen, Treasurer
Tim Ogden, Interim Secretary
Lucy Bernholz
Virginia Ford

———–

In addition to the above statement from the Board of Directors as a whole, we would like to separately add the following about Holden’s actions, given our greater personal experience with him than other board members’.

We would like to make clear that the actions that Holden took to conceal his identity were improper and indefensible, as were his attempts to ameliorate the situation by offering a financial contribution.

While in this situation Holden acted improperly, we would like to emphasize that we believe Holden’s commitment to the GiveWell cause is genuine. Those who have commented that his involvement in GiveWell is an attempt to run a scam on the public or to gain financially are simply wrong. His substantial sacrifices financially and personally to take GiveWell from an idea to a reality demonstrate his passion for the values of the organization.

In addition, we would like to emphasize that we believe and have seen in action his stated commitment to the values of openness and honesty. While there are many incidents where Holden did not clearly identify his affiliation with the organization, we view these actions to be the result of his core mistake of thinking improperly about how to represent himself when communicating online.

In our previous professional and personal experiences with Holden, he has shown directness and honesty with those around him, and demanded the same from others. He has acknowledged weaknesses and worked with sincerity to improve those limitations. And, in the past when he has realized that his behavior or thinking was wrong, he not only been willing to change his mind and accept explicit personal responsibility, but take steps to ensure that similar action does not happen again.

It is clear to us that he has realized that his actions were improper, taken responsibility for his actions, and is committed to making sure that he communicates in a way that holds true to the values of honesty and transparency moving forward. We know Holden is committed to these values and expect that he will demonstrate to the community that this behavior was not consistent with his character through his continued work at GiveWell.

We recognize that people can make mistakes and that they can learn powerful lessons from those failures. When we think about the whole picture of Holden, and view his mistakes in the context of his contribution to GiveWell so far as well as his character prior to this project, we believe that he will continue to make a positive impact for the organization in the future.

Bob Elliott, Chairman
Greg Jensen, Treasurer

I did what is apparently known as “astroturfing,” using a fake ID to try to get attention for GiveWell. Why did I do this?

1. I had a horrible lapse in judgment.

2. I have been low on sleep, and running around like a madman. I’d like to think that had something to do with it, but if you want to dismiss that and just call me an idiot, I can’t argue with you.

• I used an old Ask Metafilter account to pose a question about where to find a good charity (the kind of question I was asking everyone I knew a year ago), then I created a new account as Holden0 to answer it with GiveWell (I didn’t intend to hide the fact that the GiveWell link itself came from me, the founder of the project – but posing the setup question as an outsider was completely inappropriate).
• I made two blog comments and sent ten emails from a handle that did not identify me.
• I also made numerous other comments from a handle that did identify me (“Holden”), but did not put up front that I was the founder of the project.

The gory details are here. The current status of the situation is here.

This was a horrible lapse in judgment. I was simply trying to make comments that would not be immediately dismissed as “plugs,” but rather explored on their merits. I now recognize that by not disclosing my identity, I created the appearance of an “objective” observer, and that was wrong.

I hope as you read this, you can identify with having a horrible lapse in judgment. Everything that is true of me is something that has been untrue of me in a moment of weakness, and that includes the value of honesty, which remains the most important one. I hope you can recognize that the best thing I can do is admit it immediately, apologize, and not try to hide anything. That is what I am doing. I hope you still believe in me as someone who will admit to his lapses and mistakes, if not someone who can be counted on to be free of them.

Transparency is the one thing about GiveWell that everyone seems to like. Our focus on measurement is much more contested. I believe that the connection is tight, though, because both are necessary consequences of humility, which is probably the last word you ever thought I’d ask to be associated with.

Transparency is a really big deal to us because we believe that no matter how much we learn and no matter how hard we work, we can always be wrong. That’s why we invite as many people as possible into the conversation.

(When I look at large foundations making multimillion-dollar decisions while keeping their data and reasoning “confidential” – all I see is a gigantic pile of the most unbelievably mind-blowing arrogance of all time. I’m serious. Deciding where to give is too hard and too complex – with all the judgment calls and all the different kinds of thinking it involves, there is just no way Bill Gates wouldn’t benefit from having more outside perspectives. I don’t care how smart he is.)

Measurement is about inviting someone else into the conversation: The Facts. The Facts have a lot to say, and they often contradict what we would have thought. That’s why we have to listen to them. Like transparency, measurement takes a lot of extra effort and expense; like transparency, it can’t solve all your problems by itself; and like transparency, it’s easily worth it if you agree that the issues are extremely complex, and that no matter how much sense something makes in your head, The Facts might disagree.

(And yes, when I hear people talking about how they’re “too busy helping people” to measure, I hear arrogance there too. The only way you could ever take that position is if you are so sure of what you’re doing that you think learning more about it isn’t worth the 10% more you can spend doing the same untested thing. How high on your own infallibility do you have to be to come to that conclusion?)

To a lot of people, humility means speaking with a certain tone of voice, or just plain keeping your mouth shut. If that’s what you think humility is, we don’t have it. To us, humility is constantly saying “The things that make sense to me could be wrong – that’s why I’m going to do everything I can to test them, against others’ ideas and against reality.” Instead of being silently dissatisfied with charity, we’re loudly dissatisfied, so that anyone who disagrees can respond. Instead of happily assuming our dollars are doing good, we demand to see The Facts, so they can respond too.

Laugh if you must, but in the end humility is the defining value of the GiveWell project. We’re not here to impose our solutions on the sector; we’re here because we want to see more questions.

Audio for last Monday’s board meeting is on the way; in the meantime, here’s a summary. The meeting ran about five hours and had heated arguments, tension, drama, a couple car chases, and down-to-the-wire votes. The highlight was the Elie and I ended up reversing our position on 2 of the 3 causes we voted on. Here’s the story.

Employment assistance: HOPE Def. Year Up in Double OT

Elie and I walked into the meeting having settled on a basic judgment call: we preferred high-skill to low-skill job training, for a variety of reasons. The Board (particularly Greg and Bob) questioned all of these reasons heavily.

• We argued that without good comparison-group data, we couldn’t really use empirical evidence to establish whether a charity was making clients much better off than they would be without its help. Therefore, we preferred charities that were clearly teaching a highly specific skill – something that clients would clearly need training in – rather than working on improving motivation, which seems much harder to change in a 12-month or shorter program. By contrast, Greg and Bob felt that the easiest thing to help a person with is neither motivation nor specific skills, but extremely basic things like how to present oneself for an interview – and that this is the sort of coaching they’re unlikely to be able to obtain without a nonprofit’s help.
• Our cost-of-living analysis suggested that lower-skill jobs, paying $10/hr, don’t represent a true living wage – and that they are unlikely to lead naturally to raises, whereas high-skill jobs allow a reasonable and improving standard of living. Greg and Bob were of the mind that getting a first job is the “hardest hump to get over,” and that someone with a job can find a way to get further training and move up, without needing an intensive nonprofit-run program.

The frustrating thing is that everyone in the room was completely guessing on all of this, and I feel that the right kind of evaluation could settle these questions. We voted on this central question – high- vs. low-skill wages – and initially, high-skill won 3 votes to 2, and we prepared to vote between the high-skill organizations Year Up and St. Nick’s.

But even though my instincts still lean toward high-skill training, my confidence was substantially weakened by Greg’s and Bob’s arguments, and it occurred to me more just what a huge tossup/judgment call this all is. With that in mind, I switched my vote, preferring to “break the tie” between approaches by going with The HOPE Program, which is (a) the charity I have the highest confidence in as an organization, due to its impressive and thorough self-evaluation; (b) the charity whose effect is best supported by the (admittedly sketchy) empirical evidence we have.

The rest of the Board agreed with my way of framing the new decision, and after walking in expecting to award Year Up, we unanimously elected HOPE as the winner of our grant.

Saving lives in Africa: PSI 3, PIH 2

All of us agreed that Population Services International and Partners in Health appear to be truly excellent organizations, with dramatically different strengths and weaknesses. On one hand, PSI is a standout in terms of being able to track and document itself, and appears to conduct extremely large-scale, cost-effective activities that save huge numbers of lives. PIH is far inferior by the “dollars per life saved” metric, but of course there’s so much more to the story than that. Their model of providing comprehensive health care is extremely appealing because it is straightforward and community-based, and therefore (in my mind) a much more reliable way of ensuring that we’re truly addressing disadvantaged people’s needs.

This argument got heated and it got impolite. I took PSI’s side and accused Lucy and Tim (PIH defenders) of avoiding my arguments by using “jargon,” and focusing on PIH’s “revolutionariness” instead of just trying to figure out who’s going to help as many people as much as possible. They, in turn, accused me of being too naive to understand PIH’s awesomeness, and they may be right (though for the record, I do think PIH is awesome and would defend it over any other organization I know of but PSI, including many organizations that annihilate it on the “cost per life saved” metric).

Greg, Bob and I carried PSI over Tim and Lucy. Yes, the three outsiders outvoted the two more experienced people. Yes, this is a concern, and yes, I still feel far from 100% on the decision and still think PIH is great. But as of today, I’d still rather grant PSI for a variety of reasons, and like any donor, I’m going with what makes sense to me. All five of us agreed that it was a tough call. The conversation will continue.

Global poverty: Opportunity International rallies from 1st-quarter deficit

Another one on which I completely flipped my position on a key judgment call. Elie and I came in having read all the literature we could possibly find evaluating microfinance programs, and we’d concluded that there is reason to believe that microfinance helps people – but the evidence isn’t nearly as compelling or rigorous as generally believed. We argued that there are just too many questions about when and how microfinance helps people – and that none of the microfinance organizations we’d evaluated were able to give us good information on when and how their programs worked. We therefore preferred to go with something simpler, more concrete, and more tangible: KickStart, which markets a water pump that directly improves farm production.

Greg and Tim agreed with us about the problems (and how little is known) with microfinance, but challenged the idea that a pump is more reliable just because it is more tangible. Both are convinced that there is not enough available credit in the developing world (something I don’t have as strong an opinion on); and they pointed out that the mere fact someone repays a loan generally tells you they used it for something reasonable and presumably life-improving, even if not to create a thriving new business.

I found their arguments compelling. We all hesitated to grant a microfinance program because we don’t have nearly enough information about any of them, and they – of all organizations – should be able to provide hard data, since they essentially are banks. But ultimately, we went with Opportunity International, which had given us the best picture of what was going on in their programs. This is a cause we need to do a lot more work on, but we all agree that it’s best to give today (based on a best guess) and keep learning – not hold our charity until we’re 100% confident.

Other causes: coming soon

Despite the wish to give today, we simply need to do more work in Causes 3 and 4, so we put those off.

People get tired and cranky

Lucy and Tim were sick of jabbering, and asked that we cover only the things we had to cover for the rest of the meeting. So we skipped the Progress Report on Year 1 (good thing, because the attention we got later in the week has altered the report substantially), approved the 2007 budget retroactively, and pulled up our salaries for approval through the next Board meeting. Lucy said she would approve these salaries only on condition that we commit to discussing (at the next meeting) what metrics GiveWell will use to evaluate itself. She is insistent that we apply the same high standard of metrics and evaluation to ourselves that we want to apply to other charities. She is right, and we are working on it.

The moral of the story

Deciding where to give was exhausting, demanding, and affected significantly by the presence of different perspectives, from Greg’s financial/economic intelligence to Lucy’s and Tim’s expertise and experience. My #1 thought coming out of it all is that we need more of these heated, diverse conversations, and we need them to be happening between more different kinds of people and in more public settings. Making the world a better place is not easy; it deserves to be argued about as intensely and critically as everything else important that we do.

The things I’m proudest of about this meeting:

1. That I changed my mind on two key issues (and ended up switching my vote on 2 of the 3 causes). I feel that it always takes vigilance to keep an open mind, especially when arguing over something I’ve been working on and convincing myself of for months, so it’s always good to be monitoring whether I am operating in a manner consistent with open-mindedness.

2. That Board members were able to challenge our decisions in an informed way. The Board didn’t have access to any special materials (except on global poverty, which isn’t on the website yet but will be). They just read givewell.net , and doing so allowed them to understand the key issues enough to truly apply their own critical thinking. That’s what GiveWell is really about.