6 months ago, GiveWell pledged to give a $25,000 grant to the best organization we found in each of our five causes, and we’re going to follow through. But, I wish we didn’t have to.
We recognized that donors are more likely to trust us for some decisions (comparing organizations with similar goals) than for others (for example, trying to compare the value of saving a child’s life in Africa to that of helping improve a child’s education in the New York City). That’s why we divided charities into different causes, with the aim of giving one grant in each cause.
Having learned what I have about organizations that aim to raise incomes in New York City and those that aim to save lives in Africa, I would not donate to any of our applicants aiming to raise incomes in New York. I’d donate instead to Population Services International, our top choice for saving lives in Africa. I have two reasons for this:
- The relative cost of the programs.
- The logic underlying the approach.
1. Relative cost. The organizations we recommend – the ones in which we have the most confidence – spend $10,000-50,000 to help an unemployed/under-employed person become self supporting. For $50,000, I’d expect to save at least 50 lives in Africa by donating to Population Services International. There are reasonable arguments in favor of improving lives in the developed world – there’s value in helping people “get over the hump” that’s preventing them from living a fullly enabled otherwise happy life – and our “lives saved” number is of course a simplification that leaves out a lot. But these are not in the same ballpark. I think that the ratio of people helped (10-50:1) is just too much.
2. The logic underlying the approach. I don’t understand the “theory of change” (or, the basic logic behind the intervention) that explains the outcomes that Employment Assistance charities achieve. Are some basic soft skills (resume writing, interviewing tips, etc.) preventing their clients from finding employment? Do their clients need specific vocational skills (computer technician, environmental remediation technician, etc.)? We’ve wondered if it makes more sense to run job-training programs or just give the people money to support their basic needs, or otherwise, provide directly for their needs. In Africa, I understand the logic behind the interventions much more clearly (for example, selling insecticide treated nets that prevent mosquitoes from biting children, and thus prevent the transmission of malaria).
I end up feeling like a donation can make a huge difference in many people’s lives, with high confidence (through our top Cause 1 organizations) – or might make a difference in someone’s life, with low confidence and an unclear understanding of the logic (through our top Cause 5 organizations).
Our aim at GiveWell is to help donors find great charities to donate to by identifying the best charities in a cause (in which we can compare apples to apples). Our goal isn’t directly to choose their causes for them because the choice of apples versus orgnges is often a question of personal taste. But we hope that as we show donors more concretely what they get for their dollar, that will affect what causes they choose too. Speaking very broadly and roughly, imagine that for $25,000 you could save 25 lives in Africa or help 1 person get a better job in New York City. I know how I’d choose. What would you do?
Comments
I think now you’re getting at the heart of the issue of effective charity.
My current thinking is similar to yours. I think third world/African health initiatives probably offer much more bang for the buck than charities addressing problems within the U.S.
I’m not sure you’ve perfectly captured the math – I’ve read your piece on PSI and you’ve got a lot of estimation in there, at least some of which I think is off. But you’re probably in the right ballpark. I haven’t looked all that closely at the pieces about NYC organizations, but I’ll assume roughly the same holds there.
But even if we assume that the African health estimates are quite optimistic, and the NYC education estimates are quite pessimistic, it seems that reasonable fudge factors we might apply would still yield the same result – African health charities will likely impact more lives, more deeply, per dollar spent. This squares with my intuition – a dollar spent in a country where incomes are perhaps $1-2 per day will go much further than a dollar spent in the U.S.
So, should donors direct most/all giving to third world health (and similar causes) and avoid charities doing work in the US?
That’s a bit hard to say. It appears donors do NOT behave this way. Why not?
That’s a question with a very complex answer, IMO. And I think digging into that answer (and related questions and suggestions) is potentially as worthwhile as GiveWell’s current focus (which appears to be ranking charities within giving sectors).
Some reasons are practical. I think some donors are skeptical that charity in Africa really works. Personally, I think in most cases, well-directed charity in Africa *can* have a big impact, but I understand how others can hold a different opinion. GiveWell can be helpful here, in spotlighting charities that appear to be effective.
Other reasons relate to ways in which humans think. There’s an interesting article here:
http://www.edge.org/3rd_culture/kahneman07/kahneman07_index.html
It’s quite long, but scroll down close to the bottom, to the area just about “Session Six”.
There are two very interesting bits there:
The first is based on what appears to be a survey of how much people would pay to prevent a certain kind of environmental problem that kills birds. When the problem is described, people *are* willing to pay to save the birds. But the amount they’re willing to pay has only a minimal relationship, if any, to the number of birds saved. They can visualize the dying birds and want to save them, but they’re only willing to pay about the same total amount, regardless of whether 2,000 or 200,000 birds are at stake.
A bit further in the text, there’s some commentary (without clear data support) that “Save the Children” type programs are effective (in attracting donors) because they focus on an individual rather than a group.
Or, quoting Stalin, “a single death is a tragedy; a million deaths is a statistic”.
So, people don’t always do a good job of thinking through the numbers. It may very well be that a given donation can affect 50 lives in Africa versus only 1 in NYC, but I think many donors aren’t thinking along those lines.
There are other issues at play as well. I think the concept of tangibility is very important to charity, and further, that tangibility itself is difficult to define. Writing a check to a charity doing work in distant lands feels (in general, to me) less tangible than any number of charitable options (including volunteering) that are much closer to home. Logically, I suspect that the check will do more good, but simple logic is not the only thing at play. OTOH, I agree with your sentiment that, location aside, a clear objective of providing bed nets or other physical interventions (medicine) to poor nations feels a bit more tangible than the NYC education programs you describe.
Overall, while I think these issues are complex and not easily solvable, I think a site like GiveWell can be helpful. But I think you should consider broadening your mandate to include addressing these higher level issues (such as the relative merits of African health versus NYC education, donor motivations and decision making processes, and so on).
Also, I’d like to throw in some meta-commentary.
It appears you now have a rather different opinion of the relative merits of broad charitable causes than you did 6 months ago.
There are other signs that your opinions have evolved. Holden’s Dec. 28, 2006 post on this blog puts the cost of saving a life via one bed net program at ~$200 per life saved. Now you’re talking about ~$1000 per life saved (for a somewhat different/broader program, IIUC). That’s a pretty big swing – downgrading your estimates of effectiveness per dollar spent by about 5X. Now, the ~$1000/life estimate is yours (I think) and the $200/life estimate was Holden’s, and there may be other reasons why comparing these estimates is not an apples-to-apples task. Still, I think it’s notable.
The reason I’m pointing these two issues out is that I think a lot of what is said on this blog comes across as pretty critical of others in the charity field. I certainly don’t think that the foundations are all perfect. But I do appreciate the complexity of the task at hand (figuring out how to, well, “give well”). Perhaps the fact that your opinions have apparently changed substantially in the last 6-12 months might suggest that they could change quite a bit more in the months and years ahead, and that should be kept in mind when commenting about others.
I like the fact that you’re challenging existing practices. Doing that requires a certain degree of confidence in yourself, and your ideas. But I think perhaps the tone that GiveWell projects should also acknowledge that you probably don’t have it all figured out yet.
(And just to be clear, I certainly don’t have it figured out either, and I think GiveWell’s efforts can be helpful to me in the advancement of my own thoughts and opinions on charity.)
I’m not sure that Givewell’s hyper-critical style is the most effective, but I’m very confident that they have not implied in any way that they have it all figured out. They have emphasized at every opportunity that this is a hard problem and that they are doing research because they don’t know what to do. If they didn’t expect to learn they wouldn’t be doing research. Large changes in their opinions are strong evidence that their inquisitiveness and desire to learn are sincere.
Just not knowing everything doesn’t take away a person’s right to criticize. It’s easy to see when a particular person or group is applying less developed thought processes to a subject than you currently are. Top physicists still don’t know how the universe works in perfect detail, but if you are selling a perpetual motion machine I don’t hold their ignorance against them when they criticize your claims.
Michael – I think it’s a matter of degree.
While there are probably folks in this field who are roughly the equivalent of perpetual-motion machine salesmen, I suspect that there are also a lot of other conscientious folks applying deeper analysis.
Elie: Isn’t wishing you didn’t have to here equivalent to wishing you didn’t have to pay for goods and services in normal life? I think that African children need money more than the waiters in a restaurant, but I still leave a tip. Mentally, that’s labeled “fee for services”. Likewise, for you, $25K donations to “best in category” charities are fees for services. They have provided good documentation and now they get paid.
Since your opinions have changed, maybe you can convince some of the people at Population Services International etc that your approach is better too. If so, they could continue to do what they are doing, but could also voluntarily direct donors who share your attitudes in your direction. That could be worth much more than $25K
For the most part, that’s why I try to help Givewell. It is emotionally motivating to contribute what I can to the solution to any real problem faced by people in a manner fairly independent of the problem’s scale, but I think with fairly high confidence that the efforts that I primarily care about have much larger expected benefits than do fighting malaria or diarrhea, almost regardless of how benefits are measured (assuming human motivations).
Fortunately, it looks to me like being “conventionally good” has large instrumental benefits in terms of being “effectively good”, even though the direct good accomplished by being “conventionally good” (say, donating to sub-optimal charities) is relatively small. The skills and interests that cause someone to effectively fight malaria or diarrhea and the contacts and reputation that you and Holden are likely to develop in the course of doing so seem likely to strongly predict your ability and eventual desire to contribute to my efforts.
In analyzing what types of charity to give to (as opposed to the specific charities within those types), there are a lot of potentially game-changing assumptions. People may think that their own assumptions are universal, but I doubt that is the case. I’d imagine that there are some huge assumptions that I make that are different from others that I’m not even aware of, but here are a few areas that I *am* aware of that could be key:
1) Should we focus on things that affect the individual, or on political change that would (maybe) cause governments, with their vast resources, to address the problems? The latter approach seems attractive, but at present, I lean towards Holden’s belief in avoiding political issues, for reasons to complex to address here.
2) Should we focus on long-term or short-term goals? Looking at long intervals, research into disease treatment and vaccination may be more important than spending money on current clinical treatment. Economic programs aimed at raising third world incomes may have more long term impact than programs that focus on short term material aid (i.e. giving a man a fish versus teaching him to fish). Yet again, for reasons to complex to address here, I mainly lean towards GiveWell’s focus on application of current medicines to diseases now.
3) Give globally versus give locally? This is sort of the topic of the above blog post. An African cause *may* be able to stretch a dollar much farther than a U.S. cause. OTOH, most of us donors have very little direct experience with Africa’s problems, and it is entirely possible that many approaches we would support are in fact, not very effective, or perhaps, in the worst case, counter-productive. Perhaps our greater understanding (due to proximity) of local issues allows more effective giving locally than internationally. Having said that, once again, my instincts are that third world health causes *are* a better target for donations. Again, this could merit a deep separate discussion.
4) Very quickly, I think similar assumptions about relative long term impacts of education, economics/trade, and a free press can have a big impact on the discussion. Similarly, personal ethics and religious beliefs can have a major impact on how one weighs various causes and outcomes. And again, there are probably many other important areas that are not coming to mind for me right now.
I don’t think it’s necessary or practical for GiveWell to venture down every alternative branch in this thought process. But I do think there is value in discussing some of these issues a bit more openly. The process of arriving at the right conclusion can be as informative or more so than the conclusion itself.
On changing our minds. I think Michael put our position well. The best that Holden and can do is to tell you what we think, and why we think it while keeping in mind that we may be wrong and we have a ton more to learn. A year ago, we knew a lot less (virtually nothing as we were both still employed in the finance industry), and Holden and I thought that bed nets were a more cost effective intervention. Since then, we’ve learned more, improved our understanding, and updated our estimate. I expect that this’ll keep happening as we keep learning.
On wishing I didn’t have to give. I understand the value of this $25k and what it gave GiveWell, so of course, this grant does a great deal more than merely supporting an organization in our Employment Assistance cause. This post reflects my thoughts on giving to support one goal vs another. Knowing what I now know, I’d rather donate to Population Services International or Partners in Health over any of our recommended organizations in Employment Assistance.
Phil’s questions. Phil you lay out a series of questions (affect governments vs help individuals, give locally with higher confidence or abroad with lower confidence, etc.). I’m interested to hear your take – what do you think?
To be clear – I don’t have a problem with you guys changing your opinions. I agree with Michael – that’s probably a good sign that you are earnest in your endeavors. All I was trying to say there is that the opinion change that you’ve experienced might suggest that you take a somewhat different tone in addressing (and criticizing) others doing charity evaluations.
As for my answers to my own questions…
My opinions are rather tentative right now. Like you guys, I’m learning as I go.
But I tend to support direct intervention rather than attempting change at the political level.
I’m not entirely decided on long-term versus short-term. I guess that if clear short-term goals can be accomplished with relatively low chances of counter-productive long-term effects, then I’d favor short-term goals. I think this description applies reasonably well to many third world health initiatives.
I lean towards third-world giving versus giving closer to home, but I’m aware that a lot of giving is far less effective than the donors think (and some may have negative effects, long term). I think that, in a good case, third world health can have perhaps 50X or more effect than giving in the U.S., *but* you have to take that 50X and multiply it by some factor that accounts for poorly conceived programs, waste, and dependency created (with various ill-effects). Pulling a figure out of the air, let’s say that ratio is 0.50. That would still make third world health a very effective target for charity. But in a particularly bad case, I think it’s possible that the ratio would be negative. Obviously, that’s what we try to avoid by thinking things through and trying to give effectively.
Still, I’ve just touched on my views of these issues. Moreover, my views are probably relatively ill-informed. Like you, I read what I can (though I’ve done far less information requesting and spreadsheet analysis than you appear to have done). I have not visited Africa myself, nor had extensive contact with NGO folks (I’ve talked a bit to a couple of folks who’ve done NGO work, and tried to read some stuff, but that’s about it).
I like what you guys have done so far, especially with limited time and resources. You are closer than any other site I’ve located to my idea of an information resource for evaluating charity effectiveness. But I think you’re still missing a major part of the puzzle. You’re drilling down for a micro-view of problems. (Which charity is best, among the subset of charities that improve education in NYC?) But I think that misses the more important issue. (Is improving education in NYC a good use of charitable money?) Now, perhaps, you’re realizing that the second question is quite important indeed. I’d like to see you look at that area with as much rigor as you looked at specific charities.
One issue that I’d love to have a better understanding of is the history of aid (governmental and private) to Africa in the last ~50 years.
How much money has the West given?
How much of that has been truly charitable in nature, versus military aid and/or simply buying off a corrupt regime here or there?
Has the charitable aid been truly effective?
What lessons can we draw from past experience to guide us for the future?
I’ve read some stuff by Easterly and by Sachs, and they seem to be at such odds in evaluating aid to Africa that it’s difficult for a non-expert like me to parse their facts and opinions. Both seem to be driven by their own agendas. Indeed, I haven’t located a good source that feels neutral in analyzing what went right and wrong in the past, what we can realistically hope to achieve in the future, and the best methods for achieving that.
Elie, i think you’re right about the clear difference between these two causes, and the impact per dollar, or hour, or whatever input seems so staggeringly greater when trying to impact things like basic global health relative to training someone in the developed world to do one job or the other.
thats why, i was thinking about the causes or the places to donate, it would be to line up here are all the reasons people’s quality of life is poor (not necessarily in lives saved sense, a bit more subjective than that), here are the rough cost to improve it and basically divide the quality of life impact with the cost, and knock out the best ‘deal’ with my dollars. plain and simple, thats bang for your buck.
however, the reason why I am attracted to things like raising incomes in NYC is because I am selfish… (did i just state that explicity?). No, really, I am… because I want to live in a better community. Raising living standards of those around me in NYC will make me better off, maybe not directly or instaneously, but the end goal is to make my community better. That in turn makes my life better. I wonder how much different that is than giving to the Stamford Center for the Performing Arts so that we get Aretha Franklin to come instead of the Royal Scots Dragoon Guards Band (no joke) to come to Stamford…? In both cases they are different attempts at making my community better and making my life better, through more and less direct methods. That isnt a bad thing, but its clearer in my mind when i give to SCA or to NPR, than giving to raise incomes in NYC…
If I give, instead, with the best bang for your in mind it doesnt improve my quality of life. What is the likelihood that a child with a bednet is going to ever impact me? essentially nil. Given that Im blessed with resources to be able to help others in need, I should use that most effectively, even if I get nothing out of it. ever. Thats because real giving shouldnt be about making my life better, it should be about making others’ lives better the best way possible, whoever it is.
It’s sometimes important, with top academics like Easterly and Sachs (not to mention Howard Gardner and Arthur Jensen) to look very carefully at what they say in order to recognize that they don’t actually disagree as much as emphasize different features of the situation. For the most part, when you look carefully, Easterly thinks that solving public health problems is very cheap relative to the amount of benefit, but that we don’t know how to effectively bring about economic development in most cases.
Here’s a bit of an e-mail thread that summarizes my take on Easterly’s take.
>
> William Easterly says that $2.3 Trillion has been spent on development aid in 50 years.
> Looking at http://en.wikipedia.org/wiki/Image:Life_expectancy_1950-2005.png , and thinking bit
> 5 years of life expectancy for 2 billion people seems like an upper bound, and it seems safe
> to say that the majority of the benefits of giving are reflected in life expectancy, suggesting
> that average third world giving has benefits no greater than 1 QALY / $230. If military is 3/4ths of that aid
> historical aid might average $60/QALY, ridiculously cheap, and probably about optimal so far from the margin, so the
> historical size of aid expenditure is basically no evidence that aid is ineffective, though it may indicate that Sachs’
> hopes are somewhat overly optimistic. The world is ridiculously big so expect fixing it to be very expensive.
This is an interesting high level analysis.
It is also worth considering that we have only given $23 per recipient per year (for 2 billion recipients times 50 years). What were we expecting for that amount? Singer made this kind of point in the lectures as a riposte to people like Easterly and I think it is a good one.
Obviously the additional costs to ‘fix things’ will depend on how efficiently they are allocated. I can easily see and extra well spent $30 billion P/A doing as much as a poorly spent $300 billion P/A. Sachs is probably estimating at the efficient end of the spectrum and presumably actually knows some good efficient uses. He may be right even if we actually end up using ten times as much to do the job.
>
> Easterly thinks that in general nothing has been learned. He convincingly shows that World bank, IMF, UN, and other
> NGOs make almost identical statements about accountability, poverty targeting, etc now
> and 40 years ago or more. This suggests that they think efficiency has been low.
> Easterly does consider healthcare efforts useful.
> He advocates paying kids to stay in school, e.g. Progressa, just what Vinod Banerjee says is the most inefficient
> of a long list of commonly promoted 3rd world education programs
> He says HIV drugs have cost of $300+/life year but also cost $1200/life year for distribution/application.
> This doesn’t sound factually at odds with Lomborg, yet Lomborg makes this priority #1. Why?
> Generally thinks health care has been the major success of development.
> http://www.cgdev.org/section/initiatives/_active/millionssaved
> Much depends on government cooperation to help promote open culture e.g. Uganda and
> HIV
> Sadly, I don’t know how to set a lower bound for the historical efficiency of aid.
The official Copenhagen Consensus book (which I’ve borrowed from the FHI library) says that the estimates it worked with predicted averting 30m new infections by 2010 for $27bn. Roughly $1000 per averted infection. They seem to be advocating a program with a range of measures for reducing spread, rather than spending money on treatment. If even only QALY is lost per infected person, then this is cheaper per QALY than the above estimates (which are presumably for treatment).
My gut tells me that relatively direct attempts to improve third-world economies via first-world aid are not likely to be very productive. Large sums of external cash will likely distort internal economies and lead participants to focus on ‘harvesting’ that external cash rather than genuinely growing the internal economy.
Yes, I’m aware of micro-lending, and it probably does some good. But the concept seems oversold to me. In any case, if the amounts needed are so small, and the repayment rates are so high, it would seem like there is a relatively small amount of total capital needed for this sector, and that the recent surge of interest in micro-lending has probably filled the most urgent needs.
In any case, I’m actually pretty interested in reading what, if anything, GiveWell has to say about this when they do their main report on global poverty. The Sept. 15 blog piece Holden wrote on micro-lending was, I think, the only article I’ve seen that asked hard questions about the evidence that micro-lending really works.
I tend to think that economic growth will largely have to come from within. The most effective ways to speed up growth would be better government and legal systems, but I think those are hard for us in the West to influence. I think we’re likely to be more effective in helping with health care, probably education, agriculture and nutrition, and possibly some other issues along these lines, and in turn, these will probably have an indirect but positive economic effect. But again, I’m open to being proved wrong. 🙂
As for Sachs, Easterly, and the Copenhagen Consensus – IMO, each has an agenda to push that gets in the way of objective data presentation. (I don’t think I’ve read Sachs’ book, but I’ve seen some of his writing.) That’s why I’d like to see GiveWell tackle the issue – even if it’s just a matter of having a research assistant spend a few days pulling together data from various public sources and put them into a nice report or web page.
so is it impolitic to say I care more about people in my home area than the people in Africa? For smaller donors, charity often begins at home, and the statistics are irrelevent.
About this:
“William Easterly says that $2.3 Trillion has been spent on development aid in 50 years.
> Looking at http://en.wikipedia.org/wiki/Image:Life_expectancy_1950-2005.png , and thinking bit
> 5 years of life expectancy for 2 billion people seems like an upper bound, and it seems safe
> to say that the majority of the benefits of giving are reflected in life expectancy, suggesting
> that average third world giving has benefits no greater than 1 QALY / $230.”
I question the assumption that appears here that the aid money is the cause of the added 5 years of life. I think this issue of the difficulty of establishing causation is not reflected in the discussion today. The more global the intended change and the further out in time it is expected to be, the more difficult the causal arguement. These issues of effectiveness have been addressed particularly by evaluation researchers for decades, and I don’t find this history reflected in the discussion. While having a discussion like this in public is great, these issues are not at all new.
Andrea – if the effectiveness issue has been well examined in the past, the results seem rather difficult for myself (and I suspect, others) to find.
Can you point to sources that you’d recommend for reading up on effectiveness-related questions?
I’d like to comment on this bit in the original post:
I am just beginning to learn about evaluation myself, but here are some websites of evaluation organizations that attempt to facilitate collaboration among evaluation professionals around the world and that collect links to resources.
http://ioce.net/index.shtml
http://www.campbellcollaboration.org/index.asp
http://www.eval.org/
Couple of books:
Evaluation: A Systematic Approach by Peter H. Rossi, Mark W. Lipsey, Howard E. Freeman
Qualitative Research & Evaluation Methods by Michael Quinn Patton
Responsive Evaluation: New Directions for Evaluation (J-B PE Single Issue (Program) Evaluation by Jennifer C. Greene (Editor), Tineke A. Abma (Editor)
The philanthropy world has a long history, is decentralized and has so much at stake, that it makes sense to me that it would be difficult to collect all the information in one place that one might need to make donation decisions, and that it would be difficult to convince those raising money and providing programming to disclose all kinds of data to people new to the field just because they ask. I don’t mean that this attempt isn’t fascinating or that it might not lead to some really cool new ways of thinking and evaluating but, still, trying to determine what the facts are is always incredibly complex. I note a tendency in the discussion today to make assumptions about facts, i.e., using hunches and making up numbers for discussion purposes,that worry me a bit. That’s all.
Sorry, this is the quote I was referring to:
I don’t understand the “theory of change” (or, the basic logic behind the intervention) that explains the outcomes that Employment Assistance charities achieve. Are some basic soft skills (resume writing, interviewing tips, etc.) preventing their clients from finding employment?
Kudos to you for trying to improve the nonprofit sector. I am poking around on your website for information about board service. How are you measuring the effectiveness of a nonprofit’s board? It is a critical part of any successful nonprofit. It seems your work is focused on service delivery. What about governance, leadership and oversight? If you haven’t I urge you to get on the board of a nonprofit and experience that hard work first-hand.
Another, more pertinent, resource on evaluation in the philanthropy context:
Foundations and Evaluation: Contexts and Practices for Effective Philanthropy, eds.
Marc T. Braverman, Norman A. Constantine, Jana Kay Slater, August 2004
Andrea – the resources you’ve listed seem centered around ‘how to evaluate’. I don’t see any that look very promising in terms of actual evaluations of charity effectiveness (i.e. lists of which charities do things better, or even, which causes generally have the most bang for the buck).
The last book you mention (Foundations and Evaluation: Contexts and Practices for Effective Philanthropy) does seem interesting. I looked it up on Amazon. It seems to be about best practices for evaluation, which might be a useful topic for this site, and for me personally.
It has one customer review (5 stars). It’s kinda pricey at $67.00, and I worry that it might just be a series of essays that take a lot of pages to say “Evaluation matters, do it.”
Still, I might check it out. Have you read it personally?
Phil, I haven’t read it yet, but I am actually in the process of ordering it as we speak. I got the reading recommendation from someone in the field of evaluation when I showed her this GiveWell website yesterday. You can read excerpts at http://www.josseybass.com/WileyCDA/WileyTitle/productCd-0787970778.html
This whole issue is very difficult and complicated. Each charity would have to have the money and other resources to evaluate their efforts (which is a major obstacle, as Holden has written about), then someone would have to have the money and other resources to take those evaluation results and put them in a form that can be compared–possibly in many instances impossible, due to the conceptual limitations of measurability–, then make them available. None of this directly supports the mission of charities, many of which are scraping by as it is. Maybe this is what GiveWell can work on doing—translating all the work of evaluators into a form that donors can use.
It’s just very important not to reinvent the wheel.
I’m still digesting this discussion, will come back and post some thoughts soon. In the mean time, does this address your question, Phil:
http://urban.org/center/cnp/Projects/outcomeindicators.cfm
I think it’s a big mistake to assume that those who’ve been working in the sector haven’t been addressing evaluations and issues of efficacy.
I don’t think it’s a question of reinventing the wheel. Rather, I think we’re living in an age where we don’t even have wheels yet.
OK, I’m stretching the metaphor a bit. 🙂
In any case, I think people may have considered *how* to do evaluations in the past (not necessarily very focused on public charities, though). But I don’t think there’s been a whole lot of effort to actually DO the evaluations of charities, and publish that information in forms that are easy for the public to access and use in decision making.
I agree that many areas are not readily subject to measurement. But some are (at least, roughly), and I’d rather see a partial analysis (that acknowledges its limitations) than no analysis at all.
To me, spending 1-10% or so of charitable donations on measurement and evaluation is a fairly trivial cost, in the greater scheme of things. Money well spent can be many times more effective than money poorly spent.
My post immediately above (#25) was in response to Andrea.
Amanda – I’ll take a look at the site you link to. It looks deep, hopefully it’s useful.
I can certainly believe that at least some charities evaluate their projects and try to be effective. The problem is that, from a donor perspective, such information is very hard to obtain and process. Most charities will, of course, say that they’re doing great, important, effective work. Some charities may actually have internal data to prove that. But in my experience, it’s difficult for a donor to find such information at all, much less presented in a way that can be easily digested, and that allows useful comparisons across charities.
As Holden has repeatedly pointed out, the existing charity guide type services (except GiveWell) primarily focus on collecting and presenting very basic information, and to the extent that they rate charities, it’s largely based on what Holden labels the “straw ratio”. Holden thinks that ratio is pretty much useless. I think it has *some* value, but among the overall pool of reasonably financially efficient charities, that it’s not very useful in highlighting the most effective.
Charities use the “straw ratio” because it’s easy in that it doesn’t cost them any money to produce it. I want to emphasize, evaluations are extremely expensive, and it’s easy to throw out numbers like “1 – 10%” of charitable donations for use in doing evaluations, but that number has no basis in actual practice, and many donors themselves don’t want to pay for other-than-mission expenses.
Please check out this link:
http://www.ppv.org/ppv/youth/youth.asp
PPV did a huge randomized national level study of Big Brothers Big Sisters that was excellent, which BBBS has links to on it’s website, and you can see from reading about it what a major undertaking such evaluations are.
Phil –
Coming from the perspective of a consultant that helps nonprofits conduct their own evaluations, I would agree that they are reticent to make the results public – with good reason (and Give Well is a great example of why). When I work with nonprofit professionals, I try to emphasize the creation of an internal feedback loop – the ability for an organization to constantly monitor and improve its services. When donors push evaluation in order to make critical funding decisions, organizations have every reason to be fearful. They already know that they are not perfect, that they sometimes miss the mark , that they could be more efficient. Like any org (for or non profit) improvements can be made.
You may say that funders will not be so critical as to pull their dollars at any sign of imperfection, but if ineffective measurement tools such as Guidestar have impacted giving so strongly (to the extent that orgs are hiding their admin costs), imagine what more close scrutiny will do to the struggling organization that hopes to improve it’s outcomes?
I realize the fallacy in the logic – orgs shy away from transparency in case donors become aware of program failures, but they still expect donor dollars in order to improve their services…
A real dialog between donors and nonprofit management is necessary – what are realistic donor expectations? How can an org meet these expectations over time?
Some foundations are actually engaging in this, and they are assisting orgs to build sustainable evaluation processes. Apparently, Give Well expects all this to be made available to the general public – I hope you can see why that might not be welcomed by many nonprofit organizations.
As for the apples and oranges question – I worked with an organization in India that helps grassroots nonprofits become more effective. It also assists with international funding agencies and individual donors to be more effective in their giving (dasra.org). Working with them, I talked to many local organizations that had received international funding – and they had a lot to say. International funders often fail to take local political structures and environmental realities into consideration and thus can be terribly ineffective. There has been a lot of discussion about how disaster and relief agencies handles the 2005 tsunami.
I think it’s very likely that ‘bang for your buck’ looks a lot better when you can’t see the effects first hand.
In addition, what about a discussion of civic responsibility? Do I, as a local resident of Chicago owe more (or have more responsibility) to my local community (I certainly utilize local public services) than I owe internationally? I honestly don’t know the answer to that (as someone in Bangladesh probably sewed my clothes). But when discussing philanthropy, are we even thinking in terms of responsibility or are donors just benefactors? If so, then spend your money where you will – it’s a personal choice.
I realize the above paragraph is more about ideology than effectiveness, but people are making ideological choices about where to give their money. This concept of more impact could be taken to the nth degree – and at some point we’ve got an iron cage on our hands (any Weber fans out there?)
Attempting to measure impact of necessity requires the ability to reduce things to countable units which is famously problematic. Social science–and this is what we are talking about when we talk about measuring the impacts of attempts to improve the human condition– is so fraught with methodological difficulties that it is not even considered a real science by some people. This is really what you have to grapple with when you talk about measuring impacts—the whole history of thinking about science, which stretches back centuries. That it hasn’t been done as well as most of us would like up to now, does not imply incompetence, ignorance or bad intentions.
Re: Where to give (US or international)
I agree that there are some philosophical questions at play here. It’s actually a rather complex, values-driven question, IMO. I think there’s value in discussing the issue. For now, GiveWell has sort of punted on this issue – it has chosen its mix of 5 target areas without much discussion of why they were chosen, and is then comparing charities within each area, rather than across areas. I think that was a reasonable way to go to get started (though I might have preferred a split more along the lines of 1 NYC/2 US/2 Africa).
Anyways, if and when they wrap up their first round of analysis and begin planning for next year, I think it would be useful to consider higher level questions, such as how to balance donor interest (for various reasons, some of them sound, IMO) in US charities with the likelihood that African charities achieve more bang for the buck (lives impacted per dollar).
Re: Measuring effectiveness
I can understand why charities are not eager to fess up to failures publicly. I don’t necessarily think it’s reasonable to ask them to do so. If Ford manufactures a car that’s a lemon, they may know it (after the fact) and analyze why that car turned out to be a lemon. I don’t expect Ford to be eager to share their analysis publicly. But I *do* want to be able to read Consumer Reports or something along those lines, doing independent research and coming to the conclusion that that car is a lemon.
Evaluating charities is a bit different from evaluating cars. C.R. can buy a car through a dealer, and with that car in hand, do its evaluation (though they’re also helped, I think, by the feedback their own readers provide on reliability). C.R. does not need Ford’s cooperation to evaluate them. (By the way, I don’t want to come across as picking on Ford too much – I was happy with my last Ford – I’m just using them as an example.)
GiveWell would have a very difficult time evaluating charities without the cooperation, to some extent, of those charities. But there are a lot of charities out there, and if only 40% of the charities in a given area are willing to cooperate with GiveWell, then I’m still interested in reading GiveWell’s analysis of those 40%. If GiveWell becomes enough of a player, then hopefully that 40% would become 70%, but there’s a lot of value even if the number of cooperating charities is only 40% (or even 20%).
Simple solution: Employ NYC people to help you save lives globally. Then not only are able to support themselves, but through working on a bigger cause they gain even more of the most valuable commodity: Dignity.
Andrea: Do you know what “upper bound” means. It’s polite not to question an assumption that a person doesn’t make.
Jimmy: Can you think of any third world poverty reduction or public health job that unemployed New Yorkers are likely to have a comparative advantage in?
Michael Vassar,
But I didn’t question any assumptions Phil didn’t make. Read his posts.
Your “upper bound” qualification allows you to wiggle out of the fact that your post was full of speculation as to real costs and effects. I understand that people use imprecise or speculative numbers for purposes of argument, but at some point the numbers need to be real, or your conclusions get too fuzzy and out there. I believe that the difficulty of establishing real numbers is the crux of the entire problem posed by this organization, has been grapple with by many people long before this organization was conceived, and that familiarizing oneself with this long history would keep one from naively assuming these questions are at all new. In one of Holden’s essays, he seems to make the claim and what distinguishes him from all the other folks who have ever thought about donating, is that he believes improving people’s lives is hard. I supposed I should cut him some slack, but this particular assertion of his is absurd, and I think it turns people off to the whole project. I know he’s a smart guy, but this assertion is astonishingly naive.
Andrea: It’s not “wiggling out”. Setting upper bounds and lower bounds leaves us MUCH more informed than we were before doing it. There’s a whole field called “computer science” that does little else. Quantitative thinking with imprecise numbers is spectacularly valuable in a variety of domains. This is not a claim that I am willing to debate though, any more than I want to debate the value of empiricism.
Michael, I don’t agree in the least that wild speculation, particularly speculation with enormous numerical ranges thrown in, necessarily leaves anyone more informed, even if you try to bully with the use of quotation marks around the words “computer science.” Knowing what’s real, as opposed to speculating about an unknowable future, is extraordinarily difficult. Futurists can be annoyingly facile, and the inability of anyone to present evidence that their predictions are wrong lets them off the hook of having to defend what they say. Not everything is defensible, just because you say it is, and refuse to debate about it, or because you define disagreement with you as impolite.
Thanks for all the comments. It’s hard for me to tell whether this is a temporary spike due to all the press, or whether we’re reaching the point where it might be appropriate to bulk up our comments feature (like using vBulletin instead, as we now are for the main givewell.net). I lean toward the latter, because this is a mostly thoughtful conversation going on between a few people, and I hope you’ll all stick around.
I see a few major discussions going on, and I’ll give my quick thoughts on each.
1. Evaluating within causes (apples-v-apples) vs. between causes apples-v-oranges)
Our opening strategy was to pick a few different fruit types, pick our favorite of each, and do as much analysis as possible of “what you get for your dollar” to allow more informed comparisons across causes. We never intended to “punt” or stay silent on the apples-vs-oranges question; blog posts like this one were always part of the plan. But blog posts like this one aren’t possible until we’ve looked very closely at the organizations in question and come up with estimates of “the best you can hope for your dollar” in each. All in all, I think this is a pretty fair and reasonable plan: do the intense analysis needed to name our favorites within causes, and use that to clarify the debate across them. That said, there are some possible improvements.
First and foremost is that we can have more “layers” of comparison. As you can already see on our current site, we have some splintering within causes (like Cause 5); yet we also have some strong thoughts across them. Addressing this issue is basically a matter of scope (covering more causes) and web design (laying things out to make clear that there are several layers of decisions that progress from more to less philosophical).
Then there are a couple of Phil’s ideas. We could try to weigh “goodness” vs. “donor appeal” for different causes. This is something we are already implicitly doing, but in a different way from what Phil implies. We are not using psychological studies; I think hyperfocusing on things like “Studies show that people give less when they are thinking numerically” is largely a mistake and a waste of time, for reasons I’ve gone into briefly before and may go into more later. Rather than looking at people “probably wilL” prefer, I want to look at what they do prefer. The GiveWell business model is to choose our causes through a combination of our personal interests and passions and the market’s demand. We already did this in a loose and informal way for year 1: we did Cause 3 because it was particularly important to one of our major donors, whereas the decision to do Cause 2 was largely because we personally wanted to. In the future we are going to get more formal about this, essentially letting our donors “bid” to determine the direction of our research. Letting the dollars speak – specifically, the dollars that are likely to be influenced by our project at all – seems smarter than theorizing about what people prefer and why they prefer it.
We could also look at the history of aid in different regions. I am not wild about this idea but I will think about it. We are trying to figure out what the *best* approach in each cause can accomplish, not what the average can. There is obviously a relationship, but with all we have on our plate for making progress on the former, I’m hesitant to spend much time on the latter.
2. How much can or should we expect charities to share about their outcomes? Will donors flee at the first sign of imperfection?
Early in our development, the picture is probably going to be as Phil describes. Charities are unlikely to share negative information, so instead we’ll just end up evaluating a limited pool, which is better than evaluating nothing.
As time goes on and if/when we grow, we hope to create an environment where openness comes before salesiness. We will reward charities that openly share their info above those that share nothing, even if those that share their info reveal that they are failing at their mission and need to keep working. I’ve even toyed around with the idea of “failure grants” to organizations that are willing to publicly disclose something that isn’t working and give a coherent plan for improving it.
As I’ve sort of written before, I think the idea that this environment will lower giving or punish transparency is overblown. People who have such a simplistic view of the world will probably ignore our site altogether, or blindly use our recommendations (which will lead them, too, to reward more honest organizations). People who bother to dig enough to see the truth will generally be the people who can handle it. Maybe I’m wrong. It’ll be up to individual charities whether the benefits of our endorsements and funding outweigh the potential costs of transparency. But it seems clear to me that the latter is better for society as a whole, and I prefer to push for it as hard as I can.
3. Should we use imprecise/guesswork-based numbers, or no numbers?
I’m not sure I understand this argument. It seems clear to me that we should use everything at our disposal, as long as we keep it in context. Ultimately NO relevant number is completely precise, but estimates and guesswork can help us compare things better than if we just took a wild guess. The important thing is not to confuse guesswork with precise calculation; that’s why we like giving ranges and putting constant reminders in our analysis that they are rough (and disclosing everything that went into them, so that nobody has to “trust” our numbers).
Andrea, I agree with you that Michael shouldn’t be a bully or refuse to be criticized. I don’t think that is what he is trying to do. I imagine that, like me, he is having trouble understanding what the argument is for doing zero analysis vs. potentially faulty analysis (as long as the potential flaws with the analysis are kept in mind and exposed to view, as they are for GiveWell’s analysis and not for any other funder’s analysis that I know of).
GiveWell has a strong preference for giving based on guesswork vs. waiting till all the facts are in. That’s because we believe in giving today instead of tomorrow. We’ll never have a perfect understanding or all the facts; we just want to do the best we can, which will often be very, very far from true understanding (but also very far from the way most donors make decisions today).
4. How could Holden and Elie think that their 26-year-old punk selves can add something to the experienced world of philanthropy? Do they really think they’ve thought of anything that others haven’t? Isn’t the mere fact that they think this evidence that they are naive idiots who should be ignored?
I’m kind of tired of answering this question. Do you want me to cite historical examples of young/punk outsiders’ changing fields for the better? It would be trivially easy. That doesn’t mean we’re going to do it. It doesn’t mean we’re not naive. But it seems obvious to me that when you see a problem, you should speak up, and expect all the “experts” to explain why you’re wrong – rather than just shutting your mouth and assuming the “experts” have it covered. Especially when we lack strong reasons to believe the experts are True Experts.
People have made blanket statements about the complexity of the problems we’re working on – statements I agree with – but no one has given any argument for why the kind of analysis we do (rough, guesswork-based, flawed) is worse than the kind of analysis other donors do (completely uninvestigative; focused on questions like “how much of my money is getting spent directly on activities?” which does, yes, tell me that they are implicitly thinking of helping people as something easy – an essential quality of anything where number-of-dollars-spent is more important than approach).
More minor points coming up in the next comment.
Phil: I think you completely misinterpret the “negativity” or “criticality” of our tone. You seem to think we are convinced that all foundations are dumb and incompetent and that we are awesome, or something. What we actually think is that we know next to nothing, have unlimited room to learn, and are just doing the best we can … and as I tried to make clear in the discussion on the post before this one, we don’t have a strong opinion on whether foundations are better than we are (we could see it going either way).
What I think is confusing you is:
RG:
“Often you have people who are in their 40s and have never been part of the labor pool. Or their last job was 12 years ago. We have to teach some of our trainees that you must come on time every day and that you have to call when you’re sick. We’ve had to tell people that you don’t curse a blue streak in a work environment (or at least not ours).”
Believe it or not, this is clearer to me than most of the “theories of change” we’ve gotten from our applicants. I’m guessing it’s because you are being more informal in your tone, and thus feel more free to give specific and vivid examples.
I think it would be clearer still if accompanied by more specifics – data on things like “# people who cursed in such-and-such context on entering the program, and # people who used curse words by the end”, and also just large numbers of highly specific stories (not the typical fundraising stories that focus on how someone turned their life around and attribute it all to the “magic” of the program, but instead stories that focus on very specifically what a person was doing wrong and how they learned to change it).
This is an ongoing conversation. The fact is that most of the people who fund these programs aren’t close to the ground and don’t understand them. That’s why I think it is productive to put our uninformed opinions out there and continue to work on learning and communicating how things really work.
You also point out that donations are a personal choice; I agree, and that’s why we separate apples and oranges in the first place (more on this in the above comment).
Jen: we really haven’t looked into evaluating boards at all. I’m interested in your thoughts on how we can do so. I am skeptical of the importance of boards, because it isn’t clear to me how important a body that meets a few times a year can be. I’d rather judge a mature organization on what it’s accomplished, anyway, than on who its board members are. But I’m listening.
Comments are closed.