• Inadequate evaluation is currently a far bigger problem than overly expensive evaluation.
• 10% is too much if equally good evaluation can be done for less, but it doesn’t strike me as too much in a cost/benefit sense. I would guess that truly meaningful evaluation can easily double or quadruple the effectiveness of a project (or lead people to move from a completely ineffective project to an effective one).

Holden and I are no longer on opposite sides - we agree that his “top charities are good bets” and not making his good-bet list “is not the same as saying that they are proven to be ineffective”(to quote him).

For my part I don’t see anything in the rest of your comment that I disagree with or that is related to the point we were discussing. Am I missing something?

Spending 10% overheads to run annual or program impact assessments is simply unacceptable for either a non-profit or for-profit enterprise. The dev sector has become too accustomed to accepting these ridiculous figures for field work instead of pushing back on organizations to spend 1-3% on impact studies, which YES, is possible because I conduct many of them every in Africa. Firms & organizations need to stop agreeing to pay consultants $500 - $700 per day and perhaps hire internally. In Africa, a firm can hire a full time employee to write, do research etc.. per annum for the same cost as one consultant-based field report which may or may not be written properly with acceptable M&E and social impact frameworks.

And as a social entrepreneur, I not only expect but hope that the “3rd sector” is going to be held to a standard of effective reporting rather than having to choose between either the non-profit or the for-profit sectors. If a business model claims social impacts as well as financial (income generation / livelihoods), then it should be prepared to quantitatively or qualitatively report on that impact. If the big boys aren’t prepared to meet this standard, I see us spiraling out of control with a negative impact on the sector as a whole.

We are a relatively new sector blending the dev sector as well new models of private, personal and donor funding. There are no organizations which monitor our activities (*not a bad thing) so our commitment to transparent and effective reporting standards must be paramount. We can not exist on buzz words alone!

Unfortunately, it seems like everyone is now a social entrepreneur. For now it is only in forums like this that we question and challenge each other. While I would also like to see more impact studies from Acumen and other funders, they are a relatively new organization in relation to large for-profits or non-profits. In time more lessons will be learned with both capacities and reporting transforming.
the

We need to learn from the mistakes of funding transparency in the dev sector, define and meet a high standard. http://blog.givewell.org/?p=493 (another Holden blog). If we don’t, I believe our community will eventually burn out and be considered another one-hit wonder by both the for-profit and dev secotrs, both of which continue to be skeptical about our long term viability.

Thank you for acknowledging that the organizations GiveWell rates as a “bad bet” might, in fact, be effective.

I’m heartened to see GiveWell hold itself to the same standard it sets for those it rates. I now trust that your top-rated charities are as effective as you say, and I hope GiveWell will continue to keep in mind that lack of studies supporting new strategies doesn’t mean that those innovations are not just as effective.

Best regards,

I believe that for an individual/casual donor who does not have their own research capacity, our top charities are a good bet and any other charity is a bad bet (which is not the same as saying that it’s proven to be ineffective).

Why don’t you simply adjust your conclusions to those supported by the evidence? The fair claim would seem to be that GiveWell has a list of organizations that you like best, and that givers would be unlikely to waste their money supporting them, but not that givers would necessarily do worse elsewhere.

Frankly, I think most people who give to the Acumen Fund and similar organizations are giving based on an appealing story and/or an impressive set of people, not on any observation of these organizations’ actual effects on the world. I would guess that for many donors, this video is as far as the “due diligence” goes.

Thank you again for an opportunity to expand my argument. I think I ought to point out that I don’t work for Acumen Fund. I admire its work and I’m passionate about social enterprise, so perhaps I’m not always as objective as some others might be. But I will try to answer your challenge fairly, as an interested third party.

Holden, I’m NOT arguing that “it is impossible for Acumen Fund to provide evidence of impact, even if it has a strong impact, given what it does.” I’m arguing that GiveWell’s measure doesn’t work in this case, and I’m arguing that this is because Acumen Fund’s business-focused process doesn’t itself produce the type of documentation that GiveWell seeks.

Yet, I do assert that “the effect is crystal clear”, and I appreciate your invitation to point to “any evidence of any kind, by any yardstick” to support that claim. That’s a very generous standard, and so I feel compelled to try to meet it.

In my previous comment I explained why the effect might be vague within the documentation produced by Acumen Fund: it’s veiled in the weaving of the value chain. Now let me explain why the effect is nevertheless crystal clear, using an analogy to illustrate.

Here in Florida, we had an advertising campaign that sought to deliver a message that was aimed at reducing teen smoking (The “Truth” campaign). This advertising campaign is strongly analogous to Acumen Fund’s focus on “distribution channels” for the bednet enterprise, and the aim of reducing smoking is strongly analogous to the aim of reducing insect bites. Surveys here immediately showed that teen awareness of the message increased following the advertising, just as sales figures immediately showed that many more bednets were reaching consumers. And we know from past studies that advertising and increasing distribution channels both often work and that when they do then we see the kind of immediate results we have here. Thus, we have a strong argument based on reason — an inductive as well as a deductive argument — that both the ad campaign and Acumen Fund’s strategy succeeded. Indeed, this is as we would expect, and the expectation is obviously why so many successful businesses engage in one or both of these methods.

Thus, I believe I’ve met your challenge to “point to evidence - of any kind, by any yardstick” that supports my claim that the effect is crystal clear: The long-proven strategy that Acumen Fund is bringing to the base of the pyramid, where it has been most lacking, is producing tremendous results.

Naturally, further study provided the causal link between the ad campaign and the increased awareness of the message, and still further study linked the campaign causally to reduced smoking, and we expect that eventually we will have the causal link between the ad campaign and reduced disease. Here in Florida, we have billions of dollars from a tobacco lawsuit to draw from to pay for those studies. But Acumen Fund relies on philanthropy, and so those future studies are less sure to happen. Fortunately, the business world understands, and thus we are seeing a revolution in philanthropy led by business that is focusing on social enterprise, and I expect that funding for the causal studies that GiveWell apparently relies upon will follow. Meanwhile, I hope and expect Acumen Fund’s “focus on evaluation” will remain primarily on identifying the best targets for its strategy, rather than proving what is plain to see.

And will we see less disease? Will it be proven that this reduction in disease corresponds to fewer insect bites? Will it be proven that bites were reduced by more bednets from the Acumen Fund partner? And will it be causally proven that Acumen Fund’s focus on distribution there resulted in more bednets reaching customers? Time will tell. And the same kinds of studies will be necessary to prove cause and effect for increased employment, and for reduced poverty. Meanwhile, Holden, it appears that it is clear enough for most reasonable people.

Now, will you answer my challenge: Can you point to evidence - of any kind, by any yardstick - for a claim like this one?: GiveWell’s failure to find a track record for Acumen Fund’s assertions of success “shows you how much more you can accomplish with your gift” somewhere else.

I thank you again, sincerely, for your work and attention to my criticism. I think I went too far in saying that it was “poorly done” on your part to conclude that lack of evidence is evidence of the contrary (better places to send contributions). In hindsight, I think you are wrong, but I always understood your effort was well-intended, not poorly done. I’m sorry for that. I also am a strong supporter of micro-enterprise, and if you click on my name it will open my website (sunmoney.org), where I introduce a strategy that seeks to combine the larger-scale financing of social enterprise with the tiny-scale of microfinance by creating the “metamarketplace”. I believe this is a sustainable solution for people everywhere, and so for our planet as a whole. Welcome to the future?