This week, The Chronicle of Philanthropy wrote an article about the creation of the Alliance for Effective Social Investing. We wholeheartedly applaud Steve Butz and other members of the Alliance for their efforts, and really hope they succeed. There are too few organizations focusing on the effectiveness of charitable programs, and we’re excited to see their first results.
I’ve briefly looked at the survey they plan to send to nonprofits and here are some quick thoughts. Ultimately, the survey focuses on procedures and processes as opposed to impact and results and therefore has two problems:
- Charities often say they track outcomes even if they don’t. This happened to us consistently last year. (You can view our Round 1 application we sent to international charities linked on this page along with all the materials we received. In particular, look at the answers to section III on the application. )
- There are no specifics about what each organization does and what effects it has. All the questions are abstract about whether or not each organization tracks their outcomes. Donors need to know what impact they can expect from their donations not whether or not a charity has a “process in place” to track outcomes.
[As an aside, I’d really appreciate a tool that simply lists all of an organization’s programs. If there’s one type of information I’d like for all charities, it’s a simple specification of what they do and where they work. This is not currently available anywhere. (Guidestar offers very brief summaries of a charity’s programs off its 990, but nowhere can I see the specifics of each of its activities.) We’re currently working on building such a tool for international aid, with the help of some great volunteers.]
Finally, I’m concerned that this survey won’t accomplish its most important goal: distinguishing between effective and ineffective organizations. We’ve analyzed the Children’s Scholarship Fund and the Nurse-Family Partnership. Both organizations collect a large amount of data about their clients, and I believe they would each answer the Alliance’s survey identically. Nevertheless, we believe that NFP is running an effective, impactful program and we strongly recommend them; we think that CSF’s strategy is marginally effective (i.e., not making a substantial difference) at best and ineffective at worst.
Any useful evaluation tool has to distinguish between two programs like CSF and NFP. If it doesn’t, it falls short in the most important way.
All of this criticism is offered in the hope of dialogue and improvement. We’re rooting hard for our competitor, and if they want any help or information from us, they’ll get it.
Comments
The Alliance for Effective Social Investing isn’t a GiveWell competitor. The alliance doesn’t rate organizations. What the Alliance is trying to do is develop a set of standard indicators that can inform an investor of the risks and responsibilities of a social investment. As Elie stated, the indicators developed so far do focus primarily on processes and procedures. Insight into process and procedures provides an understanding of an orgs ability to collect data, analyze their performance, and make adjustments to improve. This is essential in assessing the likelihood that an org can achieve its objectives. The alliance is providing these indicators to anyone interested in using them – including GiveWell.
Hey Elie, thanks for taking note of the project. I’m one of the Alliance Members and I don’t see us as competing with GiveWell at all. The Alliance tool is not meant to provide an analysis of the impact that a nonprofit achieves. It is a tool to evaluate the potential that an organization has to create impact.
Let’s look at a for-profit example. Investors care about how much future profit a firm makes. But just understanding how much profit a firm is currently making won’t completely answer your questions. Just as important are understanding if the firm is run in such a way that it has the potential for continued profits.
So I see the Alliance tool as something that GiveWell might use in your analysis, but it does not at all replace the work you do.
I will agree with you on one important point. Surveying nonprofits rather than directly observing what they actually do is problematic. People don’t always behave the way they say they behave (not just because of lying, but because there is generally a gap between people’s actions and the way they see themselves). I’ll be attending the first meeting of the Alliance members next week and I’ll be raising questions about the potential issues with the survey methodology.
Just to be clear, the tool was created by Steve Butz, the Alliance members have been recruited to help develop the tool. So I didn’t have any role in creating the tool as it exists today.
Jeff and Sean, thanks for the comments.
I guess I’m pretty unclear on what your goals are for the Alliance. Is your goal to create a resource to help donors decide where to give? (If so, I have some thoughts on how I’d use it and what I think it’s missing.)
If that’s not your goal, what are you ultimately trying to achieve? How do you envision people using this tool and what will they do with it?
Elie, I also work with the alliance. The alliance wants to create a standard set of indicators that can be used to influence a more effective flow of funds into the sector. Our hope is that the indicators we create are widely adopted by rating sites and funding organizations. There are still many questions that need to be answered. The alliance will meet in person for the first time next week. The self-evaluation point that you and Sean make is just one of many that will be discussed at the meeting. As the work of the alliance progresses, information will be posted to the alleffective.org website. Success will require inclusion of a broad range of ideas and feedback. We welcome your thoughts.
Jeff, thanks for the response. Here are a few quick thoughts – I hope they’re useful.
You wrote, “The alliance wants to create a standard set of indicators that can be used to influence a more effective flow of funds into the sector. Our hope is that the indicators we create are widely adopted by rating sites and funding organizations.”
As someone who works for a ratings agency/funder, here are my thoughts on how I might use this survey:
I’m thinking back to our applicants from last year, and how they’d answer these questions, and I can’t picture incorporating that into our decision process. For that reason, the survey as it’s currently constituted is not something we’d likely include in our ratings.
It’s possible that a few sample responses would help demonstrate the value more clearly. I’d be interested to see survey answers from a few, real organizations and some explanation of how a funding organization might use those answers to inform their decision making process.
Elie,
Thanks for including us in your blog- my hope is to respond more thoroughly- this will probably not happen until the weekend…
Anyway, an explanation of the tool- which is absolutely still in development- and the thinking behind the tool is available online http://www.alleffective.org (see the Guide to Effective Social Investing)
I think the context of the guide will further clarify our thoughts on “rating” non-profit organizations.
Clearly, there is more work to be done.
I am a member of the Alliance and one of the authors of the document under discussion. I think much of this discussion misses a simple point. For a social service agency and its programs to have a high capacity to produce social value, it must rate high in THREE DOMAINS:
1. Tactical data use – with 2 indicators:
*data integrity
*an outcomes focus
2. Strategic data use – with 2 indicators:
*relating staff efforts to client outcomes
*making adjustments to services in light of results
3. Program value – with 2 indicators:
*evidence of program effectiveness (ranging from from anecdotes only to multiple RCTs)
*capacity to deliver a program with fidelityt to the model (implementation and performance standards are met consistently)
The 26 questions allow ratings of all 3 domains and 6 indicators. So Elie’s first comment does not do our work justivce because of course we do look at evidence that the agency and its program(s) can produce significant changes in people’s lives. Then later on Elie fails to appreciate that we recognize a scale of increasing rigor for demonstrating program effectiveness.
And to be clear: a program that rates high in the domains of tacical and strategic data use but not in program value will ALWAYS be a high risk investment! That is, program value is the necessary condition of value creation – but it isn’t sufficient without the other two domains (correctly referred to as process domains).
I hope that clarifies things abit…
David
I think there’s been a miscommunication. The key part of Elie’s comment was this:
David, I agree that all 3 of the things you list above are important. The concern is more that asking about them, in the way the survey does, isn’t likely to shed much light. The questions are largely judgment calls, and knowing the applicant’s subjective judgment on the extent to which they use data tactically/strategically isn’t nearly as useful as seeing that data for oneself. I say this based largely on our experience reading applications in the past: seeing how a nonprofit describes its data gathering process and seeing the data tend to create completely different impressions.
What’s good about the RCT question is that it’s a clear factual question. I know exactly what a “Yes” means for that question.
I hope there is no misunderstanding about the practical use of the assessment tool. Although the article does not highlight our thoughts about this particularly well:
“One big unknown in Mr. Butz’s effort is how to use the tool to rate thousands of social-service groups and make the results available to donors. Mr. Butz says that charities might conduct the evaluations themselves, at least at first, something that would not require a big outlay of money.
But he would prefer to have an independent group get involved. David E.K. Hunter, a consultant and former director of evaluation and knowledge development at the Edna McConnell Clark Foundation, who helped Mr. Butz create the assessment tool, envisions hiring a team of professional evaluators, a costlier endeavor.”
There is a clear understanding on our part that an objective, independent assessment of the organization would produce more reliable ratings. This is something we intend to work very directly towards over the coming months.
Also, in the example Elie gives above, it is fairly clear that the two organizations cannot be rated equally, specifically on the question of social value creation. I would expect it to be fairly obvious that an external evaluator would not “Strongly Agree” that both organizations (at least given the surface information presented in your report) have been “proven” to create social value.
We are hoping that the assessment tool can serve two purposes:
1. Internal Self Assessment for the Purpose of Organizational Learning. The tool can be used for a self-assessment to be completed by the organization. In our view, It is best if multiple people answer the assessment questions – including the executive director, the director of programs (where one exists), each program director, and at least one direct service worker in each program. It will be useful to aggregate and average the results. However, the greater the degree of agreement among all those who rate their organization, the greater the likelihood that the aggregated scores are capturing a reasonably accurate picture of the organization for each indicator.
2. External Assessment for the Purpose of Creating a Valid Rating that is to be Disseminated as an Aid to Social Investors who Want to Understand the Level of Risk that a Potential Social Investment Poses. A trained evaluator will make a one-day site visit and use the tool to review all data sets used by the organization at all levels to drive performance, and also conduct selective interviews as indicated, answering the same set of questions. The evaluator’s assessments of each indicator will be disseminated by a trusted organization whose mission is to help social investors understand the degree to which social investments in a given nonprofit organization are likely to be “blue chip” with minimal risk to the objective of achieving an immediate social return, or “social venture” investments with less immediate social value but with developmental possibilities for the creation of high social value in the future.
We believe that both uses of this tool are important. We intend the former will guide organizations in their ongoing efforts to become strong, reliable, sustainable, and effective creators for social value. The latter, we intend, will contribute to a the creation of a much more effective social sector with revenues channeled with high intentionality to both “high risk” and “blue chip” organizations…depending on the nature of the investments that funders want to make.
Finally, I have some reservations to share around exactly how valuable reviewing the “data” directly is to an outside observer. In my experience in both working directly in the non-profit sector (10 years) and working to provide non-profit organizations tools to assist them in managing their performance (9 years) the data being reported as outcome data can be very suspect indeed. I believe the culture of an organization around data integrity, the ability of an organization to relate their services data (typically outputs) and program outcome data, and most importantly the degree to which they are willing to manage their performance accordingly are as much indicators of a high-performing non-profit organization as the raw data they are able to provide. The point is that both are necessary.
We have assembled an excellent group of industry luminaries to discuss these issues next week so we would hope to have even more useful information to share at that point.
Thanks for the clarifications. It seems to me that (1) and (2) are fundamentally different in terms of both what challenges they raise and what problems they would solve:
(2) sounds quite similar to – in fact a higher-intensity version of – the approach GiveWell took in our first year. The challenge, I believe, is simply that it takes a lot of resources (time, money, talent) to rate even a single organization in this way. Trying to rate enough to create credible recommendations for donors raises questions about how to perform such evaluations in a cost-effective, large-scale way.
Tackling this problem would make the Alliance a direct competitor to GiveWell, by which I mean an organization trying to accomplish what we’re trying to accomplish. We would be thrilled with this development – it would mean we’d have someone to share ideas with and benchmark ourselves to. Since we’re mission-driven and not profit-driven, “competitor” need not mean “adversary” – we’d be happy to share info and help each other out.
(1) clearly would not involve the same challenges of scale and expense that (2) does. At the same time, it seems to me that (1) alone has very little potential to accomplish the Alliance’s stated mission of “changing how funds are distributed in the social services sector – from giving based solely on anecdotes to informed social investing.”
Comments are closed.