- “Local people may be the experts, but for outsiders deciding where their donations can do the most good, getting access to local knowledge and acting on it appropriately requires real-time feedback loops that most aid projects lack.
- Over a little more than a year, GlobalGiving combined staff visits, formal evaluation, third-party observer reports called visitor postcards, and internet feedback from local community members to create a nuanced, evolving picture of a community-based youth organization in Western Kenya that had received $8,019 from 193 individual donors through the GlobalGiving website.
- Initially, youth in Kisumu were happy with the organization. Among other things, the founder used the money to fund travel and equipment for the local youth soccer team. But the first tip-off that something was going wrong came when a former soccer player complained through GlobalGiving’s online feedback form that “currently the co-ordinator is evil minded and corrupt.” The view that the founder had begun stealing donations and was stifling dissent among his members was expanded upon by other community members, visitors to the project, and a professional evaluator.
- In the end, a splinter group broke off and started a new sports organization, and the community shifted their support to the new group. Reflecting the local consensus, GlobalGiving removed the discredited organization from its website.” (Emphasis mine)
Aid Watch stresses the “new way to evaluate a project” angle on this story, and we think it’s a good angle. But another angle is that most aid projects don’t receive this level of scrutiny, and the project that was put under this particular microscope – more as a way of testing the microscope than because there were concerns about the project – turned out discredited.
This is a story that I feel should affect your default assumption about whether an aid project is working.
The comments on Aid Watch’s post are also worth reading. One problem with the “funding a project at a time from many different organizations” approach is that it isn’t clear what one does with evaluation and feedback, when it’s available. Knowing how a project went is certainly better than not knowing, but the ultimate goal is to translate knowledge of what’s working into improved performance.
That’s an argument for focusing on the organization rather than the project level. Organizations can be given incentives to learn from their mistakes and improve their projects. Funding tiny organizations for one-off projects, it isn’t clear how to impose any kind of accountability.
Comments
Holden,
Thanks for your post on this. I would like to elaborate a bit here about our approach, which will hopefully address a couple of your points. I realize we may not agree completely and welcome the discussion since I don’t think there is a single silver bullet here.
First, we agree that most aid projects don’t get this level of scrutiny. That is part of the problem. In the current system, people affected by the project don’t have an avenue to voice their concerns, and thus it is difficult for projects to be refined or redesigned during implementation to maximize impact. Instead, the current evaluation approach only looks backwards and tells you ex-post whether something worked or not. Even when evaluators ask beneficiaries how they felt about a project (which is relatively rare), the information comes too late to make a difference. Worse, the incentives are often too attenuated to make much of a difference in future project design. (I worked in the official aid sector for 15 years, and I can testify to this reality. We spent a huge amount on evaluations, but the sad truth is that good ex-post analysis did not reliably translate into better projects in the future.)
So we recently ran this pilot to test whether it is feasible to allow beneficiaries to be able to provide their feedback – and to do something with that feedback right away, which is the ultimate form of accountability. In our view, the ability to learn and adapt quickly is key to any effective system for funding aid projects. While a number of technical and other issues remain to be addressed, we would like to see this feedback loop rolled out on a large scale. Good project organizations will welcome this type of real-time feedback ( good organizations such as Seva Mandir or BRAC spend a lot of time listening to beneficiaries, and welcome new tools or listening more effectively.) Bad organizations will fear it, because it makes it harder to hide poor performance. Mediocre organizations will have an incentive to listen more.
Even though we feel that feedback at the project level is critical, we agree on the importance of rating an organization overall. On GlobalGiving, organizations build an implicit track record as they post multiiple projects, and this track record affects their ability to list additional projects. More broadly, groups such as yours are trying to build an intelligent rating system for the larger organizations, and we applaud that. We also applaud Charity Navigator’s attempts to come up with more meaningful metrics of success and impact. However, we believe these expert and data-driven analyses should be complemented by user-based and community-based feedback on organizations, and to that end we are partnering with GreatNonProfits to pilot this feature on GlobalGiving in the near future.
Overall, our goal is to promote a vibrant ecosystem of organizations that are constantly innovating, experimenting, and learning. There will be both big organizations and small ones, and like any well functioning economy there will be both entry and exit of new organizations based on performance. Making this vision a reality is a long term process that will require many like minded market players to work together – both loosely and formally – over years.
Hi Dennis, thanks for the thoughts. I think we are in agreement on all the issues you raise. I think that introducing faster, more flexible ways to get real-time feedback on a project is a great thing, but that ultimately incentives have to be at the organization level.
By the way, we’d potentially be interested in partnering with you as well to give another perspective on the general transparency (and, when it’s possible to say anything about it, general performance) of the large organizations behind some of your projects.
Holden,
Great – let’s explore in the coming year. Drop by next time you are in DC and we’ll scope out some possible approaches.
Dennis
Comments are closed.