We’ve been working on an update of our disaster relief report, and came across an American Red Cross evaluation from December 2010 stating:

If you would like to access this report, please get in contact with the ALNAP secretariat.

We emailed the ALNAP secretariat, saying:

I am writing from GiveWell, an independent, non-profit charity evaluator to request access to the ALNAP report “American Red Cross – Haiti” that is listed on the ALNAP website at http://www.alnap.org/node/7131.aspx. Would it be possible to send us a copy of the report?

The secretariat responded that the evaluation cannot be shared externally due to an in-house policy.

Why should this report be confidential over a year after its publication?

(Thanks to Eliza Scheffler for finding this.)

Update: the page linked to in this post regarding the evaluation appears to have been removed (very recently - I am writing this at 11:02am and it was up as of 10:30am). Here is Google’s cache of the site and here is a copy of the Google cache stored on our server (for when the Google cache expires).

We recently complained that “UNICEF provides no information about where the money goes and what projects are in progress.” Some might feel that this complaint comes from unrealistically high standards of transparency, especially for organizations such as UNICEF. How is an organization spending $2.7 billion a year supposed to report its activities?

Our answer would be: “like the Global Fund to Fight AIDS, Tuberculosis and Malaria (GFATM) does.” (Page 55 of its 2008 annual report shows that its budget size is very close to UNICEF’s at $2.7 billion.)

GFATM provides an online program search of all its activities. For any grant it has given (example), you can see (if completed) the grant proposal, grant agreement, and reports on progress. In other words, you can see how much has spent and how (and whether) progress has been tracked.

GFATM recently released the kind of document we have never seen from any other charity approaching its size: an overall evaluation of its activities and impact. Not a general discussion of the organization; not a “meta-evaluation” discussing the quality of past evaluations; a discussion of the overall impact of all of GFATM’s activities across the world. Furthermore, this report was in no way a fundraising document; it was frank about the fact that inadequate evidence exists for GFATM’s impact to date (see the discussion at our review).

GFATM comes under a lot of criticism, even from its own evaluators. We ourselves have many reservations about its work, as our review establishes. But we have seen very few charities - and no other charities approaching its size - that can make as strong a claim to being a transparent organization and a learning organization.

GFATM proves that neither size nor celebrity support need stop a charity from being clear about what it’s working on and how it’s going.

Why has Kiva been singled out for so much criticism lately (see GiveWell Board member Tim Ogden’s summary)?

Part of the answer is that Kiva has arguably been misleading donors - but that can’t be the full answer. David Roodman’s original post could never have come about without the fact, as Mr. Roodman puts it, that “the way Kiva actually works is hidden in plain sight.” And our followup analysis on repayment rates was only possible because Kiva makes all its repayment data publicly and easily available. As Elie said in that post, “Similar analysis would be impossible for nearly every other charity I can think of.”

Contrast Kiva with, for example, UNICEF. Kiva makes it possible to trace the path of your donation, to the extent that such tracing is realistic (and it largely turns out to be more along the lines of “you funded a certain MFI” rather than “you funded a certain person”). UNICEF doesn’t even seem to have a breakdown of how much money is going to each continent. We definitely can’t find information on questions like (a) What specific projects are you funding? (b) What is your role in each? (c) What new projects are planned, and where? (d) How is each project going, whom is it affecting, and how?

There are no strange patterns in UNICEF’s numbers because there are no numbers. There are no contradictions because there is no concrete information. And the intent here isn’t to single out UNICEF - it’s merely one of the vast majority of international aid organizations about which we know essentially nothing.

Giving an impression to donors that’s undermined by the facts is a minor scandal. When will complete opacity - simply sharing no information at all - be a major one?

How transparent is CARE?

On one hand, it maintains a site at www.careevaluations.org that currently lists 448 project evaluation documents (352 of which are in English). We haven’t found anything comparable for any other of what we call the “household name” charities – enormous, well-known, aggressively fundraising international aid charities (usually members of the InterAction network) that conduct a huge array of different programs in different places.

On the other hand, it does not appear to link to this website anywhere from its main website - in fact, there appear to be only four external links to the site anywhere on the Web.

Looking through the evaluations provides an interesting example of what one of these “household name charities’” operations and impact evaluation look like. The variety of the projects and of the evaluations is huge. Some evaluations examine measures quite relevant to “impact,” such as reported behavior change and children vaccinated (example); others are looser, mentioning regional trends in disease burden but focusing on qualitative generalizations (example); others do not examine life outcomes at all, but simply make qualitative observations about strengths and weaknesses of the program evaluated (example). The quality and tone of the studies varies considerably as well. The use of “control groups” to assess impact is rare but occasional; none that we examined have what we consider to be a high level of rigor, but many appear encouragingly honest about program weaknesses as well as strengths.

Note that this set of evaluations appears to be far from comprehensive: CARE currently lists 845 active projects, whereas the database (which in some cases includes more than one evaluation per project, and goes back to 1991) contains only 448 evaluations as of this writing.

This isn’t the level of impact evidence that we see from our recommended charities, but some evaluation is better than no evaluation and non-publicized disclosure is better than no disclosure.

As a side note, CARE appears to be the only “household name charity” that turned down government funds during the debate over US-provided food aid. We aren’t sure whether they have the right side of this debate, but the turning away substantial money is unusual among charities, and suggests that CARE’s staff aren’t always putting fundraising first.

Bottom line: we’d recommend these charities over CARE, but we’d recommend CARE over other “household name” charities.

Global Health Report:

A computer science group from the State University of New York at Stonybrook presented three applications or “apps,” that is to say mini-computer programs, that they had designed for use on no-frills mobile phones owned by women working in the informal Senegalese economy. The pilot tests for two of the apps—a dictionary and a book-keeping calculator—were deemed successes. The third app—for measuring profit and loss—was judged a failure.

…

The pilot was a failure because the fish sellers found the mobile phone profit-and-loss calculator useless. The Stonybrook group did not learn why the app was useless, however, until a second round of testing in which one of the Senegalese computer science students happened to have a grandmother who was a fish seller. After talking with the fish sellers, he explained to the Stonybrook group that all the prices for both fresh fish and dried fish are fixed. Since everyone charges the same price for fish (one for dried, the other for fresh) on any given day, there is no way for the women to wait until the price is right.

It’s a small example, but we wish we saw more stories like this - public sharing of people trying a program, critically assessing it, and learning from what doesn’t work.