These charts from Smile Train imply an appealing story:
(a) Smile Train performs surgeries for $250 apiece.
(b) Smile Train’s main use of donations is to fund $250 surgeries.
(c) A donation to Smile Train funds more surgeries than would the same donation to another cleft palate organization.
(d) If Smile Train had much more money, it would fund many more $250 surgeries.
But after some basic investigation of their website and financials (and conversation with the organization), we believe that:
(a) Smile Train’s total expense per surgery is well over $250.
(b) Directly funding surgeries (the program focused on in the charts above) accounts for under half of Smile Train’s expenses.
(c) Smile Train makes direct grants to other cleft palate organizations, some of which use the same model as the “mission groups” that its charts imply are overly costly.
(d) If Smile Train had more money, it could mean more grants to “mission groups,” general education about cleft palate conditions, and a host of other activities, and would probably not mean more directly funded surgeries.
We feel that, as a result, many of Smile Train’s donors – including Prof. Steven Levitt of Freakonomics and Superfreakonomics – have a highly inaccurate picture of how their donations are used. Details follow.
(a) Smile Train’s total expense per surgery is well over $250.
The same Smile Train page linked above claims 90,000-95,000 surgeries provided in 2008. If this number is entirely correct (not exaggerated or estimated) and we divide it into the roughly $100 million of “money raised” shown above, we obtain a cost-per surgery of over $1000.
In fact, the situation is a bit more complex, because money raised doesn’t equal money spent. To get the latter, we can look at Smile Train’s most recent IRS Form 990, which puts total expenses around $85 million for an implied cost-per-surgery closer to $900.
It’s possible that some surgeries are performed for $250 apiece or less. But are these the surgeries you’re paying for when you donate? That’s where the next few points become relevant.
(b) Directly funding surgeries (the program focused on in the charts above) accounts for under half of Smile Train’s expenses.
Here’s the big picture on Smile Train’s expenses, as far as we can understand them from the most recent IRS Form 990:
“Treatment partnerships” appears to be the core program of funding developing-world doctors to perform surgeries.
The “No information available on website” item refers to $23.6 million spent on “provid[ing] materials on cleft lip and palate for free to anyone interested in this birth defect” (according to the Form 990). We have no other information about this activity.
“Education grants” appear to refer to training developing-world surgeons. “Treatment grants” appear to correspond to grants to other organizations.
(c) Smile Train makes direct grants to other cleft palate organizations, some using the same model as the “mission groups” that its charts imply are overly costly.
Prof. Steven Levitt of Freakonomics and Superfreakonomics writes,
The typical model for cleft repair in developing countries has been to convince U.S. doctors and nurses to volunteer a week’s time, fly to a country, and do 80 surgeries … Smile Train instead partners with and trains local doctors to do the surgeries, which turns out to be far more efficient.
In fact, Smile Train makes grants to a variety of other organizations, including some that focus on the “mission” model (flying U.S. doctors and nurses overseas). The most recent list of grants starts on page 18 of the Form 990, and includes:
- $1.4 million to GiveWell standout Interplast, which funds both missions and work with local doctors.
- $70,000 to Surgical Volunteers International and $25,000 to the Smiles International Foundation, both of which appear to be missions only.
- A wide variety of other grants, including for organizations focused on research, organizations that work primarily in the U.S., and more.
(d) If Smile Train had more money, it could mean more grants to “mission groups,” general education about cleft palate conditions, and a host of other activities, and would probably not mean more directly funded surgeries.
I first encountered some of these oddities back in 2006, and wondered why Smile Train wasn’t simply putting all available funds toward its most cost-effective surgeries (presumably, the ones that cost $250 each). The answer I got at that time was that there was a “doctor shortage” – Smile Train was paying $250 per surgery as much as it could, but had more money than it could use that way, so it engaged in other activities and made grants to other organizations.
Given this situation, I think Smile Train’s approach makes sense. How many charities are raising more cash than they can productively use, and sitting on it instead of giving it away to other organizations?
In terms of spending money it has appropriately, Smile Train may be doing well. But how it brings in that money is another question. When Steve Levitt doesn’t seem aware of how funds are being used, it seems to me that donors are getting the wrong impression.
Prof. Levitt notes that “One thing they don’t do, but maybe they should do, is literally link each $250 donation to a particular child and send before and after pictures.” Smile Train can’t take this suggestion, because it’s not providing a surgery for every $250 raised.
Comments
Suppose that Smile Train has a comparative advantage in fundraising than its competitors due to some combination of misleading advertising and marketing savvy.
Assuming that ST’s misleading ads draw more money into the cleft palate nonprofit sector than would otherwise be donated, since ST donates its excess (minus fundraising overheads, naturally) to them, isn’t this a net gain for the sector as a whole?
Oasis, what if those extra donations would have been made instead to other more promising treatments, such as vaccinations, bednets, and condoms?
i was being sarcastic. it was a parody of their possible justification for lying
Ian, are we to infer from your response to Oasis (re “more promising treatments”) that children born with cleft palates who will likely die from malnutrition (due to poor swallowing ability) are less worthy of life than a child vulnerable to malaria or other disease? While I applaud your critical analysis of charities, you offer few alternatives to those who seek to aid specific, neglected populations.
Can you do this same analysis of CHARITY WATER’S claims that $20 provided x for x for x years, or $5000 provides a well for x people for x years?
Reader,
I doubt that Givewell folks have the time to chase down every charity out there, but you might want to see their detailed detailed analysis of water supply programs in general and their discussion of a selection of water charities.
Stacy, one doesn’t have to believe that children with cleft palate are “less worthy of life” to believe that – for example – charities that distribute antimalarial nets are more worth funding. We’ve previously argued that importance of a problem is not the same as worthiness of a cause.
Reader, thanks for the suggestion. I took a quick look It does not appear that their figures contradict each other in the clear way that Smile Train’s do/did. I have questions about exactly what the numbers mean, as well as general concerns about water charities. But we will need to check this group out more to have much to say about it.
Children with clefts are being helped more efficiently by a local doctor in their hometown. The results are immediate and obvious. I found this on their website under financials” While our cost per surgery varies across the 78 different countries we work in, our contribution per surgery averages about $250. This amount reflects the contribution per surgery that we provide our partner hospitals. They also cover a significant portion of each surgery as part of our cost-sharing agreement.” It seems to me there is much more to funding surgeries than just the direct contribution–training, medical equipment etc. I have been a donor for years and continue to support The Smile Train.
As a donor who is intimately familiar with the sector, and with Smile Train, I would like to respond.
First, much of what you say seems moot, as the chart in question is indeed no longer on the site, undoutedly removed because it reflected old, outdated information. But let?s take a closer look at your argument anyway.
?(a) Smile Train?s total expense per surgery is well over $250.?
I am well-acquainted with Smile Train?s financials, and can assure you that over the past 10 years, their average cost-per-surgery is $250.
?(b) Directly funding surgeries (the program focused on in the charts above) accounts for under half of Smile Train?s expenses.?
As I am sure you aware, there is considerably more to funding surgeries than simply the direct-per-surgery contribution. Smile Train must pay for training, education, medical equipment, travel, food, advertising, outreach efforts, doctor salaries, etc.
?(c) Smile Train makes direct grants to other cleft palate organizations, some of which use the same model as the ?mission groups? that its charts imply are overly costly.?
Cost-effectiveness is always the goal, of course, and I think that is clearly reflected in Smile Train?s model as well as their demonstrated success. But what do you mean when you say ?overly costly?? Certainly there are times when local doctors are simply not an option. Haiti is one good example, where there is not a single local surgeon who does clefts. And in those unfortunate cases, of course a mission group must fly in, and expenses are consequently higher. I trust you would agree that it is certainly better to pay more for the surgeries and help these kids than to provide no surgeries.
?(d) If Smile Train had more money, it could mean more grants to ‘mission groups,’ general education about cleft palate conditions, and a host of other activities, and would probably not mean more directly funded surgeries.?
What is the basis for this hypothetical? The number of surgeries Smile Train provides is directly linked to how much money they raise. The more money they raise = more children they help. It?s just that simple.
Sonya,
I’m gonna take you up on this one:
Question (a) Is cost per surgery more or less than $250?
I’m afraid you’ll need a more convincing argument than “trust me” to make us share your view. Let’s start with this: Smile Train’s 2008 form 990 says they spent $86.7 million in that year. $86.7 million divided by $250 yields 346,000. On what basis do you conclude that in the fiscal year ending in 2008, Smile Train’s activities resulted in 346,000 more surgeries than would have been performed otherwise? If it’s not, and if Smile Train’s mission is to ensure that surgeries are performed, then in what sense is the claim true?
Cheers,
–Ian
The amount of money Smile Train spends on marketing is outrageous. Their fundraising letters are essentially blackmail, saying that they’ll never ask you for money again if you give them something. If you don’t fork over they send letters about once a month with a picture of that poor child on the envelope to shock and disgust people. I have never seen any charity engage in such a despicable marketing campaign.
In the fundraising materials I’ve received, they stated that “overhead”, including the costs of fundraising, was totally covered, although I can’t remember if they said a large endowment or the board of directors or what. But I distinctly remember that it said that 100% of my donation would directly fund cleft lip and palate surgery–that if I donated $250, that would pay for one cleft repair. That (along with Walter Cronkite and many many other well-known people) really impressed me and is why I started donating small amounts occasionally.
So was that a total lie?
I went to look at the “financials” section on their website, and I see they are no longer doing the $250=one surgery thing. Now they discuss other things that are not actual surgeries but that are important–operating room equipment, orthodontics, speech therapy, even food to build their bodies up enough so that they can be operated on. You know what? That doesn’t bother me, even though it does raise the strict money in/surgery out ratio.
Perhaps their endowment ran out. Compared to many charities, their fundraising+management-to-service ratio looks pretty good.
Rene,
Agreed that it’s absurd to ask Smile Train to pay for surgeries only without creating an environment where successfully surgeries are even possible. But those costs must be accounted for in determining their cost efficiency.
The program expenses ratio, on the other hand, is totally useless. No problem with a charity spending a lot of money on fundraising if it brings in the cash, and lots of charities run an extremely clean ship but nonetheless have little to no impact.
Givewell calls the ratio of fundraising and management to program expenses the “straw ratio”, and they go at it with a fair amount of anger and rage in these blog posts. (You might want to read bottom to top on that page)
Ian,
Thank you for the link. At first glance, looking at “overhead” vs. “money going to whatever the charity does” seems a reasonable way to tell a decent charity from a not-so-good one. For example, the Police Protective Fund (money for widows and children of police officers, “police officer down” education” spends something like 78% of donations on overhead, leaving only 22% for any actual charity work–once I learned that, I decided to never donate to them again.
Frankly I had never known (or thought about) what “managerial expenses” actually was. I assumed salaries, rent, utilities, etc. I guess I assumed that “evaluation of effectiveness” would be included in program expenses.
So if it’s not, then there IS no way to really evaluate any charities effectiveness. The planning and evaluation isn’t being done, so how CAN one judge? He’s convinced me that it should be part of a charity evaluation but until that becomes part of it, what’s left is the dreaded ratio and the charity’s own promo material, including the films (for example) of many happy families with children no longer ostracized from school and life.
I would LIKE to get a better handle on what charity is doing the best with the money I may give it, but how?
Rene,
I agree that it’s certainly possible for a charity to overspend on fundraising. But it’s hard to tell what constitutes “overspending”. For example, consider a charity that holds lavish events in France for its largest donors, which end up costing 50% of the amount donated. That doesn’t actually mean it’s a bad organization — if the programs are effective, then you can give freely, turn down the lavish trip to France, and create impact.
Likewise, it’s possible (though unlikely, in my opinion) that the Police Protective Fund is having an impact. If you write a check to the charity, it won’t go through the telemarketers and won’t be subject to the same fundraising overhead as other organizations.
Likewise, there’s not really an upper threshold on administrative expenses. Partly that’s because there’s a large amount of leeway on what constitutes program expenses, but mostly because proper administration makes for a more effective organization. Consider a charity that distributes vaccines and spends 80% of its money thinking about the most efficient way to do so, while spending only 20% actually distributing the vaccines. If the analysis makes the distribution 5X more effective (very possible), then it was worthwhile.
I’ve exaggerated the numbers, but the latter example is basically VillageReach in a nutshell.
Finally, to answer your last question — if financial analysis doesn’t provide a clear picture, how do you decide how to give? The only answer I’m aware of is the GiveWell approach, which requires a painstaking commitment to skeptical analysis. What I mean by “The GiveWell” approach is to research a charity’s evidence of impact (most charities have none) to find a conservative quantitative figure for what the organization is actually accomplishing. This process is hard because charities are rarely forthcoming and because you have to take into account the various biases and unintended consequences that can crop up, like selection bias and offsetting effects.
The GiveWell approach is also the approach taken by any high-performance foundation, although foundations are more likely to take risks on promising but unproven programs.
Because this analysis is so time-consuming and difficult, I’m extremely grateful that someone else (Elie and Holden) have taken it upon themselves to do the research for the benefit of the public.
Cheers,
–Ian
Sonya, you have claimed that Smile Train’s implication of providing an additional surgery for every additional $250 is correct. But you have not provided any evidence. As discussed above, the evidence we have points strongly to the claim’s being false.
Rene,
Operation Smile is another organization that seems rather similar to Smile Train.
Here’s a bit of research on them:
In the “Donate Online” section of their website, they say “For as little as $240US or just $20US a month you can help provide a cleft lip surgery that will transform the life of a child.” The 2009 Holiday Gift Catalog, under “Give a surgery” also indicates $240 to “Help Give a Smile to 1 Child”. This holiday guide allows donors to support certain related activities (Blankets, various hospital supplies and such) that might suggest to some donors that the total effective cost is somewhat higher than $240/surgery, but overall, IMO, it would be easy for a donor to think that $240 = 1 smiling kid.
So, what’s the reality?
In the 2008 Financial Statement on their site (under ABOUT US – Financials), if I read it right, as of 6/30/08, their preceding year total expenses were $56,268,802. That includes fundraising and admin of $11,893,851 and $2,710,783 respectively. Revenues were $59,809,711 (they had a modest surplus for the year). Revenues include $22,240,120 in “Contributed program services” (basically medical supplies and services, at fair market value, if I understand correctly) and a different line for $4,335,273 of donated “Medical supplies and air travel”). So fundraising is a significantly higher percentage of things if we look just at cash donations than if we include the FMV of all the donated stuff. But maybe the fundraising costs also encourage doctors and such to donate their time. In any case, IMO, assuming their accounting for the donated services is reasonable, we should count the doctors’ donations of their time and the other in-kind donations as we count other things.
OK, so how many surgeries does this produce?
Per their 2008 annual report, pages 34-35, they provided 19,000 physical exams and had 10,868 patients treated. I’m going to assume that a patient treated is basically a smile surgery performed, and that the physical exams are basically an adjunct to the delivery of those surgeries (pre- and post- op exams and such, including some screening).
So, if we take the total expenses, divided by the total patients treated, we have:
$56,268,802 / 10,868 = $5,177.47
So it seems to cost a bit over $5,000 per surgically delivered smile, including all kinds of donations, against the primary outcome this charity delivers (again, ignoring the physical exams on the assumption that they’re basically part of the cost of delivering surgeries.
Again, the site indicates $240 to “Help Give a Smile to 1 Child”.
Again, note that Operation Smile is not the same organization as Smile Train. But we see a similar pattern of something advertised at what appears to be a fraction of the true cost of delivering it.
It may very well be that these surgeries are worth the higher cost, but the presentation of information on the website is troubling to me.
My suggestion of where the $250/surgery number comes from:
– Take the $23.6 million they claim is for “providing cleft palate treatments for free” not through other organizations besides Smile Train
– Take the 95,000 surgeries they said they did in 2008
– divide $23.6 million by 95,000 = $248
However, very clearly, this means that only a fraction of the donations they receive actually goes towards this number. To a donor that’s not on their board, this still means that 100% of their donation is going to cleft palate surgery, it’s just not the surgery they said they were supporting.
Emily, your calculation seems plausible – it could be that a specific sub-program of Smile Train results in a surgery for every $250 spent. However, it’s not just Board members who don’t fund this program; in fact less than half of Smile Train’s funding is going to it. I would guess that the marginal donor (i.e., someone who comes in today with a donation that is additional to Smile Train’s existing revenue sources) is not funding this program at all, because I would guess that there is no “room for more funding” in the program.
There is something else that worries me in Smile Train’s accounting.
Their 990 for the fiscal year 2007 has $32,242,254 spent towards “Publication and printing”. Only $13,541,747 of the $32,242,254 was booked as “Fundraising”, while the remaining $18,700,507 was booked as “Program services”.
Therefore,”Publication and printing” accounts for 27.5% of the “Program services” total for that fiscal year ($68,118,525). This fraction appears to be very large. Is this related to “Free Training and Education” materials given to surgeons?
While I think that givewell’s analysis of these organizations is great, I think that it would be immensely valuable to give the organization a chance to respond.
Liz, we do give organizations a chance to respond. In addition to the open comments section of this blog, we also offer a prominent “Charity response” to any interested group – see https://www.givewell.org/united-states/charities/Year-Up for an example.
To me the numbers look odd. Smile Train’s website says that 5000 children are born with a CP in the USA each year. I am sure they are right. The USA has a population of 307million. Although ethnic incidences vary, and so do birth rates, you could assume that with a population of 1Billion in Africa around 15,000 to 30000 children are born with a CP in Africa each year.
If Smile Train single-handedly did nothing else but fix every newborn in Africa without any state medical intervention at all, Smile Trains $87m annual spend would result in a cost of between $3,000 to $6,000 per case.
But Smile Train maintain they do 95,000 procedures each year so they must be doing much much more than the equivalent of fixing every newborn in Africa by themselves.
Goatero, according to Smile Train’s website, it works in 78 countries, many of which are located outside of Africa. Together these countries are home to about 5.5 billion people with about 111 million births per year (data from the World Bank for 2008; details in this spreadsheet). The World Health Organization estimates that “cleft lip and/or palate…occurs in approximately 1 per 500-700 births,” while noting that “the ratio vary[s] considerably across geographic areas or ethnic groupings.” Thus a crude estimate of the number of children Smile Train could theoretically provide operations to would be 158,000-222,000 per year. Accounting for the fact that many children with clefts require multiple surgeries, the total number of surgeries needed in the countries in which Smile Train works is likely at least an order of magnitude higher than you suggest.
We would guess that the bottleneck to providing more low-cost surgeries is not lack of patients, but rather a lack of surgeons (or lack of surgeons with free time) who are within reach of these patients.
Natalie Stone, GiveWell Research Analyst
Check out this blog http://www.loveandwaterinternational.com/2010/01/smile-train.html
i think that smile train is doing an amazing job of letting kids smile again
http://extratv.warnerbros.com/2010/09/christie_brinkley_onboard_the_smile_train.php
So, if you wanna help these kids and smile train might be shady/ineffectual-in a sense-what do you do?
Rebecca, our top recommendation in this cause is currently Interplast – see our surgery page – though if you’re just looking to help people as much as possible we recommend our top charities.
This post has generated a large number of short comments both from people who support Smile Train (and disagree with our post) and those who don’t and comment on use our post as a platform for voicing their disapproval of Smile Train.
In general, we are liberal with the comments we let through, only moderating those comments that are obviously spam. We are particularly sensitive to this issue when the topic under discussion is the effectiveness of a particular organization. Because of the number of comments coming through on this old post, we now plan to start moderating comments on this post that don’t add new information to the discussion.
We’re planning to moderate positive comments along the lines of:
or
and negative comments along the lines of
As the founder and executive Director of of Surgical Volunteers International I found you article on Smile Train misleading concerning my groups use of Smile Train funds. the first part of the article lumps us with the mission groups that spend over $1,000 per patient. Nothing could be further from the truth.
We have no paid employees, no office bills, no phone, fax or internet. All that is done from my house and not charged to the charity. We did received a grant of $70,000 from ST. With that grant we did 280 cleft patients and all are documented on Smile Train Express with data and photos. tflood1431@gmail.com
What the hell is the point here ? Jesus , learn how to construct an argument before attacking a charity. This math wouldnt withstand the curiosity of a 5th grader.
Comments are closed.